❓ How to use cached hidden states in run_generation ?
## ❓ Questions & Help

https://github.com/huggingface/transformers/blob/5c3b32d44d0164aaa9b91405f48e53cf53a82b35/examples/run_generation.py#L124

This line states that we could use `cached hidden states`. Correct me if I'm wrong :

* **Without using `cached hidden states`** : every step, the next token is predicted, but also all previous tokens are re-computed (which is useless because we already predicted it !)
* **Using `cached hidden states`** : every step, the next token is predicted, but previous tokens are not re-computed, because we are using their cached states.

So using cached hidden states would greatly increase the inference speed, specially for long generations.

---

My question is : **How to do that ?**

From the documentation I understand how to get the `cached hidden states` from the forward pass of the model, but I don't understand how to use it at the following step ?
Hi! Yes, you understood the gist of it. The self-attention related to already computed tokens is not computed again.

In order to use the past, you would get the past from the model pass (I'm using GPT-2 in this example, XLNet would have `mems` instead of `past`):

```py
logits, past = model(**inputs)
```

and you would then use  the past on the following pass as follow:

```py
logits, past = model(**inputs, past=past)
```<|||||>Thank you for your fast response @LysandreJik !

Now it's very clear, but I have one more question :

For XLNet and TransfoXL, we need to use memory in order to not recompute previously generated token. This is ok when not using the memory for something else.

**But what if the memory is already used for something else ?**
Like we have a memory of 256 for XLNet, representing previous segments (or whatever), if we update the memory everytime a new token is generated, it means we are loosing part of the memory (and after generating 256 tokens, we will not be able to see anymore the memory of previous segment !).

**Is there a way around this problem in the current API ?**<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
