Getting different topk results when using past + attention mask for more than 1 sentence
# ❓ Questions & Help

Hi. I'm having some issues when using the past + attention mask functionality. Results are not the ones I was expecting to get... I'm using latest master as the latest release 2.5.1 is failing (`RuntimeError: The size of tensor a (6) must match the size of tensor b (3) at non-singleton dimension 3` at modeling_gpt2.py:150). With master there's no error event being thrown.

My code works with a single sentence, but it doesn't work with more sentences because I get different predictions as I'm expecting to get the same topk results by using past vs no using past at all.

Code snippet below:
```
from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda and" and "Please help me with this"
docs = ["I like to", "Please help me"]
# note: comment the above line and uncomment the following line to make it work with 1 document
#docs = ["I like to"]
docs_tensors = tokenizer.batch_encode_plus(
    [d for d in docs], pad_to_max_length=True, return_tensors='pt')

docs_next = ["soda and ", "with this"]
# note: comment the above line and uncomment the following line to make it work with 1 document
#docs_next = ["soda and "]
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# predicting the rest of the phrase
attn_mask = torch.cat([docs_tensors['attention_mask'], docs_next_tensors['attention_mask']], dim=-1)
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=attn_mask, past=past)
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Results with past:", words)

#####################
docs_full_tensors = tokenizer.batch_encode_plus(
    [d + n for d, n in zip(docs, docs_next)], pad_to_max_length=True, return_tensors='pt')
logits, _ = model(docs_full_tensors['input_ids'], attention_mask=docs_full_tensors['attention_mask'])
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Results without past:", words)

```

Output results (please note the inconsistence between both results - I'm expecting them to match like the next test):
```
Results with past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just', 'The', 'A', 'I', '"', 'This', 'In', 'It', 'B', 'As', 'S', 'We', 'M', 'P', 'C', 'There', 'If', 'T', '1', 'By', 'F', 'You', 'D', 'Image', 'An', 'When', '(', 'On', 'What', 'For', 'L', 'H', 'R', 'About', '[', 'From', 'G', 'After', 'E', 'One', 'K', 'With', 'Still', 'So', 'W', 'by', 'N', 'My', 'Please', 'How', 'O']
Results without past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just', '.', ' site', ' page', '!', ' project', ',', ' website', ' article', ' thread', ':', ' story', ' one', ' and', ' message', ' post', ' issue', ' great', ' blog', ' amazing', ' thing', ' little', ' problem', '\n', '?', ' book', ' game', ' by', ' to', ' wonderful', ' in', ' awesome', ' guy', ' community', ' new', ' mod', ' information', ' web', ' beautiful', '...', ' man', ' stuff', ' work', ' place', ' video', '."', ' app', ' kind', ' piece', '!"', ' world']
```

If I were uncommenting the lines stated in the code I would get same results (...but for only 1 sentence):
```
Results with past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just']
Results without past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just']
```

## Details
<!-- Description of your issue -->
Original stackoverflow question that @patrickvonplaten had answered initially:
https://stackoverflow.com/questions/60459292/using-past-and-attention-mask-at-the-same-time-for-gpt2/
Hi @Damiox, 

First, you have to be careful if you do LM inference with batches of different lengths. Besides using an attention mask, you also have to change the position_ids when using GPT2 (see #3021 ). Considering batch language generation with GPT2 I highly recommend making sure that the input_ids batch is does not need padding (which is actually the case in your example above). If you do have to pad certain batch_idxs then make sure you change the input_position_ids as well and use an attention_mask, but even then it's not promised that GPT2 will generate good results. 

Second, if your prefix input_ids are padded, let's say:
`[ [ I like cats ], [ I like ] ] -> encode_plus -> [ [0, 1, 2], [0, 1, <PAD> ] ]` and you then append something `-> [ [ that are ], [ dogs] ]`, you essentially put in `[ [0, 1, 2, 4, 5], [0, 1, <PAD>, 3, <PAD>] ]` into the model, which is different from what would happen if you `encode_plus` everything directly (`[ [ 0, 1, 2, 4, 5], [0, 1, 3, <PAD>, <PAD> ] ]`). That's why you a) have to make sure `input_postion_ids` are correct and also b) that you sample from the last not padded token for the next word, which is not that simple (again see #3021 ). 
Again, I recommend to only work with `input_ids` that are not padded or just do `batch_size=1` LM inference. 

Three, when appending words of lists make sure that you have a space `" "` between them. In your example above printing the list `[d + n for d, n in zip(docs, docs_next)]` actually gives: 

`['I like tosoda and ', 'Please help mewith this']`,

which is probably not what you want. If you would change `docs_next = ["soda and ", "with this"]` to `docs_next = [" soda and ", " with this"] `both outputs actually produce the same results, but this is only because `docs_tensors['input_ids']` is not padded (same lengths for all batch_idxs)
<|||||>Hey @patrickvonplaten thanks for all your help. Indeed, there was an issue in the code snippet I shared. Thanks!
I see now that pads add complexity because it needs the position_ids to be accommodated, but if it's correctly implemented from my side... shouldn't it just work? I'm confused about your phrase in `it's not promised that GPT2 will generate good results`. <|||||>GPT2 was never trained on padding tokens so I'm not 100% sure whether you will get good results. But it's for sure worth trying out, if you absolutely have to use batch_size > 1<|||||>Alright. I think it should be fine by manipulating the tensors in the "past" as long as I keep consistency across the 24 items in the "past". Right?
For instance... I see that if I generate a past for 1 input with 256 tokens, then I'm getting back a "past" which is a list of 24 tensors of 5 dimensions (e.g. (2, 1, 16, 256, 64)). On the other hand, if I generate a past for 2 inputs with 540 tokens, I'm getting back a "past" which is a list of 24 tensors of 5 dimensions too (e.g. (2, 2, 16, 540, 64)). So I think that if I wanted to exclude the last sentence from the last "past" I can simply manipulate the 2nd dimension in all the 24 items and drop the corresponding value from it. I guess... ? <|||||>> So I think that if I wanted to exclude the last sentence from the last "past" I can simply manipulate the 2nd dimension in all the 24 items and drop the corresponding value from it. I guess... ?

I don't really understand this sentence. If you are concerned about sampling from a padded past, you don't have to manipulate the past variable I think.

Let's say we forwarded:` ['I', 'live', 'in', <PAD>, <PAD>]` then the past variable will be of dimension `(2, 5, 16, num_tokens, 64)`. If you now want to sample from 
`past = key, values of ['I', 'live', 'in', <PAD>, <PAD>]` and 
`input_id = ['London'] `
then you just have to sample from the outputted token because the operation is the same as 
sampling from the last token of `['I', 'live', 'in', <PAD>, <PAD>, 'London']` and the last token is not a `<PAD>` token from which output logits it can be sampled from. <|||||>@patrickvonplaten I think from your above example, the past variable will be `(2, 1, 16, num_tokens, 64)` instead of `(2, 5, 16, num_tokens, 64`. Right?

> So I think that if I wanted to exclude the last sentence from the last "past" I can simply manipulate the 2nd dimension in all the 24 items and drop the corresponding value from it. I guess... ?

What I meant here is to manipulate the past tensors in case the tensor dimension I want to predict  for the subsequent batches should exclude some sentence. I found removing information from the past tensors may be complicated. So please see my example above with another approach to use "past" I am experimenting now.

For now to simplify this a bit, I'm not focusing on padding the past. Thus, it'll be generated from a batch of N docs. For instance below N=3:
```
['I', 'live', 'in', 'NYC']
['I', 'live', 'in', 'Barcelona']
['I', 'live', 'in', 'Moscow']
```
Note: I'm going to make sure to group docs by tokens length. I'm going to get a `past` from that. Past here will be a list of 24 elements with dimensions `(2, 3, 16, 4, 64)`

Then I'm planning to use that past variable along with M suffix phrases. Those suffix phrases may have different lengths and below to different sentences that were calculated above, so I'm planning to add padding here first. Also another characteristic is that M will be always equal to or greater than N. For example, M=6 (it's a coincide that num_tokens is also 6 here):
```
['and', 'I', 'speak',  'english',   'fluently', '<PAD>']
['and', 'I', 'live',   'in',       'North',     'Manhattan']
['and', 'I', 'like',   'football', '<PAD>',     '<PAD>']
['and', 'I', 'don\'t', 'speak',    'catalan',   '<PAD>']
['and', 'I', 'take',   'the',      'bus',       '<PAD>']
['and', 'I', 'drink',  'vodka',    'sometimes', 'you?']
```
Note: I'm also building an attention mask properly as you indicated by concatenating a tensor full with 1s with length = 4 to make this work. For example:
```
['1', '1', '1', '1', '1', '1', '1', '1', '1', '0']
['1', '1', '1', '1', '1', '1', '1', '1', '1', '1']
['1', '1', '1', '1', '1', '1', '1', '1', '0', '0']
['1', '1', '1', '1', '1', '1', '1', '1', '1', '0']
['1', '1', '1', '1', '1', '1', '1', '1', '1', '0'] 
['1', '1', '1', '1', '1', '1', '1', '1', '1', '1']
```

To make "past" fit here, I'm planning to expand the past variable. So for each tensor in the past array, expand the tensor results as if I had sent to gpt2 initially the following first batch to get the "past" for:
```
['I', 'live', 'in', 'NYC']
['I', 'live', 'in', 'NYC']
['I', 'live', 'in', 'NYC']
['I', 'live', 'in', 'Barcelona']
['I', 'live', 'in', 'Barcelona']
['I', 'live', 'in', 'Moscow']
```


In this case, I'll build a past with dimensions: `(2, 6, 16, 4, 64)` instead of the original dimensions: `(2, 3, 16, 4, 64)`. I found https://pytorch.org/docs/stable/torch.html#torch.repeat_interleave very useful for this...

Do you think this make sense? Any warning about this? Thanks<|||||>@patrickvonplaten can I not manipulate the `past` variable as explained above? Do the other dimensions contain some kind of aggregated data that makes the past to be immutable?<|||||>@patrickvonplaten just to confirm I'm on the right track...: Can I manipulate the dimensions for the layer tensors in the past array? It's working for me with latest release (2.6.0), but just wanted to make sure I'm on the right track if I go ahead with this. So basically I want to make sure I can expand the dimension 1 of every tensor in the past array from N to M. So that I can re-use that past with different predictions that will reuse that past. Hope it makes sense my question. 