ProphetNet generation inconsistent with batch size?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

It says to ping @patrickvonplaten about ProphetNet


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python

from transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration

checkpoint = 'microsoft/prophetnet-large-uncased-squad-qg'

model = ProphetNetForConditionalGeneration.from_pretrained(checkpoint)
tokenizer = ProphetNetTokenizer.from_pretrained(checkpoint)

fact1 = "Bill Gates [SEP] Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975."
fact2 = "42 [SEP] 42 is the answer to life, the universe and everything."
fact3 = "attention [SEP] A transformer is a deep learning model that adopts the mechanism of attention, differentially weighing the significance of each part of the input data."

def try_variable_batches(facts):
    inputs = tokenizer(facts, padding=True, truncation=True, return_tensors="pt")
    
    question_ids = model.generate(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask'], num_beams=5)
    
    return tokenizer.batch_decode(question_ids, skip_special_tokens=True)

print(try_variable_batches([fact1, fact2, fact3]))
''' 
['who founded microsoft?',
 'what is the answer to life, the universe and everything?',
 'what mechanism does a transformer adopt?']
'''

print(try_variable_batches([fact1]))
# ['along with paul allen, who founded microsoft?']

print(try_variable_batches([fact2]))
# ['what is the answer to life, the universe and everything?']

print(try_variable_batches([fact3]))
# ['a transformer adopts the mechanism of what?']
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I'm expecting that batching shouldn't change the results. I tried with another QG model and got consistent results with different batch sizes. If it helps at all, based on other experiments the performance of the model seems best with a batch size of 1.
Hey @deklanw,

Thanks a lot for the very clean issue report! This is indeed a problem and we should try to fix it. 

At the moment, I'm a bit overwhelmed with issues, so would you maybe be interested in working together to solve the issue? I suspect that the problem actually lies in the froward pass of Prophetnet, so here is what we could/should do:

1. Create a test that verifies that the encoder attention mask works correctly in the forward pass:
- Pass a batch_size=1, no padded input to `model(input_ids=input_ids_single, attention_mask=None, decoder_input_ids=torch.ones((1,)))` vs `model(input_ids=input_ids_batched, attention_mask=attention_mask, decoder_input_ids=torch.ones((input_ids_batch.shape[0], 1))`

Here we should first check if the encoder outputs are exactly the same and then whether the decoder outputs are the same.

Would you be interested in opening a PR where such a test is added and that would help us see where the problem could be? :-)<|||||>If you don't find any time - don't worry, I'll try to allocate some in ~1,2 weeks for this<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.