Batching does not work with any XGLM model (simple script to reproduce issue included)
## Environment info

- `transformers` version: 4.17.0
- Platform: Linux-4.4.0-140-generic-x86_64-with-LinuxMint-18.1-serena
- Python version: 3.7.10
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.3.4 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@patrickvonplaten, @anton-l

Models:

- XGLM

## To reproduce

Here is a simple python script to reproduce the issue.  When using any XGLM model when I start using batching the behaviour changes and I get gibberish results often.  You can also reproduce this issue by padding an input text with a pad token and using attention mask 0 here because it is something to do with handling of attention mask.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
​from transformers import pipeline

# it fails with either of these models which are both xglm
MODEL = 'KoboldAI/fairseq-dense-125M'
MODEL = 'facebook/xglm-564M'
​
tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL)
​
text = 'What is the only '
​
p = pipeline('text-generation', model=model, tokenizer=tokenizer)
​
print('expected')
print(p(text, do_sample=False, return_full_text=False, max_new_tokens=64))
​
print('without batching it works as expected and gives same response as above')
print(p([text] * 2 + ['some random text which is longer than the other'], do_sample=False, return_full_text=False, max_new_tokens=64))
​
print('with batching its broken and output is different even though the operation is deterministic')
print(p([text] * 2 + ['some random text which is longer than the other'], do_sample=False, return_full_text=False, max_new_tokens=64, batch_size=3))
```
Super keen to resolve this issue and am willing to contribute if given suggestions.<|||||>cc @patil-suraj <|||||>Hey @ri938 ! This is because the inputs are padded to right by default, but for batch generation the padding side should be left. If you set `tokenizer.padding_side = "left"` then it should work as expected.<|||||>Thanks this indeed did fix the issue with the facebook model but not the kobold model. I guess in this case its an issue with the kobold model and I shoud raise the issue with the provider of this instead.<|||||>Can you provide some Q&A  demo for the xglm model,Thanks<|||||>@htthYjh XGLM is an auto-regressive model like (GPT-2, 3) so you could try to do QA by providing some QA prompts and then feeding a que and asking it to generate the answer.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.