Problem with GPT2/DistilGPT2 prediction - dimension mismatch
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Linux-4.18.0-193.el8.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: single gpu

### Who can help

Perhaps @patrickvonplaten, @LysandreJik could help?



## Information

Model I am using: GPT2/DistilGPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I'm able to train GPT2/DistilGPT2 successfully. However, during prediction, I consistently get the following error:
```
***** Running Prediction *****
  Num examples = 1922
  Batch size = 1
  0%|          | 0/1922 [00:00<?, ?it/s]Traceback (most recent call last):
  File "../../models/jigsaw/tr-3.4//run_puppets.py", line 283, in <module>
    main()
  File "../../models/jigsaw/tr-3.4//run_puppets.py", line 212, in main
    pred_results = trainer.predict(test_dataset = eval_dataset)  # call predict to get access to both metrics and predictions
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer.py", line 1287, in predict
    return self.prediction_loop(test_dataloader, description="Prediction")
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer.py", line 1353, in prediction_loop
    preds_host = logits if preds_host is None else nested_concat(preds_host, logits, dim=0)
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 47, in nested_concat
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 47, in <genexpr>
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 47, in nested_concat
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 47, in <genexpr>
    return type(tensors)(nested_concat(t, n, dim) for t, n in zip(tensors, new_tensors))
  File "/u/ioana/.conda/envs/tr34/lib/python3.8/site-packages/transformers/trainer_pt_utils.py", line 49, in nested_concat
    return torch.cat((tensors, new_tensors), dim=dim)
RuntimeError: Sizes of tensors must match except in dimension 3. Got 53 and 23

```

It doesn't seem to be a function of training (e.g., I've trained for 1-2-3 epochs, same results for prediction; the good news is that training seems to work perfectly, or at least, it seems to train and I get a model checkpoint). 

Any hunch on where I should start looking for a problem? I have no experience with these models. I checked other issues that were closed and some indicated that may be an attention problem. Thanks in advance!

I've used the same scripts successfully with 10+ different models, including GPT. 

PS: I may upgrade to the newer version of the library, but that requires some work on my side to update my code as sometimes the upgrades are not backward compatible... 
Maybe @sgugger has seen that error previously?<|||||>Yes, and it has been fixed... but only in the more recent versions of Transformers.<|||||>Excellent, thank you guys, I'll work on upgrading tomorrow.<|||||>If it can help you, we have a [migration guide from versions v3 to v4](https://huggingface.co/transformers/migration.html#migrating-from-transformers-v3-x-to-v4-x).

Please let us know if you run into any issues not described here!<|||||>will take a look, thank you! My tasks are similar to text classification in Glue, so I usually start from the sample code and modify it accordingly.<|||||>These are some different bits that I found by glancing at the new sample code to run GLUE. @LysandreJik 

- Columns in the train/dev/test datasets: Fill in the `task_to_keys` dictionary with appropriate column names 
- The scripts assume the datasets have a column called `label`; label_to_id needs attention
- Data processing/tokenization changed the api
- The dataset ingestion changed (the code is in there, though, on an else branch if the task is not a glue task)
- Metric computation also changed<|||||>It also looks like I have to install `pip install datasets` separately. I don't think I had to do that before.<|||||>Thank you for mentioning all of this. I believe the changes you're seeing are related to the example scripts, which are not static, and not related to the core of the library.

The changes related to the core of the library here would be those that were applied to the `Trainer`; did you manage to run your previous example script with the latest version, after updating it w.r.t the migration guide?<|||||>Correct. Training works, I have to fix a few things related to prediction & metrics computation. I have some home-brewed code that computes metrics for test (which is missing in the original sample code). I'll figure it out soon. So far, easier than expected. The data ingestion simplified a lot and I'm actually surprised it worked :D <|||||>I figured out the (one of the?) problem: some name collapse on my side. Fixed. Fingers crossed it works now. In any case, not too painful to upgrade. But now I expect backward compatibility 🥇  since most of the big pieces have gone through lots of refactoring. <|||||>I fixed all the problems and I managed to reproduce some old results with BERT-finetuned model. Thanks for your help!<|||||>Glad you could get it to work!<|||||>@LysandreJik It turns out I still have a problem with fine-tuning GPT with padding and batching. I have the following lines in my code:

```
    if tokenizer.pad_token is None:
            tokenizer.pad_token = tokenizer.eos_token
            config.pad_token_id = config.eos_token_id
```

which work fine for GPT2*. However, the padding fails in GPT. After some digging, I realized there is no eos_token so the statements above have no effect. 

This is the tokenizer configuration:
```
{"unk_token": "<unk>", "model_max_length": 512, "name_or_path": "openai-gpt"}
```

Any advice on how to fix this? I'd like to run GPT with padding & batching if at all possible. Thanks!<|||||>The open-ai GPT model has neither a pad, nor bos, nor eos token, which means that you will have to set them yourself. I'd advise to either set the `<unk_token>` to the EOS token:

```python
tokenizer.eos_token = tokenizer.unk_token
```

The other solutions is to add a special token before fine-tuning as explained here: https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_special_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_special_tokens<|||||>Thank you for this suggestion, I'll give it a try. 