RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Tokenizers: 0.8.1rc1
Python: 3.7.6
Pretrained Model: GPT2
Pretrained Tokenizer: GPT2

## Question
I'm getting the following error when I try to evaluate a model while it's training:
```python
wandb: Waiting for W&B process to finish, PID 242876
Traceback (most recent call last):
File "run_finetune_gpt2.py", line 163, in <module>
    main()
File "run_finetune_gpt2.py", line 150, in main
    trainer.train()
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 537, in train
    self.evaluate()
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 745, in evaluate
    output = self._prediction_loop(eval_dataloader, description="Evaluation")
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 838, in _prediction_loop
    preds = torch.cat((preds, logits.detach()), dim=0)
RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
```

I printed out the `preds` and `logits` shapes inside `trainer.py`:
```python
preds.shape = torch.Size([])
logits.shape = torch.Size([])
```
I can't recall exactly how my `TrainingArguments` object was set up but I think it was something like this:
```python
training_args = TrainingArguments(
         output_dir=output_dir,
         do_train=True,
         evaluate_during_training=True, 
         eval_steps=5, 
         logging_dir=logging_dir,
         per_device_train_batch_size=32,
         num_train_epochs=1,
         save_steps=100)

trainer = Trainer(
         model=model,
         args=training_args,
         train_dataset=sd_dataset_train,
         eval_dataset=sd_dataset_test, 
         data_collator=sd_data_collator)
```

I was using `GPT2LMHeadModel`. I tried to put together a code sample but couldn't reproduce the error. Has anyone run into this before when trying to evaluate while training?
## System Info
Debian 10
Pytorch: 1.6.0
Transformers: 3.0.2
Python: 3.7.8
Pretrained Model: AlbertPreTrainedModel (albert-base-v2)
Pretrained Tokenizer: AlbertTokenizer (albert-base-v2)

## Question
I'm getting the same error, but when trying to evaluate the model after training.
```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-7c7016f6f03e> in <module>
----> 1 res = trainer.evaluate(val_dataset)

/opt/conda/lib/python3.7/site-packages/transformers/trainer.py in evaluate(self, eval_dataset)
    743         eval_dataloader = self.get_eval_dataloader(eval_dataset)
    744 
--> 745         output = self._prediction_loop(eval_dataloader, description="Evaluation")
    746 
    747         self._log(output.metrics)

/opt/conda/lib/python3.7/site-packages/transformers/trainer.py in _prediction_loop(self, dataloader, description, prediction_loss_only)
    834                     preds = logits.detach()
    835                 else:
--> 836                     preds = torch.cat((preds, logits.detach()), dim=0)
    837                 if inputs.get("labels") is not None:
    838                     if label_ids is None:

RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
```
The values of `preds` and logits at this point are:
```
ipdb>  preds
tensor(0.4661, device='cuda:0')
ipdb>  logits
tensor(0.4578, device='cuda:0')
```
Replacing `torch.cat` with `torch.stack` seemed to do the job, is there a reason for using `torch.cat` here?
```
ipdb>  torch.stack((preds, logits.detach()), dim=0)
tensor([0.4661, 0.4578], device='cuda:0')
```
These are my training arguments and trainer:
```python
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=train_dataset,
    compute_metrics=compute_metrics,
)
```<|||||>Same issue<|||||>I am getting the same issue. CC: @sgugger<|||||>Same issue!<|||||>It's not that we don't want to fix that issue but no one as given us a reproducer and it was linked to a version of transformers that is now quite old. So please do let us know if the error persists on v3.5.1 (after upgrading transformers) and on which script.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Found the same issue on `transformers==3.4.0`. But after upgrading to `transformers==4.2.2` the problem fixed. FYI.<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.<|||||>Facing the same issue with > 4.2, now the issue is File "***/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 48, in torch_pad_and_concatenate
    if len(tensor1.shape) == 1 or tensor1.shape[1] == tensor2.shape[1]:
IndexError: tuple index out of range