run_clip.py RuntimeError
### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

Hi, @patil-suraj When I run `run_clip.py` following the steps in the [README](https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/README.md), I get an error like the following:

```
[INFO|trainer.py:2644] 2022-08-02 04:07:15,699 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-4500
[INFO|configuration_utils.py:446] 2022-08-02 04:07:15,701 >> Configuration saved in clip-roberta-finetuned/checkpoint-4500/config.json
[INFO|modeling_utils.py:1567] 2022-08-02 04:07:17,602 >> Model weights saved in clip-roberta-finetuned/checkpoint-4500/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
 33%|███████████████████████████████████████████████████████████████████████████████▉                                                                                                                                                                | 4623/13872 [1:56:27<3:50:22,  1.49s/it]Traceback (most recent call last):
  File "/home/gsj/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 537, in <module>
    main()
  File "/home/gsj/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 508, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1502, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1744, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 2474, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 2506, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
    return self.gather(outputs, self.output_device)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 181, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 78, in gather
    res = gather_map(outputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 69, in gather_map
    return type(out)((k, gather_map([d[k] for d in outputs]))
  File "<string>", line 10, in __init__
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/utils/generic.py", line 188, in __post_init__
    for element in iterator:
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 69, in <genexpr>
    return type(out)((k, gather_map([d[k] for d in outputs]))
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py", line 75, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/comm.py", line 235, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Input tensor at index 1 has invalid shape [4, 4], but expected [4, 5]
```

How to solve this error. Thanks!


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python run_clip.py

### Expected behavior

run run_clip.py success
Hey @gongshaojie12, thanks for the issue! Could you please provide the full command you run? cc @ydshieh; I believe you have worked with this script in the past<|||||>@gongshaojie12 Could you also provide a bit more information. For example, do you download and use the COCO dataset as in the README?<|||||>Hey @ydshieh @LysandreJik thank you very much for your replies.

My steps are as follows:

1,Create a `VisionTextDualEncoderModel`

```
from transformers import (
    VisionTextDualEncoderModel, 
    VisionTextDualEncoderProcessor, 
    AutoTokenizer, 
    AutoFeatureExtractor
)

model = VisionTextDualEncoderModel.from_vision_text_pretrained(
    "openai/clip-vit-base-patch32", "roberta-base"
)

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
feat_ext = AutoFeatureExtractor.from_pretrained("openai/clip-vit-base-patch32")
processor = VisionTextDualEncoderProcessor(feat_ext, tokenizer)

model.save_pretrained("clip-roberta")
processor.save_pretrained("clip-roberta")
```

2,Manually download COCO dataset to /home/gsj/data directory
![image](https://user-images.githubusercontent.com/6407116/183041989-726d1ec6-798d-48f0-bc1d-bf88fb10c59d.png)


3,Full run command:
```
python run_clip.py \
--output_dir clip-roberta-finetuned \    
--model_name_or_path clip-roberta/ \    
--data_dir /home/gsj/data \    
--dataset_name ydshieh/coco_dataset_script \    
--dataset_config_name=2017  \   
--image_column image_path  \   
--caption_column caption  \   
--remove_unused_columns=False  \   
--do_train  \
--do_eval   \  
--per_device_train_batch_size="64"   \  
--per_device_eval_batch_size="64"  \   
--learning_rate="5e-5" \
--warmup_steps="0" \
--weight_decay 0.1  \   
--overwrite_output_dir
```<|||||>In addition, I commented the line `image_transformations = torch.jit.script(image_transformations)` and added some `prints`. The complete `run_clip.py` is as follows:
[run_clip.zip](https://github.com/huggingface/transformers/files/9267204/run_clip.zip)
<|||||>Hi @gongshaojie12 , I have to change

```python
        train_dataset = dataset["train"][:2000]
```
to
```
        train_dataset = dataset["train"]
        data_args.max_train_samples = 2000
```
otherwise get an attribute error. (Ideally, we should specify this limits in the command line). Same for the validation dataset.

I am wondering if you face the issue when running on the whole dataset. With the limits of `2000` and `500` (that are in your script), I am not able to reproduce.<|||||>@gongshaojie12 I want to double check if you use multiple GPUs ??<|||||>> Hi @gongshaojie12 , I have to change
> 
> ```python
>         train_dataset = dataset["train"][:2000]
> ```
> 
> to
> 
> ```
>         train_dataset = dataset["train"]
>         data_args.max_train_samples = 2000
> ```
> 
> otherwise get an attribute error. (Ideally, we should specify this limits in the command line). Same for the validation dataset.
> 
> I am wondering if you face the issue when running on the whole dataset. With the limits of `2000` and `500` (that are in your script), I am not able to reproduce.

Hi @ydshieh thank you for your reply. Because the GPU machine is in the company, I can't run it on the whole dataset right now, when I come back to the company in two days I will run on the whole dataset,and feedback the results.

At the same time, after adding the code `data_args.max_train_samples = 2000`, I will also test whether it is running normally on my GPU machine<|||||>> @gongshaojie12 I want to double check if you use multiple GPUs ??

Hi @ydshieh ,yes, I used two GPUs for training<|||||>> Hi @gongshaojie12 , I have to change
> 
> ```python
>         train_dataset = dataset["train"][:2000]
> ```
> 
> to
> 
> ```
>         train_dataset = dataset["train"]
>         data_args.max_train_samples = 2000
> ```
> 
> otherwise get an attribute error. (Ideally, we should specify this limits in the command line). Same for the validation dataset.
> 
> I am wondering if you face the issue when running on the whole dataset. With the limits of `2000` and `500` (that are in your script), I am not able to reproduce.

Hi, @ydshieh When running on the whole dataset, I still get the following error:
```

[INFO|configuration_utils.py:446] 2022-08-07 22:36:34,789 >> Configuration saved in clip-roberta-finetuned/checkpoint-1500/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 22:36:36,673 >> Model weights saved in clip-roberta-finetuned/checkpoint-1500/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.5521, 'learning_rate': 4.2791234140715114e-05, 'epoch': 0.43}
 14%|█████▏                              | 2000/13872 [50:51<4:57:11,  1.50s/it][INFO|trainer.py:2644] 2022-08-07 22:49:16,799 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-2000
[INFO|configuration_utils.py:446] 2022-08-07 22:49:16,800 >> Configuration saved in clip-roberta-finetuned/checkpoint-2000/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 22:49:18,741 >> Model weights saved in clip-roberta-finetuned/checkpoint-2000/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.5047, 'learning_rate': 4.098904267589389e-05, 'epoch': 0.54}
 18%|██████▏                           | 2500/13872 [1:03:34<4:45:42,  1.51s/it][INFO|trainer.py:2644] 2022-08-07 23:01:59,622 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-2500
[INFO|configuration_utils.py:446] 2022-08-07 23:01:59,624 >> Configuration saved in clip-roberta-finetuned/checkpoint-2500/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 23:02:01,520 >> Model weights saved in clip-roberta-finetuned/checkpoint-2500/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.4655, 'learning_rate': 3.9186851211072664e-05, 'epoch': 0.65}        ^[[B^[[B^[[B
 22%|███████▎                          | 3000/13872 [1:16:13<4:34:09,  1.51s/it][INFO|trainer.py:2644] 2022-08-07 23:14:38,286 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-3000
[INFO|configuration_utils.py:446] 2022-08-07 23:14:38,287 >> Configuration saved in clip-roberta-finetuned/checkpoint-3000/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 23:14:40,239 >> Model weights saved in clip-roberta-finetuned/checkpoint-3000/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.4323, 'learning_rate': 3.7384659746251445e-05, 'epoch': 0.76}        ^[[B^[[B^[[B
 25%|████████▌                         | 3500/13872 [1:28:56<4:20:36,  1.51s/it][INFO|trainer.py:2644] 2022-08-07 23:27:21,056 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-3500
[INFO|configuration_utils.py:446] 2022-08-07 23:27:21,057 >> Configuration saved in clip-roberta-finetuned/checkpoint-3500/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 23:27:22,967 >> Model weights saved in clip-roberta-finetuned/checkpoint-3500/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.4047, 'learning_rate': 3.558246828143022e-05, 'epoch': 0.87}
 29%|█████████▊                        | 4000/13872 [1:41:37<4:07:33,  1.50s/it][INFO|trainer.py:2644] 2022-08-07 23:40:02,408 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-4000
[INFO|configuration_utils.py:446] 2022-08-07 23:40:02,409 >> Configuration saved in clip-roberta-finetuned/checkpoint-4000/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 23:40:04,339 >> Model weights saved in clip-roberta-finetuned/checkpoint-4000/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
{'loss': 0.3859, 'learning_rate': 3.3780276816608994e-05, 'epoch': 0.97}
 32%|███████████                       | 4500/13872 [1:54:19<3:55:46,  1.51s/it][INFO|trainer.py:2644] 2022-08-07 23:52:44,544 >> Saving model checkpoint to clip-roberta-finetuned/checkpoint-4500
[INFO|configuration_utils.py:446] 2022-08-07 23:52:44,546 >> Configuration saved in clip-roberta-finetuned/checkpoint-4500/config.json
[INFO|modeling_utils.py:1567] 2022-08-07 23:52:46,431 >> Model weights saved in clip-roberta-finetuned/checkpoint-4500/pytorch_model.bin
/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
 33%|███████████▎                      | 4623/13872 [1:57:33<3:51:51,  1.50s/it]Traceback (most recent call last):
  File "/home/gsj/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 539, in <module>
    main()
  File "/home/gsj/transformers/examples/pytorch/contrastive-image-text/run_clip.py", line 510, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1502, in train
    return inner_training_loop(
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 1744, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 2474, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/trainer.py", line 2506, in compute_loss
    outputs = model(**inputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
    return self.gather(outputs, self.output_device)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 181, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 78, in gather
    res = gather_map(outputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 69, in gather_map
    return type(out)((k, gather_map([d[k] for d in outputs]))
  File "<string>", line 10, in __init__
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/transformers/utils/generic.py", line 188, in __post_init__
    for element in iterator:
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 69, in <genexpr>
    return type(out)((k, gather_map([d[k] for d in outputs]))
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/_functions.py", line 75, in forward
    return comm.gather(inputs, ctx.dim, ctx.target_device)
  File "/root/anaconda3/envs/h-transformers/lib/python3.9/site-packages/torch/nn/parallel/comm.py", line 235, in gather
    return torch._C._gather(tensors, dim, destination)
RuntimeError: Input tensor at index 1 has invalid shape [4, 4], but expected [4, 5]
 33%|███████████▎                      | 4623/13872 [1:57:34<3:55:12,  1.53s/it]
You have new mail in /var/spool/mail/root

```

Also, when adding the code `data_args.max_train_samples = 2000`, it works fine
<|||||>Hi, it turns out that the last batch has only 9 examples, and it is splitted to a batch of `4` and another `5` elements (as we use 2 GPUs). This causes some issue for CLIP model. You can actually get the same issue very quickly by specifying
```python
--max_train_samples=137 --max_eval_samples=137
```
(remember to **remove the places of `2000` and `500` in your code first**)
Here `137 = 128 + 9 = 2 * 64 + 9` (so we have a complete batch and a remaining batch)

A quick solution is to add 
```
--dataloader_drop_last True
```

<|||||>Hi, @ydshieh I got it, thanks a lot!