stuck in the evaluation_loop of trainer.py when training
### System Info

- Ubuntu: 20.04.5
- GPU: A800(80G) x 8
- CUDA: 11.7
- NCCL: 2.14.3
- python: 3.9.16
- deepspeed: 0.9.5
- transformers: 4.30.2

### Who can help?

@sgugger


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Problem Description:
I'm doing a ***LLM pre-train***. The dataset is ready. The class dataset is my own implementation.

The base model if from the configuration file: LLAMA 7B huggingface
And other parameters are:
* batch_size: 2
* gradient_accumulation_steps: 2
* eval_batch_size: 1
* eval_steps: 12
* save_steps: 12

Additionally, ds_config is:
```
"zero_optimization": {
     "stage": 1,
     "offload_param": {
       "device": "cpu"
     },
     "offload_optimizer": {
       "device": "cpu"
     },
     "allgather_partitions": true,
     "allgather_bucket_size": 5e8,
     "contiguous_gradients": true,
     "reduce_bucket_size": 5e8,
     "overlap_comm": true
   },
   "optimizer": {
     "type": "AdamW",
     "params": {
       "lr": 1e-05,
       "betas": "auto",
       "eps": 1e-08
     }
   },
```

### Question:
***When I start training from beginning to step=12, according to my settings, the first evaluate should be triggered. At this time, everything is normal: the train is normal, and the evaluate is triggered.
But evaluate never ends.***

I logged in my dataset class and saw that the first 12 train steps ended normally. And, the first batch val data has been returned from my \_\_getitem\_\_ to the trainer of huggingface, and then it gets stuck. No return, no more information.  No re-entry into the val dataset ( i implement it and no next \_\_getitem\_\_ called ).


### Running stack
At this point, my cpu and gpu are fully loaded.
| NVIDIA-SMI 515.86.01 |   Driver Version: 515.86.01 |   CUDA Version: 11.7     |
|--|--|--|
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. MIG M.|
|---|---|---|
|   0  NVIDIA A800 80G...  On    N/A   47C    P0    76W / 300W | 58951MiB / 81920MiB|  100%      Default  Disabled |
|   1  NVIDIA A800 80G...  On  N/A   48C    P0    74W / 300W | 59001MiB / 81920MiB|    100%      Default  Disabled  |
|   2  NVIDIA A800 80G...  On  N/A   47C    P0    72W / 300W  | 58999MiB / 81920MiB |  100%      Default  Disabled |
|   3  NVIDIA A800 80G...  On  N/A   43C    P0    69W / 300W | 58953MiB / 81920MiB| 100%      Default  Disabled |
|   4  NVIDIA A800 80G...  On  N/A   43C    P0    71W / 300W | 58953MiB / 81920MiB |  100%      Default  Disabled |
|   5  NVIDIA A800 80G...  On  N/A   44C    P0    70W / 300W  | 58999MiB / 81920MiB | 100%      Default  Disabled |
|-------------------------------|----------------------|----------------------|
```
MiB Mem : 257598.2 total, 144592.4 free,  65631.8 used,  47374.0 buff/cache
MiB Swap: 532480.0 total, 531717.9 free,    762.0 used. 144353.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2335083 root      20   0  111.9g  12.9g   5.5g R 200.0   5.1 179:15.41 python3.9
2335101 root      20   0  111.7g  13.3g   5.9g R 199.7   5.3 178:57.67 python3.9
2335097 root      20   0  111.7g  13.3g   5.9g R 100.3   5.3  94:33.70 python3.9
2335099 root      20   0  111.7g  13.1g   5.7g R 100.3   5.2  95:05.42 python3.9
2335091 root      20   0  111.7g  13.1g   5.8g R 100.0   5.2  94:48.45 python3.9
2335095 root      20   0  111.7g  13.1g   5.7g R 100.0   5.2  95:00.15 python3.9
2335098 root      20   0  111.6g  13.1g   5.7g R 100.0   5.2  94:45.88 python3.9
2335096 root      20   0  111.7g  13.2g   5.8g R  99.7   5.2  94:39.61 python3.9
```

I figured out a way to print out all stacks of all active thread :

```

Printing stack
-- Thread ID: 140306341001024 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(


Printing stack
-- Thread ID: 140281920169792 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)


-- Thread ID: 140275561256704 ---
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()


Printing stack
-- Thread ID: 139870071547712 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()



-- Thread ID: 140452146153280 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()



Printing stack
-- Thread ID: 139800833808192 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)




-- Thread ID: 139794038388480 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/DebugStack.py", line 17, in _ThreadPrintStack
traceback.print_stack(frame)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 937, in _bootstrap
self._bootstrap_inner()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/tensorboard/summary/writer/event_file_writer.py", line 244, in run
self._run()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/tensorboard/summary/writer/event_file_writer.py", line 269, in _ru
data = self._queue.get(True, queue_wait_duration)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/queue.py", line 180, in get
self.not_empty.wait(remaining)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 316, in wait
gotit = waiter.acquire(True, timeout)



-- Thread ID: 139793899173632 ---
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 937, in _bootstrap
self._bootstrap_inner()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/tqdm/_monitor.py", line 60, in run
self.was_killed.wait(self.sleep_interval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 581, in wait
signaled = self._cond.wait(timeout)




File "/root/mambaforge/envs/trlx_env/lib/python3.9/threading.py", line 316, in wait
gotit = waiter.acquire(True, timeout)
Printing stack
-- Thread ID: 140320438421312 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()




Printing stack
-- Thread ID: 140180603557696 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)



-- Thread ID: 140174133536512 ---
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()



Printing stack
-- Thread ID: 139808714364736 ---
File "/data/work/ChatGPT/trlx_rlhf/sft_fullpretrain/./train_gptj_summarize.py", line 281, in <module>
trainer.train()
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 1645, in train
return inner_training_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2020, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 2321, in _maybe_log_save_evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3053, in evaluate
output = eval_loop(
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3266, in evaluation_loop
logits = self._pad_across_processes(logits)
File "/root/mambaforge/envs/trlx_env/lib/python3.9/site-packages/transformers/trainer.py", line 3410, in _pad_across_processes
sizes = self._nested_gather(size).cpu()
```

### What I have tried
Also, I tried:
1. Change model (change to gpt-j 6b)
2. Change deepspeed stage (1, 2, 3)
3. Change batch_size (1, 2, 4)
None of them work, they are all stuck in the above-mentioned evaluate.  I need help.


### Expected behavior

I want evaluate to work properly. Don't get stuck.
We can't really help without knowing the code you are running.<|||||>@sgugger

I have located the problem statement:
```
trainer.py
def _pad_across_processes(self, tensor, pad_index=-100):
     ....
     # Gather all sizes
     size = torch.tensor(tensor.shape, device=tensor.device)[None]
     sizes = self._nested_gather(size).cpu()
     ....
```
Stuck at .cpu()
I disassembled self._nested_gather(size).cpu() and found that it was stuck in .cpu()
In the above statement, tensor.shape==[1, 4608, 106078],  tensor.device == 'cuda:0' .
Why is it stuck there?<|||||>You might have tensors of different sizes on your GPUs. This usually causes a hang when gathering.
Again, it's hard to know for sure when we don't know the code you are running :-)<|||||>Hi, I met similar problems with the latest Transformers, where training gets stuck in the evaluation_loop of trainer.py. @clinton81 Have you found any solution?<|||||>Downgrading Transformers to v4.19.4 works (same code, same data, same command).<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.