longt5 error in step 13 when torch.distributed.launch
### System Info

Hardware:
6 Quadro RTX 6000 and 2 A100-40GB gpus, but I only used 2 A100-40GB gpus for this task.

Env:
transformers==4.20.1
torch==1.11.0+cu113

### Who can help?

@patrickvonplaten  @stancld 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

[@Stancld](https://huggingface.co/Stancld)
When ddp, I encountered the following error when I try to run your code in the middle of an epoch. The task is pubmed-summarization and I used [run_summarization.py](https://github.com/huggingface/transformers/blob/v4.20.1/examples/pytorch/summarization/run_summarization.py). Is there anything particular about step 13? Any suggestion for solving this error?

```
1%|█ | 13/1872 [16:51<43:45:35, 84.74s/it]Traceback (most recent call last):
File "run_summarization.py", line 737, in
main()
File "run_summarization.py", line 656, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/transformers/trainer.py", line 1409, in train
return inner_training_loop(
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/transformers/trainer.py", line 1649, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/transformers/trainer.py", line 2345, in training_step
loss = self.compute_loss(model, inputs)
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/transformers/trainer.py", line 2377, in compute_loss
outputs = model(**inputs)
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/miniconda3/envs/t5long/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 947, in forward
if torch.is_grad_enabled() and self.reducer._rebuild_buckets():

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel, and by
making sure all forward function outputs participate in calculating loss.
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 6
```


To Reproduce: almost same as [here](https://huggingface.co/Stancld/longt5-tglobal-large-16384-pubmed-3k_steps) and your [wb](https://wandb.ai/stancld/LongT5/runs/1lwncl8a/overview?workspace=user-stancld).
```
CUDA_VISIBLE_DEVICES=4,5 python -m torch.distributed.launch --nproc_per_node 2 --master_port 56666 run_summarization.py
--model_name_or_path Stancld/longt5-tglobal-large-16384-pubmed-3k_steps
--do_train --do_eval --do_predict
--dataset_name ccdv/pubmed-summarization
--max_source_length 16384 --max_target_length 512
--per_device_train_batch_size 1 --gradient_accumulation_steps 64
--optim adafactor --learning_rate 0.001 --lr_scheduler_type constant --num_train_epochs 1 --gradient_checkpointing
--bf16=True --per_device_eval_batch_size 2 --predict_with_generate --generation_num_beams 1 --generation_max_length 512
--output_dir ./tmp/longt5_pubmed --run_name LongT5-pubmed-16k-512-bs_128 --report_to all
--logging_steps 100 --eval_steps 2000 --evaluation_strategy steps --ddp_find_unused_parameters=False --no_cuda=False
```

Here is the failed [wandb](https://wandb.ai/whaleloops/pubmed_sum/runs/19h5mp66/overview?workspace=).

I tried to run with 1 GPU, and it works for 50+ steps without the error above. I also verified that ddp works for LED.

### Expected behavior

 ddp training like 1GPU training without error above
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @whaleloops and sorry for the late response. There exists a hotfix if you filter out sequences shorter than 16 tokens (to avoid examples with empty tglobal attn).

Lemme know if it helps. :)

Unfortunately, I don't have enough spare time to look at more proper fix and send PR:/<|||||>That's a tricky one it seems @whaleloops ! An alternative to filtering out short sequences could also be to always pad until max length

<|||||>Thanks @stancld , I confirmed that issue resolved after hotfix.
Though this filter excludes a few examples
split before after
train 119924 117108
valid 6633 6631
test 6658 6658