is:issue is:open Parameters which did not receive grad for rank 5: encoder.block.0.layer.0.TransientGlobalSelfAttention.global_relative_attention_bias.weight 
### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

simply using the official example https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization_no_trainer.py
1. using 
2. export CUDA_LAUNCH_BLOCKING=1, TORCH_DISTRIBUTED_DEBUG=INFO
accelerate launch --config_file='./accelerate.yaml' run_summarization_notrainer.py  --seed=42 --preprocessing_num_workers=1  --weight_decay='0.001' --output_dir="arxiv_summarization/longt5/5_beam/" --per_device_train_batch_size=1 --per_device_eval_batch_size=1  --dataset_name='ccdv/arxiv-summarization'  --num_train_epochs=10  --model_name_or_path='google/long-t5-tglobal-base' --tokenizer_name='google/long-t5-tglobal-base'  --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch'

runing the script I got this error after runing over 12 examples

Parameters which did not receive grad for rank 5: encoder.block.0.layer.0.TransientGlobalSelfAttention.global_relative_attention_bias.weight
  
What is interesting that when I put the number of processes equal to 1 in accelerate.yaml

> compute_environment: LOCAL_MACHINE
> deepspeed_config: {}
> distributed_type: MULTI_GPU
> fsdp_config: {}
> machine_rank: 0
> main_process_ip: null
> main_process_port: null
> main_training_function: main
> mixed_precision: 'no'
> num_machines: 1
> num_processes: 1
> use_cpu: false
 The script runs normally but when I put it equal to 8 then I got this error after runing over 12 examples
### Expected behavior

script run normally
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>@sgugger Hi!
Iam trying to use longt5 for summarizing task 

I am using this [script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization#with-accelerate) 

and this [model](https://huggingface.co/google/long-t5-tglobal-base)


I am getting this error

> Traceback (most recent call last):
>   File "/cephfs/home/arij/Memory-transformer-with-hierarchical-attention_MLM/Summarization/run_summarization_notrainer-Copy1.py", line 947, in <module>
>     main()
>   File "/cephfs/home/arij/Summarization/run_summarization_notrainer-Copy1.py", line 821, in main
>     outputs = model(**batch)
>   File "/home/arij/anaconda3/envs/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
>     return forward_call(*input, **kwargs)
>   File "/home/arij/anaconda3/envs/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1026, in forward
>     if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
> RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
> making sure all `forward` function outputs participate in calculating loss. 
> If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
> Parameters which did not receive grad for rank 1: encoder.block.0.layer.0.TransientGlobalSelfAttention.global_relative_attention_bias.weight
> Parameter indices which did not receive grad for rank 1: 6


 I have tried many times to use it, but it does not work any hints?