Initializing ddp is extremely slow when finetuning RAG
Hi, when I am finetuning the RAG model, it seems that the DDP process is extremely slow. I waited 1 day but still did not see the training process.


loading file None
loading file None
loading file /nfs/users/s_xiangru/model/rag/rag/rag-sequence-nq/generator_tokenizer/special_tokens_map.json
loading file /nfs/users/s_xiangru/model/rag/rag/rag-sequence-nq/generator_tokenizer/tokenizer_config.json
Global seed set to 42
Global seed set to 42
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Using native 16bit precision.
Global seed set to 42
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Using native 16bit precision.
Global seed set to 42
INFO:__main__:Custom init_ddp_connection.
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4
INFO:__main__:Custom init_ddp_connection.
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.