bug in distributed codes AssertionError: Default process group is not initialized
Hi
I am using transformers 3.5.1, on distributed fashion on multiple gpus with pytorch 1.6 and python=3.7 
I am running:

python -m torch.distributed.launch  --nproc_per_node=1 --nnodes=2         --node_rank=1 --master_addr=$host --master_port=$port finetune_trainer.py config.json 

Huggingface codes only work wiht distributed fashion when all gpus are relying on one machine.
If user wants to run two copy of codes on two machines, since in the codes it decides based on local_rank not rank, the local_rank for bot copies would be zero. 

could you have a look please?
thanks 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.