finetune_trainer with python -m torch.distributed.launch
Hi
I need to run the finetune_trainer with multiple gpus, I am getting the error of 

"Default process group is not initialized"
AssertionError: Default process group is not initialized

I am using custom dataloader, might be hard to share all the parts of codes, but I defined sampler as DistributedSampler. 
this is transformer 3.5.1, python 3.7 on GPU.

thanks
Best
Rabeeh 


Also, could you add the command to run distributed training with GPUs with finetune_trainer in README? thanks <|||||>I tried with latest version of transformers on 4 gpu with distributed training 

rabeeh@gpu4:~/transformers/examples/seq2seq$ python -m torch.distributed.launch finetune.py  --learning_rate=3e-5     --fp16     --gpus 4     --do_train     --do_predict     --n_val 1000     --val_check_interval 0.1     --data_dir wmt_en_ro     --train_batch_size=1     --eval_batch_size=1     --output_dir=xsum_results     --num_train_epochs 1     --model_name_or_path t5-smal

getting the following error, thanks 

finetune.py: error: unrecognized arguments: --local_rank=0
Traceback (most recent call last):
  File "/opt/conda/envs/transformers/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/transformers/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/transformers/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/transformers/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/transformers/bin/python', '-u', 'finetune.py', '--local_rank=0', '--learning_rate=3e-5', '--fp16', '--gpus', '4', '--do_train', '--do_predict', '--n_val', '1000', '--val_check_interval', '0.1', '--data_dir', 'wmt_en_ro', '--train_batch_size=1', '--eval_batch_size=1', '--output_dir=xsum_results', '--num_train_epochs', '1', '--model_name_or_path', 't(trans(tr(transformers) 

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@rabeehk I also encountered the same problem as you. Have you solved it?