'Default process group is not initialized' Error
I get this error when finetuning bert using code in `lm_finetuning/` folder, when I try to run it several GPUs. 
```
Traceback (most recent call last):
  File "finetune_on_pregenerated.py", line 330, in <module>
    main()
  File "finetune_on_pregenerated.py", line 323, in main
    if  n_gpu > 1 and torch.distributed.get_rank() == 0  or n_gpu <=1 :
  File "/opt/anaconda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 562, in get_rank
    _check_default_pg()
  File "/opt/anaconda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
```

sample training data(*epoch_0.json*):
```
{"tokens": ["[CLS]", "i", "'", "ve", "got", "the", "[MASK]", "scenario", "in", "mind", "do", "you", "[MASK]", "[MASK]", "##k", "[SEP]", "i", "prefer", "that", "over", "red", "##dit", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [6, 12, 13], "masked_lm_labels": ["perfect", "have", "ki"]}
{"tokens": ["[CLS]", "she", "message", "##d", "me", "suggesting", "i", "was", "ignorant", "because", "i", "[MASK]", "##t", "know", "the", "feeling", "and", "restriction", "that", "panties", "[MASK]", "on", "women", "[MASK]", ".", ".", ".", "seriously", ".", ".", "[MASK]", "panties", "she", "[MASK]", "[MASK]", "men", "don", "##t", "know", "how", "bad", "it", "is", "to", "wear", "panties", "because", "society", "[MASK]", "##t", "let", "women", "speak", "up", "about", "it", "[SEP]", "[MASK]", "yourself", "lucky", ".", ".", "[MASK]", "bullet", "dodged", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [11, 20, 22, 23, 30, 33, 34, 48, 57, 62], "masked_lm_labels": ["didn", "put", "women", ".", ".", "said", "that", "won", "consider", "."]}
{"tokens": ["[CLS]", "[MASK]", "enough", "my", "first", "name", "[MASK]", "actually", "lisa", "i", "[MASK]", "##t", "ha", "[MASK]", "minded", "[MASK]", "b", "##ds", "##m", "vi", "##ds", "at", "12", "13", "not", "made", "to", "do", "actual", "sex", "like", "u", "said", "but", "the", "[MASK]", "displayed", "play", "whipped", "on", "[MASK]", "vi", "##ds", "i", "think", "the", "pe", "##dos", "[MASK]", "'", "ve", "enjoyed", "watching", "[MASK]", "[SEP]", "this", "probably", "[MASK]", "[MASK]", "'", "t", "what", "[MASK]", "had", "in", "mind", "though", "sorry", "but", "i", "thought", "it", "funny", "when", "the", "first", "word", "was", "lisa", "an", "that", "'", "s", "my", "emil", "[SEP]"], "segment_ids": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "is_random_next": false, "masked_lm_positions": [1, 6, 10, 13, 15, 35, 40, 48, 53, 57, 58, 62, 84], "masked_lm_labels": ["funny", "is", "would", "##v", "doing", "being", "the", "would", "them", "is", "n", "u", "name"]}
```


*epoch_0_metrics.json*
```
{"num_training_examples": 3, "max_seq_len": 256}
```


Reproducing:
```
export CUDA_VISIBLE_DEVICES=6,7

python3 finetune_on_pregenerated.py --pregenerated_data training/ --bert_model bert-base-uncased --do_lower_case --output_dir finetuned_lm/ --epochs 1 --train_batch_size 16
```
The code works fine when training on one single GPU.

Thanks.
You can use `example/run_lm_finetuning` now, the scripts in the `example/lm_finetuning/` folder are deprecated (removed on master).<|||||>What kind input format is good for `example/run_lm_finetuning.py`?<|||||>Hi, there's an example using WikiText-2 in the [documentation](https://huggingface.co/pytorch-transformers/examples.html#language-model-fine-tuning). 

A file containing text is really all that's needed! You can change the way the file is used in `TextDataset` to better reflect the text you're fine-tuning the model to.<|||||>Thanks, Lysandre.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
