FileNotFoundError when running distributed Trainer
# 🐛 Bug

## Information

I'm running the language modeling example in distributed mode and am getting FileNotFoundError in the Trainer.

The error :
```
Traceback (most recent call last):
  File "run_language_modeling.py", line 292, in <module>
    main()
  File "run_language_modeling.py", line 257, in main
    trainer.train(model_path=model_path)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 451, in train
    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 327, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 212, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 193, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/home/shaoyent/transformers/examples/language-modeling/output/checkpoint-500/optimizer.pt'
```

The error seems to be the Trainer saving checkpoints in each [local master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L413) while the directory is created by [world master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L448) causing a race condition. 

I think the suggested method is to handle save checkpoints in world master, since weights should be synchronized after backprop. 

Yes, I think you're right. Relaunching training from a checkpoint would however require the user to copy the weights to other machines, right?<|||||>> Yes, I think you're right. Relaunching training from a checkpoint would however require the user to copy the weights to other machines, right?

Yes, loading from checkpoint can be done on all nodes.
<|||||>So I think we should save checkpoints and final models on each **local master**.

Do you agree?<|||||>Would there be a difference in checkpoints between each local master ?
If they're identical I'd think it be best to reduce write overhead and save only once (i.e world master)<|||||>No difference, but the issue is that you won't be able to restart training from a checkpoint (unless you copy it around) if we only save to world master<|||||>Ah, I see, thanks for the clarification. 
I'm assuming a shared file system so it's all saving and loading from the same file. 
I guess for other cluster configurations that makes sense, but care would have to be taken to avoid write/mkdir conflicts from multiple processes.<|||||>I see. Thinking about it more, yes, I think it like it if we only save from world_master. (even if it adds one step for some users). Thanks for your insight.