SageMaker Model Parallel: cluttered tensorboard plots
## Environment info

- `transformers` version: master
- Platform: SageMaker
- Python version: 3.6
- PyTorch version (GPU?): 1.6
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: Y

### Who can help


Models:All 

Library:SageMaker Model parallel


## Information


The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MLM


## Expected behavior

Currently, SM trainer class inherits `is_world_process_zero` variables from main trainer class. In main trainer class, these variables are derived using `self.args.local_rank` or  `dist.get_rank()`, which are not unique for SMMP. This cause multiple processes writing tensorboard summaries because of which there are loops in the tensorboard graph. The `is_world_process_zero` can be implemented in SageMakerTrainer as below. This makes sure that only single process is writing tensorboard summaries.

```python
    def is_world_process_zero(self) -> bool:
        """
        Whether or not this process is the global main process (when training in a distributed fashion on several
        machines, this is only going to be :obj:`True` for one process).
        """
        if self.is_model_parallel_enabled:
            return smp.rank() == 0 and smp.local_rank() == 0 and smp.mp_rank() == 0 and smp.dp_rank() == 0
        else:
            return super.is_world_process_zero()
```



Hello! Thank you for opening an issue and for offering a code sample!

Could you open a PR with your code changes?

Thank you!<|||||>@LysandreJik  Please find the PR here: https://github.com/huggingface/transformers/pull/10403/files <|||||>Cool, thanks for fixing! Just merged the PR.