could not run distribution cpu training on two CPU sockets using miprun
### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.8.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.0 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


![image](https://user-images.githubusercontent.com/36058628/172326302-8b98b2b0-9121-4e6b-be22-5a3798564909.png)
it does not train one model using two CPU sockets, but each instance trains its own model with below output
[INFO|trainer.py:1472] 2022-06-07 00:51:33,164 >>   Instantaneous batch size per device = 12
[INFO|trainer.py:1473] 2022-06-07 00:51:33,164 >>   Total train batch size (w. parallel, distributed
 & accumulation) = 12


### Expected behavior

```shell
should output following log if two cpu sockets are used for distributed training and the training performance should be doubled comparing with training with one CPU socket. 
[INFO|trainer.py:1472] 2022-06-07 00:51:33,164 >>   Instantaneous batch size per device = 12
[INFO|trainer.py:1473] 2022-06-07 00:51:33,164 >>   Total train batch size (w. parallel, distributed
 & accumulation) = 24
```
