I also tried to change the number of npro_per-node=4 as following,
python -m torch.distributed.launch --nproc_per_node=4 --nnode=1 --node_rank=0 paraphrase_simpletransformers.py
Then, I did nvidia-smi, it becomes,
±----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  4925MiB |
|    0     85911      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |
|    0     85912      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |
|    0     85913      C   …e/anaconda3/envs/pytorch_p36/bin/python  3175MiB |
|    1     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1421MiB |
|    1     85911      C   …e/anaconda3/envs/pytorch_p36/bin/python  1393MiB |
|    2     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1431MiB |
|    2     85912      C   …e/anaconda3/envs/pytorch_p36/bin/python  1465MiB |
|    3     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |
|    3     85913      C   …e/anaconda3/envs/pytorch_p36/bin/python  1465MiB |
|    4     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1431MiB |
|    5     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |
|    6     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |
|    7     85910      C   …e/anaconda3/envs/pytorch_p36/bin/python  1433MiB |
±----------------------------------------------------------------------------+