Hi @enjlee and @coincheung
Sorry for my late reply. I wrote a blog post sometime ago, that may shed some light on this issue. The blog post explains how to setup not only data parallel on single machine, but also separate a large model across multiple GPUs, and multiple replica on different nodes, and train them in a distributed fashion. In other words, each process group contains more than one GPU.
It might be an overkill, but I think it may better explain the underlining mechanism of how pytorch distributed works.
Please let me know if you have any questions after reading the blog post.

  
      My New Hugo Site – 12 Dec 19
  
  
    

Pytorch Distributed Data Parallel With Model Parallel in an HPC Environment 111

Distributed Data Parallel with Model Parallel in an HPC environment Objective This tutorial is on :
   how to separate a model and put it on multiple GPUs.    how to train such model in a distributed data parallel fashion.    how to use...