Solved by wwiiiii in post #2 
              
              
                Move module to device x is same as move all the parameters and buffers moved to that device. So two lines mentioned in the question are sufficient to guarantee each module is replicated to different GPU.