Solved by ptrblck in post #4 
              
              
                Yes, nn.DataParallel will scatter the parameters in each forward pass, so it should work. 
This is also the reason for the lower performance compared to DDP, which has a reduced communication overhead.