I think I totally understood the tutorial on using the following line of code to do data parallel.
model = nn.DataParallel(model) 

I tested it on my device with 2 GPUs and it worked. However, for a complicated model I recently wrote, it doesn’t truly train on both GPUs when I printed the ‘Inside’ ‘Outside’ debugging message.
I managed to figure out that it was because of the optimizers. To my purpose, I need to have two optimizers each contains some shared part and some unique part.
model = model.module
optimizer_1 = optim.Adam([{'params': model.shared.parameters()},
                              {'params': model.unique_a.parameters()}],
                             lr=5e-5
                             )
optimizer_2 = optim.Adam([{'params': model.encoder.parameters()},
                              {'params': model.unique_b.parameters()}],
                             lr=5e-5
                             )

I deleted anything related to optimizers and loss criterions and it worked just fine on 2 GPUs according to the printed debugging message. But whenever the optimizers and loss criterions are added, it can only be trained on one single GPU.
I wonder if there’s any suggestions on this to make it work?
Thanks!