In the tutorials, it mentions nothing about training (ie: no loss function, loss.backward(), optimizer.step() ).
When the model is converted to a DataParallel model, does the backprop get seamlessly handled behind the scenes? I guess the same gradients would be passed to all instances of the model across gpus, is that right? Is there another mechanism to synchronise the model instances across gpus?