Thanks for detailed info.
Having deep insight, we know models are replicated in DataParallel(), and the weights of both models are the same.
Where or in which code part, model weights are updated? ?(in parallel_apply() models are replicated)
Which model’s weight is saved in checkpoint? (GPU-0 model weights and GPU-1 model weights)
model present on GPU-0 weights saved in checkpoint OR model present on GPU-1 weights are saved in checkpoint?