yes, I verified with the experimen.
If we use two GPUs to train model in DataParallel(), Which portion of the saved data is identical between GPU 0 and GPU 1? weights, gradients, LR, etc the same across 2 GPUs?
I think, saved data from only GPU-0 (not from GPU-2), is it correct?