@ptrblck,
In case of DataParallel().
If we use two GPUs, initially model is replicated on both GPUs, weights and bias are same on both GPUs and after calculation, weights are updated GPU-0 (default), but GPU-1 holds the same updated value?