Yes, that’s the case since nn.DataParallel will create new model replicas in each iteration and thus when you store the state_dict outside of the forward or backward pass, there should only be a single model on the initially specified device.