Hi, unless an explicit synchronization (such as with a device to host copy or torch.cuda.synchronize) is triggered, this sort of skew is possible due to the different GPUs, but such a significant skew (many epochs) seems unlikely. Are there any other jobs/processes running on your GPU that may slow this down and is the problem consistently reproducible?
As far as model.join() that API is to support different processes having different no. of inputs, is that your case here? If not, you would not need model.join().