Solved by richard in post #3 
              
              
                I don’t think you can rely on the hooks running deterministically. However, the list of device ids you passed into DataParallel (or the default [0, 1, 2, 3] assuming 4 GPUs) specifies the order in which your data gets split across devices. 
For example, for device ids [0, 1, 2, 3], the first quarter…