As mentioned in the DataParallel doc 4: " The parallelized module must have its parameters and buffers on device_ids[0] before running this  DataParallel  4 module."
I can’t find any reference to DataParallel in the repo so not sure where you do that. By I was talking about the place where you wrap your module in DataParallel.