Seems strange. Cannot help at all but if I had to I would recommend to check where the model parameters are stored. Sayin’ that cos computations usually requires to be in the same gpu. Using dataparallel model parameters are allocated in a single gpu, therefore working this way may produce an unseen bug. Anyway you can try to use a toy model to see if you can emulate the same behaviour.