Solved by mrshenli in post #2 
              
              
                I believe this is because DataParallel (DP) would replicate the module to each provided device in every forward. So the module instance stored in DataParallel and the module instance used by forward on every device are different. 
However, even if they are the same instance, it’s still problematic.…