Solved by mrshenli in post #2 
              
              
                The following method is called by DataParallel to create replicas. So the attributes in __dict__ should be replicated as well. But as the flow is “DataParallel forward” -> “replicate models” -> “app model forward”, you need to make sure that the mode is set properly before calling DataParallel forwa…