Solved by ptrblck in post #2 
              
              
                I would recommend to scale down the problem first e.g. by removing apex (and potentially other utilities) and test if the DDP model is working as expected. 
Note that we also recommend to use the native automatic mixed precision training in the current master branch or from the nightly binaries. You…