Solved by ptrblck in post #2 
              
              
                This might be the case, as you would have to scatter and gather the results for each model. 
A quick check would be to wrap all models in an nn.Sequential container and pass this to nn.DataParallel. 
We generally recommend to use DistributedDataParallel with a single process per device, as it would…