Solved by ptrblck in post #2 
              
              
                Are you using a data parallel approach (e.g. via nn.DataParallel or DistributedDataParallel)? 
If so, then the output would be expected as the model is executed on each device.