I have met the same problem when I use multi-gpus to inference and train with torch.nn.DataParallel.
You can use this code to reproduce the problem.

  
      
      GitHub
  
  
    

pytorch/examples 4

A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples