It’s hard to tell what’s creating the issue without seeing the code, but generally you are right. nn.DataParallel should take care of pushing the parameters and buffers to the right devices. However, if  self.average was defined as at tensor, you would run into this error.