If you don’t need to update this tensor (i.e. no gradients should be calculated for it), you should define it as a buffer via: self.register_buffer('average', torch.tensor(...)). This would make sure that nn.DataParallel will push this buffer to the corresponding device. In your forward method you can access it via self.average.