DataParallel will move all properly registered buffers, parameters, and submodules to the devices.
If you are creating new tensors in the forward DataParallel won’t be aware of them and you would need to make sure to push these new tensors to the corresponding device as given in the previous post.