Hi,
When you call .cuda() all the parameters and buffers of the module are moved to the gpu.
Parameters are everything that you saved as self.foo = nn.Parameter(args).
Buffers are any Tensor that you saved on self as self.bar = torch.rand(10).
All this is done  in the __setattr__ 54 functions of the Module if you want to check the implementation details.
If you create new tensors during the forward, it’s your responsability to create them with the right type. The set of functions torch.*_like(input) and input.new_*(new_tensor_size) can be used to easily create new tensors with the same type/device(/size) as other tensors.