If you want to keep what we call buffer tensors, you can as well. Simply save them into self.my_buffer = torch.rand(2,3) in the __init__. They will be moved to the gpu the same way as the parameters when you call .cuda() on the module.
If you want to do this during the forward pass, you can give device=input.device as a keyword argument to most function to create tensors (like rand, zeros, ones…).