Regarding the answer to my second question. I actually don’t need the gradients for these intermediate variables. Can you please explain a bit more on what you mean by " However, if you have created a plain tensor and pushed it to a device via  to('cuda') , you would either have to use the  device  argument of another parameter or register it as an  nn.Parameter  (if it needs gradients) of buffer"
One solution to this problem I feel would be to define all my intermediate variables as nn.Parameters and make require_grad=False. Also, I can define them self.register_buffer and make grad=False. Will both solutions be similar?