Andreas_Georgiou:

If so, why? .to will create a new tensor that is no longer a leaf tensor from what I understand, is that correct?


Yes



 Andreas_Georgiou:

Is the behaviour different for nn.Parameter and nn.Module?


Yes, nn.Module does some peculiar things to manage nn.Parameters and their movement. You may check source code for details.



 Andreas_Georgiou:

What about model = model.set_devices() ? Does that yield different results?


Nope, as model is never cloned in your code.
PS in general, self.param.data = self.param.data.to(device) does the trick