Sorry this thread is quite long and I don’t have it in my head.
But each “self” that you get on different devices are different. So any change you make to one of them will not be reflected on the others, or the original module.
But for a given Module, while running on its GPU, if you set a value in the forward, it will still be there in the forward.
Also as mentioned in the doc, register_backward_hook() is not working properly at the moment and should not be used.