I have a module that is saving a value during a backward hook. Some math is done to grad_output, and then it is saved as self.Val = value in the backward hook function.  With a single GPU this works fine.  But when I run on multiple GPUs it tells me object has no attrinute ‘Val’.  Specifically val should be keeping track of an average gradient over time.  When I tried tracking this in the backward hook function itself it was giving me RuntimeError: binary_op(): expected both inputs to be on same device, but input a is on cuda:0 and input b is on cuda:1.  Now I am trying to just save the gradient and perform the math later but that does not work with two GPUs.