I looked into how autograd works. The graph tracks which all operations led to the loss calculation and then when we do loss.backprop() the gradients for all the parameters that influenced loss calculation are calculated. I don’t exactly know how the operation history is stored under the hood. If I can share the operation history with other device then may be it might work. Right!?