On another note, when you said that the full gradient is computed anyways, does that mean that when I apply operations on a Variable (with requires_grad=True) the gradient gets computed as well ?

I meant it’s computed during backward() and then the co-ordinates you request are narrowed down. (rather than computing only the co-ordinates you desire).