I’m not really clear on what this means:
“As you can see below, we will need to query our network for Q parameters before we update the weights. In PyTorch every call to a Variable adds it to computational graph and will consider those for backward calcaulation which will affect your gradients.”
Backpropagation only happensi n the backwars diretion. So, I’m not really sure how adding additional forwards calcualtios in your graph will cahnge anything?  For example, let’s say you have:
in = autograd.Variable(torch.rand(1,3), requires_grad=True)
out = in * 3

So, you want to back-propagatte through out, to get in.grad. So you do eg:
out.backward(torch.rand(1,3))
(just using some dummy random nubmes for the grad output.
But before doing this you want to do:
a = in * 15 + out
As far as I can tell this is completely benign, wont change anything. What makes you feel this wont be the case? Can you come up with some very simple 3-5 line test case to demonstrate the issue you fear you might be seeing?