Hi, thanks for your reply.
I am still trying to get this absolutely right, so let me rephrase my question, jus to be absolutely sure that i am doing this right.
Model M takes an input s_t and gives an output Q_t. Q_t will be the output with which we want to calculate the loss.
Now we want to create the target vector, and since this is a Q-learning approach, we have to do some more forward passes.
Using Q_t, we can compute a_t, and with a_t, we can compute r_t and s_{t+1} (these are all game mechanics).
We can feed s_{t+1} to the network again, and acquire Q_{t+1}. This will be used to acquire a_{t+1}, and that will in turn be used to acquire r_{t+1} and s_{t+2}. Now we are almost there. We feed s_{t+2} to the network, and acquire Q_{t+2}. Now we run a_t through the network again to get a ‘copy’ of Q_t (call it Q_t’) and replace Q_t’[a_t] with r_t+theta*max(Q_{t+2}) (theta is a factor that discounts future rewards). We then compute the loss with loss_function(Q_t, Q_t’).
The tricky thing here is that both the output and the target vectors that are fed into the loss function are output of the same network. I only want the backward pass to run through the part of the graph that is resposible for producing Q_t.
Am i doing this right by wrapping everything except the first forward pass (where M takes input s_t and generates output Q_t) with torch.nograd()?
If this is the case then i can start training.
Kind regards,
Stijn