So I found one error, which is that my network takes concat[s, a] as an argument and outputs a scalar Q. I’ve literally followed the stanford theory lecture, why can’t people indicate the implementation in their damn math formulas?
So this brings me to the question, how do we extract the right Q value in regards to keeping the gradient flow without problems?
Just “Q[a]” , or something like “torch.dot(Q, one_hot(a))”?
Same goes for this part of double DQN:

Should we propagate the gradient of the Q network, which is nested inside the Q_target network?