That makes sense. To be more concrete about my implementation: I’m running my agent, storing each state, action, reward, etc. tuple in a replay buffer, then after each episode sampling a mini-batch of 32 (a somewhat arbitrary choice) such tuples to compute the terms inside the expectation in the image above, then backpropagating to minimise the loss L. The parameters theta are then updated on a slower timescale (or outer loop) using x.
My understanding is that the Thomas (2014) pseudocode in algorithm 2 effectively uses a mini-batch of size one at every time-step, though with a large number of time-steps this seems like it would be less efficient than my current implementation? For reference/clarity, line 13 in Thomas (2014) contains the update that I am struggling with, where the author uses w instead of x, and also uses eligibility traces e (other than that it can be seen that the gradient update is simply that of L as given above).
The exact maths is only the loss function above and the related pseudocode/derivation in Thomas (2014), so not really that advanced! I suppose my issue is that something that seems so mathematically simple is so slow in my implementation, but this could be because I’m not appreciating some of the subtleties of Pytorch and automatic differentiation more generally.