Essentially yes, a single datum (state, action, reward, etc.) is collected at each time step t and stored in the buffer. Summing over terms then manually computing the gradient update to x (as opposed to forming the loss function explicitly and then backpropagating) might be the answer though; I hadn’t considered that! Thanks very much, I’ll try this out and see how I get on