That’s not how batch samples works. You need to sample a batch of samples simultaneously and update the actor/critic network by these samples. The loss of the actor is the mean of critic loss on these samples, not just decided by the batch_size. Hope I made my point, good luck!