Thanks.
Colloquially speaking the forward passes are “precalculated” in self.action_memory in my case.
Algorithm often calls the snippet below during an RL episode and the training loop in my question only takes place afterwards.
probabilities = F.softmax(self.policy.forward(observation))
action_probs = T.distributions.Categorical(probabilities)
action = action_probs.sample()
log_probs = action_probs.log_prob(action)
self.action_memory.append(log_probs)    

So in other words after one step(), the next backward() call doesn’t make much sense because the loss was calculated using old parameters? Am I understanding it correctly?
If so, one way to fix this error would be to execute forward passes in the loop as well, right?
Sorry if my train of thought (or wording) is confusing.