Hi! Thanks for your reply!
My reset_train_data is pretty simple - it just creates new lists for storing logprobs of actions, values and rewards and also inits hidden state of GRUCell:
    def reset_train_data(self):
        self.hidden = torch.zeros(1, self.hidden_size)

        self.values = []
        self.logprobs = []
        self.rewards = []

Also  reset_train_data is called after each training episode in both cases, so in theory if it affects backprop, it should affect regardless of case.
Max element difference when using update_episode=0 is 0.0,  1.00000e-09 * 1.3970 when update_episode=2, and increases with update_episode growth.

But more important question is why this affects convergence so drastically? Take a look at this graph:

image.png703×315 15.2 KB

As you can see, two graphs begin to diverge about 1800 episode - and update_episode is only 10 (according to measurements error is about 2e-9)