Solved by albanD in post #4 
              
              
                Given the error and how it changes with update_episode, it looks like numerical precision errors. 
Do you fix your random seed? Does this trend of one training and one not training as well is the same for many different random seed? 
Even with two different random seeds and the exact same code, espe…