What’s the whole updating old model right before you call backward and step about?

If I do it after the backward step or anywhere else, the old and new models will have same weights… I can also update the old model once, it is unclear in the paper if this update must be done once or at every steps.

Also are you asking for help on converting model to solve other environments or that model is not able to solve other environments?

According to the ppo paper 10, the set of hyperparameters I use in ppo.py should work in every environment with no modification at all. But only if I change both the batch size and the number of steps to 1000, it solves the InvertPendulum (quickly) and the InvertDoublePendulum (much slower than in the paper). With batch size = 64 and number of steps = 2048 as suggested by the paper, It solves nothing.
So I am quite certain there is something wrong somewhere in the training loop, but I can’t figure out what is wrong.