mean(1) needs to be replaced with sum(1); <–(performance improves significantly)
Also, I tried the KL in both directions on CartPole-v0 (discrete control env for which trpo helps a lot).
KL(avg || most_recent) updated more monotonically towards max reward than KL(most_recent || avg).
Once at max reward, KL(most_recent || avg) hovers closer to max_reward but has collapses that are more catastrophic than KL(avg || most_recent)