I’m confused as to why the kl divergence (model_avg || current_model) is calculated as kl = current_distribution * (current_distribution.log() - avg_distribution.log()). It seems to be the correct way in that it yields better performance (on pong at least), but i don’t understand why the kl is calculated in the opposite direction of that which the notation (and chainer implementation) describes. Is it because k (the gradient of the kl) is later transposed in the notation before being multiplied times g and calculating the kl divergence in the opposite direction can shortcut/skip/equate the transpose step?
I think I’ll try testing both versions (directions of kl) on RoboschoolReacher-v0. Performance on that environment is supposed to change the most drastically from TRPO.