We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a finegrained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.
[Summary]
This paper empirically studies the behavior of deep policy gradient algorithms during the optimization. The conclusion is that, while these methods generally improve the policy, their behavior does not comply with the underlying theoretical framework. First, sample gradients obtained with a reasonable batch size have little correlation with each other and with the true gradient. Second, a larger batch size requires a smaller step-size. Third, the value baseline is far from true values and only marginally reduces variance, yet it considerably helps with optimization. Finally, the optimization landscape highly varies with the choice of objective function and the number of samples used to estimate it.
[Decision]
I vote for acceptance. To the best of my knowledge, the findings of this paper are new and not predictable by the current theory. These negative results have some merit as they call for theory that explains the behavior of these algorithms, or an algorithm whose behavior is predictable by the current theory. The paper is well-written, with a few small issues in presentation that should to be addressed in the final revision.
[Comments]
In Fig. 4 (b) it does not look like that the value error is high. It is said that "the learned value function is off by about 50% w.r.t. the underlying true value function." This sentence should be clarified or visualized.
What is \pi in Eq (13) in A1? If it is the agent's current policy, how is it different than \pi_\theta? If \pi corresponds to the distribution of state-action pairs in the replay buffer, how can one obtain a policy \pi that has led to this distribution of states in order to construct the importance sampling ratio?
In 2.2, the claim that a learned value baseline results in significant improvement in performance should be supported by results or reference to previous work.
Figs. 6 and 7 compare the loss surface with different objectives and sample regimes. Do these factors (objective and sample size) affect the part of the parameter space that is visualized (by changing the origin and the update direction), or are they only used to evaluate the values on the z-axis for the same area in the parameter space? Observing a different landscape in a different part of the parameter space is not surprising.
[Minor comments]
- Is V_\theta_{t-1} in Eq (4) a function of state? If so, a (s_t) is missing before the plus sign.