Adaptive Batch Size for Safe Policy Gradients
Policy gradient methods are among the best Reinforcement Learning (RL) techniques to solve complex control problems. In real-world RL applications, it is common to have a good initial policy whose performance needs to be improved and it may not be acceptable to try bad policies during the learning process. Although several methods for choosing the step size exist, research paid less attention to determine the batch size, that is the number of samples used to estimate the gradient direction for each update of the policy parameters. In this paper, we propose a set of methods to jointly optimize the step and the batch sizes that guarantee (with high probability) to improve the policy performance after each update. Besides providing theoretical guarantees, we show numerical simulations to analyse the behaviour of our methods.
Summary:
This paper derives conditions for guaranteed improvement when using policy gradient methods. These conditions are for stochastic gradient estimates and also bound the amount of improvement with high probability. The authors then show how these bounds can be optimized by properly selecting the step size and batch size parameters. This is in contrast to previous work that only considers how the step size can be optimized. The result is an algorithm that can guarantee improvement with high probability.
Note: I did not verify all of the math, but at an intuitive level the theorems and lemmas in the main body are believable.
Major:
1. What am I supposed to take away from this paper? Am I supposed to think "I should use policy gradient methods with their safe step size & batch size approach if I need a safe RL algorithm"? Or, am I supposed to think "This paper is presenting some interesting theoretical insights into how step sizes and batch sizes impact policy gradient methods"?
If the answer is the former, then the results are abysmally bad. The axes are all scaled by 10^7 trajectories. Even the fastest learning methods are therefore impractically slow. This doesn't appear to be a practical algorithm. Compare to the results in your reference [23], where high confidence improvement is achieved using hundreds to thousands of episodes. So, the proposed algorithm does not appear to be practical or competitive as a practical method.
This makes me think that the contribution is the latter: a theoretical discussion about step sizes, batch sizes, and how they relate to guaranteed improvement. This is interesting and I would recommend acceptance if this was the pitch of the paper. However, this is not made clear in the introduction. Until I reached the empirical results I thought that the authors were proposing a practical algorithm. I propose that the introduction should be modified to make it clear what the contribution is: are you presenting theoretical results (e.g., like a PAC RL paper), or are you proposing a practical algorithm? If the answer is the latter, then you need to compare to existing works or explain why it is not an issue that your method requires so much data.
2. The assumption that the policy is Gaussian is crippling. In many of the real-world applications of RL (e.g., dialogue systems) the actions are discrete, and so the policy is not Gaussian. Does this extend to other policy forms, like softmax policies? Is this extension straightforward, or are there additional challenges?
Medium:
1. Line 92. You say that convergence is assured because the angular difference is at most \pi/2. A similar claim was made by Peters and Schaal in their natural actor-critic paper, and it later turned out to be false (See "GeNGA: A generalization of natural gradient ascent with positive and negative convergence results" by Thomas at ICML 2014). Your claim is different from Peters and Schaal's, but due to the history of these off-hand claims tending to be false, you should provide a proof or a reference for this claim.
2. You use concentration bounds that tend to be wildly over-conservative (Chebyshev & Hoeffding) because they must hold for a broad class of distributions. This is obviously the correct first step. However, have you considered using bounds that are far tighter (but make false distributional assumptions), like Student's t-test? If the gradient estimate is the mean of n i.i.d gradient estimates, then by the Central Limit Theorem the gradient estimate is approximately normally distributed, and so Student's t-test is reasonable. Empirically, does it tend to result in improvements with high probability while requiring far less data? It seems like a practical implementation should not use Hoeffding / Chebyshev / Bernstein...
3. Page 7 is one log paragraph.
Minor:
1. Line 74 "paramters"
2. Capitalization in section titles is inconsistent (e.g., section 3 capitalizes Policies but not Size or Scalar).
3. Line 91 "but, being the \alpha_i non-negative". Grammar error here.
4. Integrals look better with a slight space before the dx terms: $$\int_{\mathcal X} f(x) \, \mbox{d}x$$. The $\,$ here helps split the terms visually.
5. Line 102 "bound requires to compute" - requires who or what to compute something? Grammar.
6. The line split on line 111 is awkward.
7. Equation below 113, use \text{} for "if" and "otherwise".
8. Below 131, $\widehat$ looks better on the $\nabla$ symbol, and use $\max$ for max.
9. Line 140 say what you mean by the "optimal step size" using an equation.
10. Line 192, don't use contractions in technical writing (i.e., don't write "doesn't", write "does not").
AFTER AUTHOR REBUTTAL:
My overall review has not changed. I've increased to accept (7 from 6) since the authors are not claiming that they are presenting a practical algorithm, but rather presenting interesting theoretical results.