Multiple-Step Greedy Policies in Online and Approximate Reinforcement Learning
Multiple-step lookahead policies have demonstrated high empirical competence in Reinforcement Learning, via the use of Monte Carlo Tree Search or Model Predictive Control. In a recent work [5], multiple-step greedy policies and their use in vanilla Policy Iteration algorithms were proposed and analyzed. In this work, we study multiple-step greedy algorithms in more practical setups. We begin by highlighting a counter-intuitive difficulty, arising with soft-policy updates: even in the absence of approximations, and contrary to the 1-step-greedy case, monotonic policy improvement is not guaranteed unless the update stepsize is sufficiently large. Taking particular care about this difficulty, we formulate and analyze online and approximate algorithms that use such a multi-step greedy operator.
Summary: The paper studies multi-step greedy policies and their use in approximate policy iteration. The authors first show a negative result that soft-policy updates using the multi-step greedy policies do not guarantee policy improvement. Then the authors proposed an algorithm that uses cautious soft updates (only update to the kappa greedy policy only when assured to improve, otherwise stay with one-step greedy policy) and show that it converges to the optimal policy. Lastly the authors studied hard updates by extending APIs to multi-step greedy policy setting.   Comments:    1. Theorem 2 presents an interesting and surprising result. Though the authors presented the example in the proof sketch, but I wonder if the authors could provide more intuitions behind this? Based on the theorem, for multi-step greedy policy, it seems that h needs to be bigger than 2. So I suspect that h = 2 will still work (meaning there could exist small alpha)? Obviously h = 1 works, but then why when h = 3, the soft-update suddenly stops working unless alpha is exactly equal to 1?  I would expect that one would require larger alpha when h gets larger.   2. Alg 1 and theorem 4 are nice. While I didn’t check the proof of theorem 4, how fast the algorithm will converge compare to a simple one-step greedy version? Namely what is the exact benefit of introducing the extra q_{\kappa} here (lemma 3 shows that the contraction is still gamma, and has nothing to do with kappa here)?   3. Regarding the hard updates, how realistic is the assumption on the existence of a kappa-greedy policy oracle? This is different from a greedy oracle that one would typically use in, for instance, CPI, where the oracle can be implemented by classic cost-sensitive classification with cost-to-go evaluated simply by roll-outs. To compute such a kappa-greedy policy here, we essentially need to solve a MDP (i.e. planning), though it is reshaped and the discount factor is decreased. Using a conservative policy iteration for such a greedy oracle is strange. Why not just using CPI to solve the original problem? Can we show that using CPI as the oracle for computing kappa-greedy policy eventually resulting a faster algorithm than CPI itself, if one set kappa optimally?  Similarly, can we show that using CPI as the kappa greedy policy oracle in PSDP resulting a faster algorithm than PSDP?  After rebuttal:   Thanks for detailed comments in the rebuttal. Though understanding the limitation of kappa-greedy oracle and how to set kappa are important for practical applications of the proposed approaches, I do think the paper already contains significant contribution. I've updated my score.