Safe Policy Improvement by Minimizing Robust Baseline Regret
An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, which is guaranteed to outperform a given baseline strategy. In this paper, we develop and analyze a new model-based approach that computes a safe policy, given an inaccurate model of the system's dynamics and guarantees on the accuracy of this model. The new robust method uses this model to directly minimize the (negative) regret w.r.t. the baseline policy. Contrary to existing approaches, minimizing the regret allows one to improve the baseline policy in states with accurate dynamics and to seamlessly fall back to the baseline policy, otherwise. We show that our formulation is NP-hard and propose a simple approximate algorithm. Our empirical results on several domains further show that even the simple approximate algorithm can outperform standard approaches.
The paper considers the problem of robust MDP. In particular, it considers that we have an uncertain model of transition probabilities of the MDP (with the rectangular uncertainty) as well as a baseline policy. The goal is to find a new policy such that it is guaranteed to be no worse than the baseline. This is called the safe policy improvement.  A robust approach would find a policy that maximizes the worst case performance. To provide a safe policy, it has to compare it with the performance of the baseline. Since the performance of the baseline isn’t known, one can compare it with the best possible performance of the baseline. However this might be an overly pessimistic approach because the choice of the worst model for the robust policy is not necessarily the same as the choice of the best model selected to evaluate the baseline policy.  A more reasonable approach is to compare the regret of the policy compared to the baseline. The regret of a policy is defined as the difference of the performance of the baseline policy with the performance of the policy under the same probability model. By finding the policy that maximizes the negative worst case regret within the uncertainty set, we ensure that the resulting policy is safe, and hopefully less conservative than the previous approach.  The paper provides a performance bound for this solution (Theorem 5). It also shows that in general such a policy is randomized (Theorem 3). The difficulty, however, is that the corresponding optimization problem is NP-Hard (Theorem 6).  To circumvent this challenge, the paper suggests a heuristic approximation to this problem. The approximation is based on the fact that if there is no uncertainty for the MDP when actions are selected according to the baseline policy, the problem becomes a usual robust MDP problem (Proposition 7), which can be solved in polynomial time. The approximation is that the algorithm enforces the uncertainty for the actions selected by the baseline policy to be zero, as if there is no error in their model. This type of approximation is reasonable when the data is collected mostly from following the baseline policy with some occasional exploratory randomness to select other actions. In that case, the estimated model for the baseline is indeed much more accurate.  Note that this is still an approximation. The paper does not provide a guarantee on the quality of this approximation, but it empirically shows that the performance is better than a usual robust MDP approach that doesn’t change the uncertainty set (Algorithm 2). It is also much better than a certainty-equivalence approach that does not take the uncertainty into account (Algorithm 1).
Brief summary of evaluation:  Technical quality: The results seem to be technically sound. The technical tools are more or less standard. Novelty/originality: The optimization formulation seems to be novel. The derivation of the error bounds is more or less standard. Potential impact or usefulness: This might be a stepping stone for further results, but I doubt it would have a huge real-world impact as it is, because a) it works only for finite MDPs and b) the solution doesn’t have a guarantee. Clarity and presentation: It is relatively well-written.  ===  This is a reasonably good paper. It addresses an important problem that is of interest to the real-world applications of reinforcement learning. It formulates the problem as an optimization problem that is much more reasonable than a conventional robust MDP formulation. It is unfortunate that the solution to the new optimization problem is NP-Hard (Theorem 6). However showing that this is the case is important. Also providing the theoretical guarantee on how well we can expect the optimal solution to behave is important too (Theorem 5).  The suggested algorithm (Algorithm 1) is indeed very simple. It would be much better to have a theoretical guarantee on the performance of the approximate algorithm, but in lieu of that, the empirical results are reassuring.  Some detailed comments:  - Algorithm 1 (L4): Shouldn’t the term rho(pi_B,xi) be removed? It is constant anyway.  - L198: The reason #2 is not clear.  - L69: Please specify that the paper is about finite state and action spaces.  - L159 (Theorem 5): The norms as well as e_{\pi*} and e_{\pi_B} are not defined.  - L90: What does a randomized uncertain set mean?  - Please compare the choice of regret as the objective to be minimized with the following work:  Regan and Boutilier, “Regret-based Reward Elicitation for Markov Decision Process,” USI, 2009. The paper is not about the uncertainty in P, but they choose regret too.  === After Rebuttal: Thank you for the response.