Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
In classical reinforcement learning agents accept arbitrary short term loss for long term gain when exploring their environment. This is infeasible for safety critical applications such as robotics, where even a single unsafe action may cause system failure or harm the environment. In this paper, we address the problem of safely exploring finite Markov decision processes (MDP). We define safety in terms of an a priori unknown safety constraint that depends on states and actions and satisfies certain regularity conditions expressed via a Gaussian process prior. We develop a novel algorithm, SAFEMDP, for this task and prove that it completely explores the safely reachable part of the MDP without violating the safety constraint. To achieve this, it cautiously explores safe states and actions in order to gain statistical confidence about the safety of unvisited state-action pairs from noisy observations collected while navigating the environment. Moreover, the algorithm explicitly considers reachability when exploring the MDP, ensuring that it does not get stuck in any state with no safe way out. We demonstrate our method on digital terrain models for the task of exploring an unknown map with a rover.
The contribution of this paper lies in extending the work of safe exploration in Bayesian optimization in [22] to include reachability and returnability constraints within the state space.   The theoretical results appear to be sound. From the paper, it is not clear to me though what the non-trivial technical challenges are in extending the proofs and derivations in [22] to account for the additional reachability and returnability constraints.   It is also not clear to me how the trade-off of performance in exploring the maximum reachable safe set for computational efficiency for the proposed algorithm compares to that of [14] since there is no empirical comparison with [14] even though a similar setup to [14] is used. I’m also concerned about how sensitive the safe exploration performance of the proposed algorithm is to poorly estimated GP hyperparameter values which are not available a priori.   More detailed comments are available below.
NOVELTY  Lines 57-59: The authors say that “[22] considered a bandit setting, where at each iteration any arm can be played. In contrast, in this paper we consider an MDP, which introduces restrictions in terms of reachability that have not been considered in Bayesian optimization before” and “The main contribution consists of extending the work on safe Bayesian optimization in [22] from the bandit setting to MDPs.” What is the implication of such additional reachability and returnability constraints on the difference in the resulting technical proofs and derivations? Does it invalidate the proof technique in [22] or did the authors use a similar proof technique in [22] with additional non-trivial technical challenges to be tackled?    PRACTICAL SIGNIFICANCE  Lines 281-2: How do you obtain the values of the lengthscales and noise standard deviation hyperparameters, considering that the height distribution is unknown? If you set them manually, how sensitive are your results in terms of safety to different hyperparameter values? Can you justify why it is practical to do this for safe exploration?  The authors have used the same experimental setup as that in [14]. Though the authors have indicated that [14] is much more computational demanding than their proposed approach, it is not clear whether how the trade-off of performance in exploring the maximum reachable safe set for computational efficiency for the proposed algorithm compares to that of [14].  Line 78: Can the assumption of a deterministic transition model be relaxed in the proposed work? If so, what are the implications? Does [14] impose a similar assumption?  In MDP planning, besides the exploration aspect, there is also the exploitation aspect that is not accounted for nor discussed in this paper. Must the safe exploration in Algorithm 1 be performed separately first before exploiting the MDP for optimal planning? Or can the exploration be performed simultaneously with exploitation in planning like those in the following references (albeit without safety constraints/considerations):  C. K. Ling, K. H. Low, and P. Jaillet (2016). Gaussian Process Planning with Lipschitz Continuous Reward Functions: Towards Unifying Bayesian Optimization, Active Learning, and Beyond. In Proc. AAAI.  Marchant, R.; Ramos, F.; and Sanner, S. (2014). Sequential Bayesian optimisation for spatial-temporal monitoring. In Proc. UAI.   CLARITY OF PRESENTATION  Equation 1: The authors can give an intuitive explanation of how epsilon is related to noise omega_t and/or the GP predictive uncertainty of r due to limited measurements.  To avoid confusion, I recommend naming S_t in line 159 and equation 5 differently from that in line 152 because they are defined differently.  In Theorem 1, what value is delta set to?    Minor issues: Fig. 1: The notations appear to be inconsistent with that in equation 3. Missing bracket in first line of for loop in Algorithm 1. line 192: Should u_t(s) be u_t(s’) instead? line 194: has has?