Verifiable Reinforcement Learning via Policy Extraction
While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex policies (since they are nonparametric), yet can be efficiently verified using existing techniques (since they are highly structured). The challenge is that decision tree policies are difficult to train. We propose VIPER, an algorithm that combines ideas from model compression and imitation learning to learn decision tree policies guided by a DNN policy (called the oracle) and its Q-function, and show that it substantially outperforms two baselines. We use VIPER to (i) learn a provably robust decision tree policy for a variant of Atari Pong with a symbolic state space, (ii) learn a decision tree policy for a toy game based on Pong that provably never loses, and (iii) learn a provably stable decision tree policy for cart-pole. In each case, the decision tree policy achieves performance equal to that of the original DNN policy.
# Paper ID 1246  Verifiable RL via Policy Extraction  ## Summary  The main idea of the paper is to use a strong learned policy as an expert (e.g., deep neural network) in an imitation learning setup. A loss function designed to minimize the gap between the best and worst action (as defined by the expert) is used to train a CART policy. Three verifiable properties (correctness, robustness, stability) can be associated with the tree from the piecewise linearity of the partitions imposed by the tree policy that the original expert policy (e.g., a deep neural network) may not possess. However, correctness and stability require the system dynamics to be known. The empirical evaluation is somewhat limited. For example, experiments are shown on two relatively small domains (Atari Pong, cart-pole) where the model dynamics are simple enough to specify or approximate.  Overall, the paper is clearly well written. The algorithmic ideas are intuitively clear and seem sound to me. However, I'm a bit unclear about the feasibility / applicability of the verification process in other domains. The experimental section is also a bit limited. Overall, there are some interesting ideas in the paper but it's a bit difficult for me to evaluate due to the limited discussion and experiments. Detailed comments follow.   ## Detailed Comments  - The paper introduces a vanilla imitation learning setup for the task of extracting a policy from a given oracle or expert.  - A loss function designed to minimize the worst-case gap across the expected state distributions is defined. A decision tree learner is used on a dataset resampled under the new loss function which has the effect of "compiling" the original expert into a decision tree trained to minimize the expected worst-case loss across the distribution of states induced by the learned tree policy. This learning algorithm (Q-DAGGER) is the key algorithmic contribution.  - Intuitively, the main idea is to focus the training on states where the gap in the values between the best action and the worst action is largest. These are the "critical states". Standard 0-1 loss doesn't treat these states differently from other states. The idea here is that the learner should focus on avoiding the worst-case loss in states (i.e., $l_max$ is small). If such a policy can be found, then the bound improves over vanilla DAGGER.  - It's not clear to me how problems with "critical states" are distributed. For example, are the common control tasks in MuJoCo / OpenGym good candidate problems for this approach? Additional discussion characterizing problems where the methods are most / least applicable would be nice to have.  - Assuming such a problem is given, the approach proceeds by modifying the loss function to incorporate the worst-case gap in values. This requires an expert that can provide a worst-case "gap" (e.g., Q values, expert policy probabilities) between the best and worst action in any state. This seems reasonable to me.  - A tree policy can now be learned via CART in an iterative fashion (i.e. DAGGER). The main idea is to reweight the accumulated examples in the dataset using the worst-case gap values described above. This is the policy learned by the VIPER algorithm. This part is also very straightforward and well described.  - Next, the paper proposes three properties that are verifiable by VIPER under appropriate conditions. Correctness and stability require the model dynamics to be known while robustness does not. Here, I do think there needs to be additional discussion on the feasibility of obtaining the dynamics for other problems. Also, even if the dynamics are known, the difficulty of specifying the constraints in the form required by VIPER is unclear to me. A discussion of the complexity of these properties on MuJoCo tasks, for example, may be good to have. Discussion of these in Section 3 would strengthen the paper significantly, in my opinion.  - The empirical evaluation is performed on the Pong domain and cart-pole. It seems like verifying correctness and stability is rather difficult for a given domain. It might be worth discussing. Next, I'd be curious to know why robustness is only evaluated for Atari Pong and not cart-pole? Perhaps the paper could focus on a single property (e.g., robustness) and show experiments on additional domains. In its current form, the experimental section feels a bit limited. Additional experiments and their analysis would be very nice to have.  - Overall, it seems like the choice of loss function is orthogonal to the verification requirements, which only require the decision tree. Is my understanding correct? If yes, then other loss functions and their impact on tree size and properties would be quite interesting to study. For example, I'm curious to understand why exactly the trees under the 0-1 loss are larger for a particular reward target. How much of the tree size depends on the worst-case loss function seems like something worth exploring further.  Update -------- I thank the authors for their response. My overall score remains unchanged.