Learning safe policies with expert guidance
We propose a framework for ensuring safe behavior of a reinforcement learning agent when the reward function may be difficult to specify. In order to do this, we rely on the existence of demonstrations from expert policies, and we provide a theoretical framework for the agent to optimize in the space of rewards consistent with its existing knowledge. We propose two methods to solve the resulting optimization: an exact ellipsoid-based method and a method in the spirit of the "follow-the-perturbed-leader" algorithm. Our experiments demonstrate the behavior of our algorithm in both discrete and continuous problems. The trained agent safely avoids states with potential negative effects while imitating the behavior of the expert in the other states.
Learning from demonstrations usually faces an ill-posed problem of inferring the expert reward functions. To facilitate safe learning from demonstrations, the paper formulates a maximin learning problem over a convex reward polytope, in order to guarantee that the worst possible consistent reward would yield a policy that is not much worse than optimal. The assumption is that the reward is linear in known features. The authors proposed two method: (i) ellipsoid method and (ii) follow-the-perturbed leader using separation oracles and a given MDP solver. The experiment is done in a grid world setting, and a modified version of the cart-pole problem.   One piece that I failed to understand is from line 61-63: why apprenticeship learning won't return the same policy as the proposed solution? This seems like a major point for me, as the formulation looks very similar to Syed & Schapire 2008 (although they used Multiplicative Weights Update as the online learning procedure. There they also used linear reward assumption, and reduction of policy learning to feature expectation matching, similar to Abbeel & Ng 2004). Since the contrast to previous method was not discussed beyond this paragraph, I'm curious what the comparison would look like in the experiment? It seems that this discussion is missing from the experiment section. I think the paper will benefit from at least some additional intuition in the experiment / discussion sections into what's wrong with Syed & Schapire method (or also Apprenticeship Learning via Linear Programming, where iteratively invoking the oracles is not needed).   The other concern, as perhaps is also clear to the authors and other readers, is the scalability of the algorithms. Repeatedly invoking separation oracle and MDP solver may mean that this method will have limited applicability to higher dimensionalities.   Gridworld experiment: How many environments did you use for training? How many demonstrations? Cartpole experiment: how well would the method work without the addition of the two question blocks?  This is a relatively minor point, but I think the focus of the paper has a lot more to do with robust policy learning (due to worst case guarantee) than safe learning, which to me implies safe exploration / satisfying safety criteria during training in addition to high reward learning.   The paper is nicely written and well-organized.