Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.
This work presents the value iteration with negative sampling (VINS) algorithm, a method for accelerating reinforcement learning using expert demonstrations. In addition to learning an expert policy through behavioral cloning, VINS learns an initial value function which is biased to assign smaller expected values to states not encountered during demonstrations. This is done by augmenting the demonstration data with states that have been randomly perturbed, and penalizing the value targets for these states by a factor proportional to their Euclidean distance to the original state. In addition to the policy and value function, VINS also learns a one-step dynamics model used to select actions against the learned value function. As the value function learned in VINS is only defined with respect to the current state, action values are estimated by sampling future states using the learned model, and computing the value of these sampled states. Empirical results on a set of robotic control tasks demonstrate that VINS requires significantly less interaction with the environment to learn a good policy than existing, state of the art approaches given the same set of demonstrations. While the paper presents a novel and highly effective approach, there are some apparent limitations to the algorithms which should be highlighted, and there is room for improvement in the empirical evaluations. It is unclear that VINS would generalize well beyond robotic control domains. For one, its theoretical guarantees depend on the local reversibility of the dynamics, that is, for small deviations from the desired state, it is possible to return to that state in a single step. This isn*t too significant a restriction, as the ability to recover quickly from small mistakes would seem to be a necessary for any method to be able to provide similar guarantees about its behavior. The bigger issue is the use of the Euclidean metric (or any fixed metric) in the definition of conservative extrapolation. Basically, a state is said to be similar to the states observed during the demonstrations if it is close, under the Euclidean metric, to at least one demonstrated state. This is a reasonable approach in robotic control tasks, where Euclidean distance is a good measure of how similar two configurations are to one another, but it would seem to be unsuitable for domains where the observation space consists of images or other high-dimensional representations. In those cases, a useful notion of similarity would likely have to be learned from the data. In such domains, one might imagine that the conservative value function would simply learn to distinguish between real observations, and those that have been perturbed by random noise, which would never be observed in the actual task. While experimental results demonstrate a very significant advantage for VINS both in terms of sample complexity and final performance, results are presented for only two tasks, *pick-and-place* and *push*, while VINS outperforms the alternatives on these tasks, it is worth noting that its initial performance (without additional environment interact) is not dramatically superior to pure behavioral cloning. It would be helpful to see how well VINS compares against the alternatives for a much smaller number of demonstrations, say 5-20, a regime where we would expect initial performance to be poor.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work presents the value iteration with negative sampling (VINS) algorithm, a method for accelerating reinforcement learning using expert demonstrations. In addition to learning an expert policy through behavioral cloning, VINS learns an initial value function which is biased to assign smaller expected values to states not encountered during demonstrations. This is done by augmenting the demonstration data with states that have been randomly perturbed, and penalizing the value targets for these states by a factor proportional to their Euclidean distance to the original state. In addition to the policy and value function, VINS also learns a one-step dynamics model used to select actions against the learned value function. As the value function learned in VINS is only defined with respect to the current state, action values are estimated by sampling future states using the learned model, and computing the value of these sampled states. Empirical results on a set of robotic control tasks demonstrate that VINS requires significantly less interaction with the environment to learn a good policy than existing, state of the art approaches given the same set of demonstrations. While the paper presents a novel and highly effective approach, there are some apparent limitations to the algorithms which should be highlighted, and there is room for improvement in the empirical evaluations. It is unclear that VINS would generalize well beyond robotic control domains. For one, its theoretical guarantees depend on the local reversibility of the dynamics, that is, for small deviations from the desired state, it is possible to return to that state in a single step. This isn*t too significant a restriction, as the ability to recover quickly from small mistakes would seem to be a necessary for any method to be able to provide similar guarantees about its behavior. The bigger issue is the use of the Euclidean metric (or any fixed metric) in the definition of conservative extrapolation. Basically, a state is said to be similar to the states observed during the demonstrations if it is close, under the Euclidean metric, to at least one demonstrated state. This is a reasonable approach in robotic control tasks, where Euclidean distance is a good measure of how similar two configurations are to one another, but it would seem to be unsuitable for domains where the observation space consists of images or other high-dimensional representations. In those cases, a useful notion of similarity would likely have to be learned from the data. In such domains, one might imagine that the conservative value function would simply learn to distinguish between real observations, and those that have been perturbed by random noise, which would never be observed in the actual task. While experimental results demonstrate a very significant advantage for VINS both in terms of sample complexity and final performance, results are presented for only two tasks, *pick-and-place* and *push*, while VINS outperforms the alternatives on these tasks, it is worth noting that its initial performance (without additional environment interact) is not dramatically superior to pure behavioral cloning. It would be helpful to see how well VINS compares against the alternatives for a much smaller number of demonstrations, say 5-20, a regime where we would expect initial performance to be poor.
assistant
This work presents the value iteration with negative sampling (VINS) algorithm, a method for accelerating reinforcement learning using expert demonstrations. In addition to learning an expert policy through behavioral cloning, VINS learns an initial value function which is biased to assign smaller expected values to states not encountered during demonstrations. This is done by augmenting the demonstration data with states that have been randomly perturbed, and penalizing the value targets for these states by a factor proportional to their Euclidean distance to the original state. In addition to the policy and value function, VINS also learns a one-step dynamics model used to select actions against the learned value function. As the value function learned in VINS is only defined with respect to the current state, action values are estimated by sampling future states using the learned model, and computing the value of these sampled states. Empirical results on a set of robotic control tasks demonstrate that VINS requires significantly less interaction with the environment to learn a good policy than existing, state of the art approaches given the same set of demonstrations. While the paper presents a novel and highly effective approach, there are some apparent limitations to the algorithms which should be highlighted, and there is room for improvement in the empirical evaluations. It is unclear that VINS would generalize well beyond robotic control domains. For one, its theoretical guarantees depend on the local reversibility of the dynamics, that is, for small deviations from the desired state, it is possible to return to that state in a single step. This isn*t too significant a restriction, as the ability to recover quickly from small mistakes would seem to be a necessary for any method to be able to provide similar guarantees about its behavior. The bigger issue is the use of the Euclidean metric (or any fixed metric) in the definition of conservative extrapolation. Basically, a state is said to be similar to the states observed during the demonstrations if it is close, under the Euclidean metric, to at least one demonstrated state. This is a reasonable approach in robotic control tasks, where Euclidean distance is a good measure of how similar two configurations are to one another, but it would seem to be unsuitable for domains where the observation space consists of images or other high-dimensional representations. In those cases, a useful notion of similarity would likely have to be learned from the data. In such domains, one might imagine that the conservative value function would simply learn to distinguish between real observations, and those that have been perturbed by random noise, which would never be observed in the actual task. While experimental results demonstrate a very significant advantage for VINS both in terms of sample complexity and final performance, results are presented for only two tasks, *pick-and-place* and *push*, while VINS outperforms the alternatives on these tasks, it is worth noting that its initial performance (without additional environment interact) is not dramatically superior to pure behavioral cloning. It would be helpful to see how well VINS compares against the alternatives for a much smaller number of demonstrations, say 5-20, a regime where we would expect initial performance to be poor.