Generalizing Skills with Semi-Supervised Reinforcement Learning
Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while it is deployed and performing useful tasks. However, this learning requires access to a reward function, to tell the agent whether it is succeeding or failing at its task. Such reward functions are often hard to measure in the real world, especially in domains such as robotics and dialog systems, where the reward could depend on the unknown positions of objects or the emotional state of the user. On the other hand, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present, or in a controlled laboratory setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect in the unstructured real world? In this paper, we formalize this problem setting as semi-supervised reinforcement learning (SSRL), where the reward function can only be evaluated in a set of “labeled” MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of “unlabeled” MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent’s own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging, continuous control tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.
The paper proposes to study the problem of semi-supervised RL where one has to distinguish between labelled MDPs that provide rewards, and unlabelled MDPs that are not associated with any reward signal. The underlying is very simple since it aims at simultaneously learning a policy based on the REINFORCE+entropy regularization technique, and also a model of the reward that will be used (as in inverse reinforcement learning) as a feedback over unlabelled MDPs. The experiments are made on different continous domains and show interesting results The paper is well written, and easy to understand. It is based on a simple but efficient idea of simultaneously learning the policy and a model of the reward and the resulting algorithm exhibit interesting properties. The proposed idea is quite obvious, but the authors are the first ones to propose to test such a model. The experiments could be made stronger by mixing continuous and discrete problems but are convincing.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Generalizing Skills with Semi-Supervised Reinforcement Learning
Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while it is deployed and performing useful tasks. However, this learning requires access to a reward function, to tell the agent whether it is succeeding or failing at its task. Such reward functions are often hard to measure in the real world, especially in domains such as robotics and dialog systems, where the reward could depend on the unknown positions of objects or the emotional state of the user. On the other hand, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present, or in a controlled laboratory setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect in the unstructured real world? In this paper, we formalize this problem setting as semi-supervised reinforcement learning (SSRL), where the reward function can only be evaluated in a set of “labeled” MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of “unlabeled” MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent’s own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging, continuous control tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes to study the problem of semi-supervised RL where one has to distinguish between labelled MDPs that provide rewards, and unlabelled MDPs that are not associated with any reward signal. The underlying is very simple since it aims at simultaneously learning a policy based on the REINFORCE+entropy regularization technique, and also a model of the reward that will be used (as in inverse reinforcement learning) as a feedback over unlabelled MDPs. The experiments are made on different continous domains and show interesting results The paper is well written, and easy to understand. It is based on a simple but efficient idea of simultaneously learning the policy and a model of the reward and the resulting algorithm exhibit interesting properties. The proposed idea is quite obvious, but the authors are the first ones to propose to test such a model. The experiments could be made stronger by mixing continuous and discrete problems but are convincing.
assistant
The paper proposes to study the problem of semi-supervised RL where one has to distinguish between labelled MDPs that provide rewards, and unlabelled MDPs that are not associated with any reward signal. The underlying is very simple since it aims at simultaneously learning a policy based on the REINFORCE+entropy regularization technique, and also a model of the reward that will be used (as in inverse reinforcement learning) as a feedback over unlabelled MDPs. The experiments are made on different continous domains and show interesting results The paper is well written, and easy to understand. It is based on a simple but efficient idea of simultaneously learning the policy and a model of the reward and the resulting algorithm exhibit interesting properties. The proposed idea is quite obvious, but the authors are the first ones to propose to test such a model. The experiments could be made stronger by mixing continuous and discrete problems but are convincing.