Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.
The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. The paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks. Some elements were unclear to me: - *We also assume that the entropy of the resulting state distribution H(p(S | p?)) is no less than the entropy of the goal distribution H(p?(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are.* How do you ensure this in practice? - In the second paragraph of 2.2, it is written *Note that this assumption does not require that the entropy of p(S | p?) is strictly larger than the entropy of the goal distribution, p?.* Could you please clarify? The experiments are interesting, yet some interpretations might be too strong (see below): - In the first experiment, *Does Skew-Fit Maximize Entropy?*, it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster. - On the comments of Figure 6, the paper mentions that *The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts.* I*m unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time. - In the experiment *Real-World Vision-Based Robotic Manipulation*, It is written that *a near-perfect success rate [is reached] after five and a half hours of interaction time*, while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this *5.5 hours* comes from.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. The paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks. Some elements were unclear to me: - *We also assume that the entropy of the resulting state distribution H(p(S | p?)) is no less than the entropy of the goal distribution H(p?(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are.* How do you ensure this in practice? - In the second paragraph of 2.2, it is written *Note that this assumption does not require that the entropy of p(S | p?) is strictly larger than the entropy of the goal distribution, p?.* Could you please clarify? The experiments are interesting, yet some interpretations might be too strong (see below): - In the first experiment, *Does Skew-Fit Maximize Entropy?*, it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster. - On the comments of Figure 6, the paper mentions that *The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts.* I*m unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time. - In the experiment *Real-World Vision-Based Robotic Manipulation*, It is written that *a near-perfect success rate [is reached] after five and a half hours of interaction time*, while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this *5.5 hours* comes from.
assistant
The paper introduces SKEW-FIT, an exploration approach that maximizes the entropy of a distribution of goals such that the agent maximizes state coverage. The paper is well-written and provides an interesting combination of reinforcement learning with imagined goals (RIG) and entropy maximization. The approach is well motivated and simulations are performed on several simulated and real robotics tasks. Some elements were unclear to me: - *We also assume that the entropy of the resulting state distribution H(p(S | p?)) is no less than the entropy of the goal distribution H(p?(S)). Without this assumption, a policy could ignore the goal and stay in a single state, no matter how diverse and realistic the goals are. * How do you ensure this in practice? - In the second paragraph of 2.2, it is written *Note that this assumption does not require that the entropy of p(S | p?) is strictly larger than the entropy of the goal distribution, p?. * Could you please clarify? The experiments are interesting, yet some interpretations might be too strong (see below): - In the first experiment, *Does Skew-Fit Maximize Entropy? *, it is empirically illustrated that the method does result in a high-entropy state exploration. However, it is only compared to one very naive way of exploring and it is not discussed whether other techniques also achieve the same entropy maximization. The last sentences seems to imply that only this technique ends up optimizing the entropy of the state coverage, while I believe that the claim (given the experiment) should only be about the fact it does so faster. - On the comments of Figure 6, the paper mentions that *The other methods only rely on the randomness of the initial policy to occasionally pick up the object, resulting in a near-constant rate of object lifts. * I*m unsure about the interpretation of this sentence given Figure 6 because other methods do not seem to fail entirely when given enough time. - In the experiment *Real-World Vision-Based Robotic Manipulation*, It is written that *a near-perfect success rate [is reached] after five and a half hours of interaction time*, while on the plot it is written 60% cumulative success after 5.5 hours and it is thus not clear where this *5.5 hours* comes from.