Imitation learning algorithms provide a simple and straightforward approach for training control policies via standard supervised learning methods. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator -typically a person -to provide the demonstrations. In this paper, we ask: can we use imitation learning to train effective policies without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, in the setting where the tasks correspond to different goals, every trajectory is a successful demonstration for the state that it actually reaches. Informed by this observation, we propose a very simple algorithm for learning behaviors without any demonstrations, user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning settings with example demonstrations, we present the first instance of this approach as a method for learning goal-reaching policies entirely from scratch. We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems.
This work presents the goal-conditioned supervised learning algorithm (GCSL), which learns goal conditioned policies using only behavioral-cloning of the agent's own actions.  The intuition behind the algorithm is the goal of an observed trajectory can be identified after the fact, by simply looking at the states reached during that trajectory.  GCSL treats each executed action as a sample from the expert policy conditioned on each of the states reached after that action is taken.  Given a distribution over goal states, GCSL alternates between executing its current goal-conditioned policy on randomly selected goals, and learning to imitate the generated actions conditioned on the states they actually reached.  Experimental results demonstrate superior performance against a base (non-goal conditioned) RL algorithm (TRPO), and against another approach to learning goal-conditioned polices (TD3-HER), on a relatively diverse set of control problems.
A major issue is that the proof of the main theoretical result appears to be wrong.  As there don't appear to be any constraints placed on the policy pi_old, it would seem that the surrogate loss would collapse to 0 for any policy pi if pi_old is such that the target goal is never reached (the probability of any trajectory t reaching g is 0 under pi_old(t|g)).  It seems to be the case that the quality of the GCSL loss depends on the relationship between pi_old and the goal distribution p(g).  The fact that the theoretical results are incorrect does not mean that the algorithm, or the general approach do not have value, but it does highlight the fact that this approach may only be effective for a specific class of problems similar to the experimental domains.
While not a flaw in the work itself, it should be made clear in the text that the notion of optimality for the learning tasks considered in this work (i.e. achieving the goal by the end of episode), avoids one of the apparent limitations of the algorithm.  A randomly generated trajectory is itself optimal for any state that it reaches, if we define optimality as simply reaching a state.  Such a trajectory may not be the most efficient way of reaching that state however, so the relabelling process would seem to be prone to learning policies that achieve the conditioned goals, but not doing so in an efficient manner.  It isn't clear how well this approach would work for tasks where the efficiency, in terms of the time required to reach the objective, is a key part of the evaluation.  Again, this is not a flaw in the work itself, and it is possible that the algorithm will be effective in such tasks, perhaps because the likelihood of an action resulting in a given state is higher if that action brings us closer to this state.  It might be useful to conduct some additional experiments where evaluation is based on the time required to solve a task, rather than just the accuracy of the final state.