COMPETITIVE EXPERIENCE REPLAY
Deep learning has achieved remarkable successes in solving challenging reinforcement learning (RL) problems when dense reward function is provided. However, in sparse reward environment it still often suffers from the need to carefully shape reward function to guide policy optimization. This limits the applicability of RL in the real world since both reinforcement learning and domain-specific knowledge are required. It is therefore of great practical importance to develop algorithms which can learn from a binary signal indicating successful task completion or other unshaped, sparse reward signals. We propose a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents. Our method complements the recently proposed hindsight experience replay (HER) by inducing an automatic exploratory curriculum. We evaluate our approach on the tasks of reaching various goal locations in an ant maze and manipulating objects with a robotic arm. Each task provides only binary rewards indicating whether or not the goal is achieved. Our method asymmetrically augments these sparse rewards for a pair of agents each learning the same task, creating a competitive game designed to drive exploration. Extensive experiments demonstrate that this method leads to faster converge and improved task performance.
The authors propose a new method for learning from sparse rewards in model-free reinforcement learning settings. This is a challenging and important problem in model-free RL, mainly due to the lack of effective exploration. They propose a new way of densifying the reward by encouraging a pair of agents to explore different states (using competitive self-play) while trying to learn the same task. One of the agents (A) receives a penalty for visiting states that the other agent (B) also visits, while B is rewarded for visiting states found by A. They evaluate their method on a few tasks with continuous action spaces such as ant navigation in a maze and object manipulation by a simulated robotic arm.  Their method shows faster convergence (in some cases) and better performance than comparable algorithms.
  
Strengths:
Attempts to solve a long-standing problem in model-free RL (effective exploration in sparse reward environments)
Clear writing and structure, easy to understand (except for some minor details)
Novel, intuitive, and simple method building on ideas from previous works
Good empirical results (better than state of the art, in terms of performance) on some challenging tasks
Weaknesses:
Not very clear why (and when) the method works -- more insight from experiments in less complex environments or some theoretical analysis would be helpful
It would also be useful to better understand the conditions under which we can expect this to bring significant gains and when we can expect this to fail (or not help more than other methods) 
Not clear how stable (to train) and robust (to different environment dynamics) the method is
Main Comments / Questions:
The paper makes the claim that their technique “automatically generates a curriculum of exploration” which seems to be based more on intuition rather than clear experiments or analysis. I would suggest to either avoid making such claims or include stronger evidence for that. For example, you could consider visualizing the visited states by A and B (for a fixed goal and initial state) at different training epochs. Other such experiments and analysis would be very helpful.
It is known that certain reward shaping approaches can have negative consequences and lead to undesired behaviors (Ng et al., 1999; Clark & Amodei, 2016). Why can we expect that this particular type of reward shaping doesn’t have such side effects? Can it be the case that due to this adversarial reward structure, A learns a policy that takes it to some bad states from which it will be difficult to recover or that A & B get stuck in a cyclic behavior? Have you observed such behaviors in any of your experiments?
Do you train the agents with using the shaped reward (from the exploration competition between A and B) for the entire training duration? Have you tried to continue training from sparse reward only (e.g. after the effect ratio has stabilized)? One problem I see with this approach is the fact that you never directly optimize the true sparse reward of the tasks, so in the late stages of training your performance might suffer because the agent A is still trying to explore different parts of the state space. 
Can you comment on how stable this method is to train (given its adversarial nature) and what potential tricks can help in practice (except for the discussion on batch size)?
Please make clear the way you are generating the result plots (i.e. is A evaluated on the full task with sparse reward and initial goal distribution with no relabelling?).
In Algorithm 1, can you include the initialization of the goals for A and B? Does B receive identical goals as A?
It would also be helpful to more clearly state the limitations and advantages of this method compared to other algorithms designed for more efficient exploration (e.g. the need for a resettable environment for int-CER but not for ind-CER etc.).
Minor Comments / Questions:
You might consider including more references in the Related Work section that initializing from different state distributions such as Hosu & Rebedea (2016), Zhu et al. (2016), and Kakade & Langford (2002), and perhaps more papers tackling the exploration problem. 
Can you provide some intuition on why int-CER performs better than ind-CER (on most tasks) and why in Figure 1, HER + int-CER takes longer to converge than the other methods on the S maze?
In Figure 4, why are you not including ind-CER (without HER)?
Have you considered training a pool of agents with self-play (for the competitive exploration) instead of two agents? Is there any intuition on expecting one or the other to perform better?
Plots:
What is the x-axis of the plots? Number of samples, episodes, epochs? Please label it.
Please be explicit about the variance shown in the plots. Is that the std?
It would be helpful if to have larger numbers on the xy-axes. It is difficult to read when on paper.
Can you explain how you smoothed the curves -- whether before or after taking the average and perhaps include the min and max as well. I believe this could go in the Appendix.
Notation:
I don’t understand the need for calling the reward r_g instead of r. I believe this introduces confusion since the framework already has r taking as argument the goal g (eq. 1) while the g in the subscript doesn’t seem to refer to a particular g but rather to a general fact (that this is a reward for a goal-oriented task with sparse reward, where the goals are a subset of the states) (eq. 4)
Please use a consistent notation for Q. In sections 2.1 and 2.2, at times you use Q(s,a,g), Q(a,s,g) or Q(s,a).
Typos:
Page 6, last paragraph of section 4.1: Interestingly, even the … , is enough to support …
Page 7, last paragraph of section 4.3: Interestingly, … adversely affects both ...