Experience replay (ER) is crucial for attaining high data-efficiency in off-policy deep reinforcement learning (RL). ER entails the recall of experiences obtained in past iterations to compute gradient estimates for the current policy. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors, possibly undermining the effectiveness of ER. Previous off-policy RL algorithms mitigated this issue by tuning their hyper-parameters in order to abate policy changes. We propose ReF-ER, a method for active management of experiences in the Replay Memory (RM). ReF-ER forgets experiences that would be too unlikely with the current policy and constrains policy changes within a trust region of the behaviors in the RM. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods to show that ReF-ER reliably improves the performance of continuous-action off-policy RL. We complement ReF-ER with a novel off-policy actor-critic algorithm (RACER) for continuous-action control. RACER employs a computationally efficient closed-form approximation of the action values and is shown to be highly competitive with state-of-the-art algorithms on benchmark problems, while being robust to large hyper-parameter variations.
The authors introduce two new algorithms: remember and forget experience replay (ReF-ER), and an actor-critic architecture for continuous-action problems which is significantly more computationally efficient than previous approaches (RACER). ReF-ER manages the experience in the replay memory more directly and removes trajectories (episodes) that follow policies less related to the current parameterized policy (based on the importance weights). RACER's main contribution is provides a closed form approximation of the action values, enabling significant gains computationally. They provide several empirical studies in benchmark domains showing the competitiveness of their approach, and the provided more stability to various continuous control algorithms (NAF, PG, u-DDPG).
Overall, I think it is a nicely written paper with a lot of empirical evidence of the usefulness of ReF-ER. I am quite interested in this algorithm specifically, as the active management of experience in the replay memory is an important step towards the ER acting as a proxy to short term memory. To my knowledge this algorithm is novel, and performs admirably. I'm less clear of the main benefits of RACER over previous approaches, except for better computational complexity. This primarily comes from a lack of empirical comparison, and not much explanation as to why key competitors were excluded. The inclusion of RACER seems to muddy the message of the paper, and a much stronger and deeper look at ReF-ER would have made for a stronger submission.
I have several questions for clarity and more comments below, but overall I think the paper is quite useful for the community and contains interesting insight into active management of transitions in an experience replay buffer.
Pros:
------
Lots of empirical studies. And a lot of details to impart intuition of the new experience replay.
Interesting take on experience replay.
Convincing results in many simulation benchmark domains (even though the competitors are sparse).
Cons:
------
There is some ambiguity and maybe some confusion about the difference between control and off-policy learning. While I agree you are learning off-policy for control (due to the experience replay buffer containing old data), the terms off-policy and on-policy seem overused here. Statements such as "ER has become one of the mainstay techniques to improve the sample-efficiency of off-policy RL" aren't entirely correct as the experience replay buffer is primarily used in deep reinforcement learning to improve sample-efficiency, not off-policy reinforcement learning as a whole.
The RACER algorithm seems to muddy up the message of the paper quite a bit. I would have much preferred an in-depth look at ReF-ER here, rather than the introduction of two algorithms. And I think your paper would have been stronger for it. That being said, the RACER algorithm seems incomplete. While it is an improvement over prior approachers (ACER) computationally, the need to use ReF-ER is concerning. I'm also a bit confused why ACER isn't used as a competitor against RACER? Even if you aren't outperforming the other approach on all benchmarks, the improved computational complexity is still a worthwhile improvement.
No confidence bounds in the results, although these are somewhat shown in the appendix (without the competitors shown!!). I'm curious at the significance of the different parameter settings.
Questions:
----------
I'm curious as to how this is related to something like rejection sampling? Or other importance sampling approaches more directly? How does your method compare with using retrace or some other off-policy algorithms? I'm unclear on the reasons why these types of comparisons aren't made empirically, could you clarify more directly?
Does your algorithm help with variance issues of other off-policy algorithms? Such as just using importance weights instead of retrace? How would it effect tree backup or just the usual importance sampling? It seems likely that this would help here, as you are limiting the amount of data with high importance weights, although this might also add bias.
Have you removed the target network in your experiments? This detail is not obvious in the paper currently and when you introduce ReF-ER you seem to be leading to this, but never say explicitly.
You claim that ReF-ER "reduces the sensitivity on the network architecture and training hyper-parameters." I'm unclear how you show this in the results with the current paper. You do some hyperparameter studies in the appendix, but don't compare against other algorithms here. Could you share a bit further how you are measuring the sensitivities of your algorithm against the competitors?
Do you need to anneal the cmax? What are the effects if this is set to some constant?
Could you expand on the results of HumanoidStandup-v2? Why do you believe your approach does significantly worse than the baselines here?
For DDPG, what happens if you change the bounds instead of removing them entirely? Also how does your method compare on a domain without unbounded actions?
It is unclear why RACER does not work with ER/PER. Do you have any intuition here? Could this be fixed through means other than ReF-ER? 
Other minor comments (not taken into consideration for the review):
-------
Pseudo code: It is a bit unclear what algorithm 1 is supposed to be, I'm assuming ReF-ER? 
Begin revision comments:
-----
Given the revisions to this paper, I am more confident that it will be of interest to the community. The major contributions here I see is the removal of target networks given their approach. Given this I still have concerns on clarity and still am unhappy with the lack of confidence intervals in the main experimental section. I've increased my score to 7 to reflect my increase in confidence.