We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.
This paper aims to improve the efficiency of the actor-critic method. The authors first analyzed the cause of instability in the prior work, from the perspective of bias and variance. Two remedies were then presented: (i) mixing the experience replay with online learning; (ii) proposing a trust region scheme to select the behavior policies. The authors finally tested the proposed method on Atari games, and showed the better results, compared with the state-of-the-art methods.
In my opinion, the empirical results are impressive, and the authors also provided some insights for the motivation. Given the results on Atari games, this paper could be a great contribution on the actor critic methods. The propositions are presented to support relevant claims, while their significance seems a bit limited, and some further clarification is necessary. The authors also need to address a few confusing statements and missing details.
1. In Proposition 3, the authors claimed that mixing with on-policy data can reduce the bias. I checked the proof but did not find anything relevant. Also, what is the amount of bias reduced?
2. In Equation (1), could you provide a formal definition for "V"? 
3. The authors claimed at the beginning of Section 4 that the trust region method was proposed to mitigate the bias and variance problem of V-trace. However, I did not see how this is reflected in Propositions 4 and 5. Is this statement only based on empirical results?
4. It was mentioned right below Equation (4) that "Observe how this inner expectation ... matches the on-policy return...". Could you provide a formal proof?
5. What are the hyperparameters for the 9 agents used in Figure 1? Also, how did you choose "b" in trust region?
6. A few notation issues / typo:
(1) it's -> its
(2) In Equation (5), should "z \in M_{\beta, \pi} (s_t)" be "\mu_z \in M_{\beta, \pi} (s_t)"?
(3) At the 2nd line of Page 7, should the content for the indicator function be "\beta (\pi, \mu, s_t) < b"?