My title
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learning (RL) often tradeoff between the throughput of samples that an RL agent can learn from (sample throughput) and the quality of learning from each sample (sample efficiency). In these scalable RL architectures, as one increases sample throughput (i.e. increasing parallelization in IMPALA (Espeholt et al., 2018)), sample efficiency drops significantly. To address this, we propose a new distributed reinforcement learning algorithm, IMPACT. IMPACT extends IMPALA with three changes: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. In discrete action-space environments, we show that IMPACT attains higher reward and, simultaneously, achieves up to 30% decrease in training wall-time than that of IMPALA. For continuous control environments, IMPACT trains faster than existing scalable agents while preserving the sample efficiency of synchronous PPO.
Reinforcement learning (RL) training speed is broadly evaluated on two dimensions:  sample efficiency (the number of environment interactions required) and wall-clock time.  Improved wall-clock training time has been achieved through distributed actors and learners, but often at the expense of sample efficiency.  IMPACT repurposes successful concepts from deep RL - the target network, importance sampling and a replay buffer to demonstrate improvements on both axes in on three continuous environments and three games from the Atari Learning Environment.
Positives
This was a well-written paper proposing to address the sample efficiency of distributed RL algorithms.  The diagrams of the algorithm were also well-done.  Improving the sample efficiency of algorithms is an important objective and the approaches followed here are sensible.
Additionally, the ablations and examination of the sensitive hyperparameters of the algorithm are useful analyses.  These indicate relative insensitivity to the target network update frequency, but both the importance sampling equation and the circular buffer hyperparameters are described.
Negatives
IMPACT introduces additional hyperparameters which are tuned for each continuous control task and discrete control task.  However, there is no description of the hyperparameter tuning budget allocated to IMPACT, PPO, IMPALA.  
Regarding the discrete environment, the game selection should be elaborated upon and if sufficient compute is available, the algorithm should be tested elsewhere.  Specifically, it is atypical (though not necessarily incorrect) to tune specific hyperparameters for each game in the Atari Learning Environment.  Traditionally, algorithms have been justified as robust and useful by the lack of need to tune per game.  Table 4 demonstrates a high degree of tuning for IMPACT due to game-specific changes for clip param, grad clip, lambda, num sgd iter, train_batch_size, value function loss coeff, kl coeff.  However, Table 5 (IMPALA) and Table 6 (PPO) have fewer noted changes.
Small nits: 
- Define advantage in the policy gradient equation
- Figure 4 is ahead of Figure 3 in the compiled LaTeX
Questions
- How were the discrete control games selected?
- What was the hyperparameter tuning budget for IMPACT versus PPO or IMPALA?
- If a fixed hyperparameter budget is allocated in advance and new environments are randomly selected, does IMPACT favorably compare to IMPALA and PPO?
- IMPALA performs remarkably badly in the three continuous control tasks, even on wall-clock time.  What validations have been done here to ensure the algorithm is operating as intended?
I will increase my rating if the robustness and improvements of this algorithm can be validated in randomly chosen games/continuous control environments for a fixed hyperparameter budget for IMPACT and both baselines.  Also, the IMPALA baseline should be validated for the continuous control tasks - it's surprising that this once SOTA-algorithm flounders in even simple tasks like Hopper-v2 or HalfCheetah-v2.
-----------------
Update:
The authors have addressed my initial concerns carefully through extra experiments and details in the Appendix and I have updated my rating accordingly.  Thanks!