Deep reinforcement learning has obtained significant breakthroughs in recent years. Most methods in deep-RL achieve good results via the maximization of the reward signal provided by the environment, typically in the form of discounted cumulative returns. Such reward signals represent the immediate feedback of a particular action performed by an agent. However, tasks with sparse reward signals are still challenging to on-policy methods. In this paper, we introduce an effective characterization of past reward statistics (which can be seen as long-term feedback signals) to supplement this immediate reward feedback. In particular, value functions are learned with multi-critics supervision, enabling complex value functions to be more easily approximated in on-policy learning, even when the reward signals are sparse. We also introduce a novel exploration mechanism called "hot-wiring" that can give a boost to seemingly trapped agents. We demonstrate the effectiveness of our advantage actor multi-critic (A2MC) method across the discrete domains in Atari games as well as continuous domains in the MuJoCo environments. A video demo is provided at https://youtu.be/zBmpf3Yz8tc and source codes will be made available upon paper acceptance.
Recommendation: Weak reject
Summary:
The paper proposes a variant of deep reinforcement learning (A2MC) for environments with sparse rewards.  The approach replaces the standard environment reward function with a combination of the current reward and the variability of rewards in the last T timesteps, with a goal of decreasing variability.  The authors further propose a “hot-wiring” exploration strategy to bootstrap agents by taking either random actions or actions that have been done in the recent history.  Empirical evaluations in standard benchmarks including several sparse reward Atari games show empirical improvement of this approach over a baseline (ACKTR).
Review:
The paper has strong empirical results that show the A2MC outperforming or reaching the same performance as the baselines in a large number of Atari and MuJoCo domains.  The authors also provide results with and without the hot-wiring feature, which helps isolate its contribution.  However, overall the paper lacks theoretical rigor and most of the proposed changes are done without principled reasons or convergence guarantees.   There is no way of telling from the current paper whether these changes could lead to divergence or suboptimal behavior in other domains.  Examples of such changes include:
* The averaging of the reward terms at different timescales in Equation 4 is the core of the algorithm but is derived ad-hoc.  Why is this a good equation?  Is lack of variability really a desired property and may it lead to a suboptimal policy?  Can anything be said about how it changes behavior in a tabular representation?
* The exponential equation with a constant of 100 appears out of nowhere in equation 5.  Is this a general equation that will really work in different domains and reward scales?
* The variability weights in equations 6 and 7 are never tested empirically – what happens if they are left out?  Where did this equation come from?
* Overall, it is unclear if the combination of rewards at different time scales in equation 10 is stable and leads to convergence.  The terms show resemblance to the eligibility trace equations but lack their theoretical properties.
To make the paper ready for publication, the authors need to justify which of these changes are “safe” in that they guarantee the behavior of the algorithm cannot become much worse, or need to point directly to other methods in the literature that have used such changes and cite the pros and cons that were seen with those changes.
Related to the theme above, the paper does not properly cite other methods used with sparse rewards in traditional RL or Deep RL, especially eligibility traces, which seem highly related to the current approach.  The following related work edits are needed:
* The overall approach is thematically similar to eligibility traces (see the standard Sutton and Barto textbook), except that the authors here use variability rather than combining the reward terms directly.  Eligibility traces are built exactly for these sorts of sparse reward problems and combine short and long-term rewards in a TD update. But there has been substantial investigation of their theoretical properties in the tabular and function approximation cases. The current method needs to compare and contrast to this long-standing method both theoretically and empirically. 
* Two other methods that should have been considered in the experiments are experience replay and reward shaping, both of which are beneficial with deep RL in sparse domains.  Experience replay is mentioned in the paper but not implemented as a competitor.  I realize ER is not as computationally efficient as the new approach but it is an important (and stable) baseline.  Reward shaping is not mentioned at all, but is again an important and stable baseline that has been used in such problems – see “Playing FPS Games with Deep Reinforcement Learning” (AAAI, 2017).
* Finally, the related work section mentions a lot of competitive algorithms but does not implement any of them in comparison, which makes it hard to claim the current approach is the best yet proposed.