The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statistics over the path ensemble, which allows one to flexibly drive the learning process in a promotion focus or prevention focus. On top of this formulation, we systematically study the impacts of different regulatory focuses. Our findings reveal that regulatory focus, when chosen appropriately, can result in significant benefits. In particular, for the environments with sparse rewards, promotion focus would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, prevention focus is preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.
The paper studies the problem of advantage estimation for actor-critic RL algorithms. The key observation is that the advantage can be computed using 1-step returns, 2-step returns, etc. The paper suggests that, instead of choosing a fixed n, we should aggregate these advantageous together. If the maximum is taken, the resulting policy will be exploratory (i.e., have a "promotion focus"); if the minimum advantage is taken, the resulting policy will be risk sensitive (i.e., have a "regulatory focus").
The paper presents results on a number of tasks. A few toy examples show instances where the proposed method works better. Experiments on sparse reward environments show that taking the max advantage provides for exploration that outperforms a baseline (EX2). On a walking talk show that the min-approach can outperform state-of-the-art on-policy RL (PPO); the paper suggests this is because the min-approach is implicitly risk sensitive. Experiments on some of the Mujoco control tasks and some of the Atari games show that taking the advantage with the maximum absolute value performs well, as compared to PPO.
Overall, I think the main contribution of this paper is the finding that there exist smarter ways of computing the advantage. A second contribution is connecting the ideas of risk-sensitive and risk-seeking control with ideas from psychology (regulatory fit theory). 
I am leaning towards rejecting this paper. On the one hand, the paper is well written, well motivated, and the experiments quite thorough. On the other hand, I don't think the paper is particularly useful, either from a theory or algorithmic perspective. It is not surprising that if we add an additional hyperparater to an existing RL algorithm and tune that hyperparameter, we'll do better than the existing RL algorithm. The ablation experiments in Fig 6 and Fig 7 suggest that how the advantage estimates are combined, and how they are "mixed" with a standard advantage estimate, matter a lot. Further, the fact that using a smaller number of advantage estimates worked better (point #2 on pg 5, Effect of Ensemble Size in Appendix A) suggests that the ensemble size is an important hyperparameter, and that risk-seeking / risk-aversion (i.e., regulatory vs promotion focus) cannot alone explain why the proposed method works. I think that, for this idea to be useful, it must be equipped with a fixed value for these hyperparameters that works well across a wide range of tasks (e.g., a learning rate of 3e-4 works well for Adam on most tasks), or an automated strategy for choosing this hyperparameter (e.g., the automatic entropy tuning in SAC). I would consider increasing my review if either of these were included.
A second concern is that, despite the close connections with risk-sensitive and risk-seeking control (discussed in Section 2), none of these prior works are compared against. Many of these prior works include a temperature parameter for trading off risk-seeking vs risk-aversion (e.g., \beta in Eq 11 of [Mihatsch 2002]), which is arguably a more transparent (to the user) and easier to analyze (for the RL researcher) than the order statistics used in this paper. Given the wealth of prior work on risk-sensitive and risk-seeking control, I think it's important to know whether these prior methods already solve the problem at hand (choosing between risk-seeking and risk-aversion). I would consider increasing my review if one of these methods were included as a baseline.
Minor comments
* "that humans own" -> "that humans' own"
* The max strategy seems quite closely related to UCB-based exploration. I'd be curious to learn about some discussion on the similarities/differences
* Fig 6 -- Can you add error bars to show the variance across random seeds?
* Appendix D -- How does this approach and the results shown in Fig 9 differ from standard GAE? 
----------------------- UPDATE AFTER (NO) AUTHOR RESPONSE ----------------------------
The authors did not post a response, so I will maintain my vote to "weak reject" this paper. I would encourage the authors to incorporate the feedback in all reviews and submit the paper to a future conference.