Model-based Reinforcement Learning(RL) has shown great advantage in sampleefficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.
Summary
This paper expands on previous work on hybrid model-based and model-free reinforcement learning. Specifically, it expands on the ideas in Model-based Value Expansion (MVE) and Stochastic Ensemble Value Expansion (STEVE) with a dynamically-scaled variance bias term to increase risk aversion over the course of learning, which the authors call Risk Averse Value Expansion (RAVE). Experimental results indicate notable improvements over their selected model-free and hybrid RL baselines on continuous control tasks in terms of initial learning efficiency (how many environment steps are needed to achieve a particular level of performance), asymptotic performance (how high the performance is given the same large number of environment steps), and avoidance of negative outcomes (how infrequently major negative outcomes are encountered over the course of training).
Review
The core contribution of the paper is an extension to STEVE that uses an idea from risk-averse RL of biasing the underlying estimators away from high-variance predictions, and adds a dynamic weight to that bias.
Strengths
- The addition of a risk-aversion term to STEVE is a good contribution to the literature on safe RL. While it may be possible to criticize this contribution as somewhat trivial, I am disinclined to do so, as finding simple ideas that are effective is the hallmark of an important contribution, in my opinion.
- Under the assumption that the baselines are fair, the empirical results show substantial improvements for a good selection of challenging tasks along three metrics: initial efficiency, asymptotic performance, and avoidance of negative outcomes.
- The paper provides a careful and detailed section on the relevant preliminaries, giving precise notation that is expanded step-by-step from basic actor-critic approaches all the way through the description of RAVE. (Although see my comment below about possibly making the notation more concise.)
- Overall, the paper’s presentation is clear and natural. (Although see my comment below about a strong need for proofreading.)
Weaknesses
- The paper opens with a claim that MBRL has worse performance than MFRL in environments with “noisy environments and long trajectories”. However, recent work [1] has shown that MBRL can outperform MFRL on many of the tasks considered in this paper using far fewer environment steps, even when training directly from pixels.
- The experiments are all based on observations of the true state vectors rather than pixels, which dramatically simplifies working with imagination rollouts, as the dimensionality is so much lower. This hides the high computational cost and modeling challenges involved in using imagination-based MBRL or hybrid RL approaches in real-world environments, and (in my opinion) is a major shortcoming of this line of research (not only of this paper). Since the DDPG family of algorithms doesn’t have to do rollouts, they have a strong advantage over this type of approach on more realistic settings where the true state is unknown and must be inferred from high-dimensional observations. If experiments are not going to be done using pixels, the discussion should at least mention the trade-offs involved between the proposed algorithm and the baselines in that setting.
- The core baseline of the paper is DDPG, which is unnecessarily weak. D4PG [2] came out a year and a half ago and has the same high-level properties as DDPG, but outperforms it on all of the Mujoco tasks considered in this paper. Additionally, it includes some of the ideas presented in this work, but in a model-free setting, including a distributional treatment and multi-step rollouts, so it is easy to imagine that some of the experimental gains presented here would be erased when using it as a baseline.
- The use of ensembling is another confounder in the experiments. Ensembling models almost always yields an improvement, so any technique that relies on some form of ensembling needs to additionally demonstrate that the gains presented are not solely due to ensembling. In this case, the experiments with STEVE and RAVE should minimally be performed with different numbers of models in their ensembles.
- (Minor) Similarly, the prediction horizon can have a large impact, and only STEVE and RAVE consider prediction horizons other than 1. Showing how the horizon affects performance would help make the comparisons more fair.
- Figure 1 shows a bias problem in some current approaches in a clear toy problem, but does not show whether RAVE addresses that problem. Even though this section precedes the presentation of RAVE, it is necessary to show that the proposed solution actually helps on the toy problem.
- There are a number of problems with the proposal for adaptively computing the alpha parameter. In general, such adaptive hyperparameters add a great deal of complexity to hyperparameter tuning, so such suggestions should be either strongly motivated by theory or by empirical results. Neither seems to be the case here.
  - Only one proposal for adaptive alpha (equation 13) is considered. Its justification is plausible, but it would be more convincing if other dynamic approaches were considered in the experiments. For example:
    - alpha(env_step) = alpha * env_step / max_env_step
    - alpha(s_t,a_t) = alpha * min{1.0, 1 / (Z * ||E[fhat(s_t,a_t)] - s_t+1||^2)}
    - The opposite of the proposed approach, where alpha starts high and gets lower over the course of training.
  - Indeed, a potentially useful quantity during evaluation, when s_t+1 is unknown, would be based on the variance of the predicted next state, rather than the difference of the expectation of the predicted next state and the true next state. E.g.:
    - alpha(s_t,a_t) = alpha / Var[fhat(s_t,a_t)]
    This formulation says that the precision of the prediction determines the confidence of the model, which is also intuitively reasonable (to me, anyway) and doesn’t rely on knowing the future.
  - (Minor) The adaptive variant appears to be only slightly better than alpha=0.5, but requires two hyperparameters that are unspecified in the main body of the paper — alpha and Z. The appendix lists those parameters, but no discussion is made on how much tuning was done to determine that pair of parameters.
  - (Minor) A plot of how the dynamic value of alpha changes during training would be useful, at least in the appendix.
Recommendation
In light of my comments above, I cannot currently recommend the acceptance of this paper at ICLR. However, I think that the core idea is likely to hold up under more careful experimental comparisons. If the authors submit a revised draft that addresses the substance of my concerns, I would be very likely to increase my rating. In particular, I would like to see much more careful experimental treatment of the idea, so that readers can have high confidence about the circumstances where RAVE is likely to be a good choice.
[1] Hafner et al., “Learning Latent Dynics for Planning from Pixels”. ICML 2019. https://arxiv.org/abs/1811.04551
[2] Barth-Maron et al., “Distributed Distributional Deterministic Policy Gradients”. ICLR 2018. https://arxiv.org/abs/1804.08617
Other Comments and Questions
- This paper needs a lot of proofreading. A few examples of errors that should be fixed before publication:
  - “equation. 2”: This should read “Equation 2”, “equation 2”, “Eq. 2” or “eq. 2”. The period in the last two options indicates that letters have been elided. The mistake of using a period where no elision has happened occurs throughout the paper.
  - “prophesy”: This is not the correct word. Just say “prediction”.
  - “image”: This is often used when the correct word would be either “imagine” or “imagination”.
  There are many other errors that could be fixed easily with the help of a native English speaker.
- The mathematical notation in sections 3 through 5 is precise, but it is also a bit heavy. Consider whether there would be any ambiguity added if, for example, {\hat Q}^DVE_{\zeta_s,\zeta_r,\zeta_d,\theta’,\phi’} were instead notated {\hat Q}^DVE_{\zeta,\theta’,\phi’}.
- Figure 2 adds nothing of value to the paper and should be removed.
- Consider comparing on DMControl, which is the same set of tasks as Mujoco, but the scores are standardized such that each task has a maximum reward of 1000 per episode.
- Why does DDPG get worse over time on the RoboSchool tasks? Without a clear explanation, it looks like a bug, and a bug like that calls into question the rest of the DDPG results as well.