Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning
Model-based Reinforcement Learning(RL) has shown great advantage in sample-efficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.
In this work , the authors combine model-based value expansion(MVE) with model-free reinforcement learning and also take into account the high-order stochastic characteristics of the environment to make the value expansion algorithm in modeling error propagation of dynamics risk-averse. They propose a novel Hybrid-RL method, namely the Risk Averse Value Expansion(RAVE), that uses an ensemble of probabilistic dynamics models to generate imaginative rollouts and to model risk aversion of risks by estimating the lower confidence bound of the ensemble. They showed that by taking the risk (in terms of variance) of the dynamic propagation error into the account, RAVE can achieve comparable performance with other state-of-the-art baselines in benchmark experiments including MuJoCo and robo-school. Also they showed that RAVE reduced the variance of the return and thus prevented catastrophic consequences such as falling. I found this work interesting as the authors try to take the uncertainty of probabilistic dynamics model into account for estimating the value function and its confidence bounds in model-free RL. Utilizing such confidence interval of values, they can have a way of doing exploration while being risk-averse. While similar approaches of modeling the mean and variance of Q functions can be found in some existing work, such as boot-strapped DQN (Osband*16), and efficient exploration via Bayesian DQN (Azizzadenesheli*18), none of these work model the variance of the Q-function using the error propagation of the dynamics model. Through extensive experiments, it has also demonstrated the effectiveness of RAVE in terms of achieving good performance, approximation error in value estimation, as well as robustness to failure in standard mujoco benchmarks, which provides readers some detailed understanding on how this risk-averse uncertainty modeling/propagation technique helps in exploration. However, while i found this idea interesting, it appears to me that the current work is still quite preliminary (without theoretical concentration inequality based variance bounds for guiding exploration), and empirically, it would be great to compare this method with the aforementioned bootstrapping/bayesian based approaches. My other major comment is on claiming the proposed method to be *risk-averse*, because in RL, risk-averse methods are commonly known to not only optimize the expected return but also to provide guarantees to other moments of the return, such as variance of CVaR. However, while the method used variance of the expected value (due to the error propagation of the dynamics) for exploration, I am not seeing the risk-averse optimization criteria being studied here. Therefore calling the RL method risk-averse might be a little mis-leading.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Risk Averse Value Expansion for Sample Efficient and Robust Policy Learning
Model-based Reinforcement Learning(RL) has shown great advantage in sample-efficiency, but suffers from poor asymptotic performance and high inference cost. A promising direction is to combine model-based reinforcement learning with model-free reinforcement learning, such as model-based value expansion(MVE). However, the previous methods do not take into account the stochastic character of the environment, thus still suffers from higher function approximation errors. As a result, they tend to fall behind the best model-free algorithms in some challenging scenarios. We propose a novel Hybrid-RL method, which is developed from MVE, namely the Risk Averse Value Expansion(RAVE). In the proposed method, we use an ensemble of probabilistic models for environment modeling to generate imaginative rollouts, based on which we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on different environments including MuJoCo and robo-school show that RAVE yields state-of-the-art performance. Also we found that it greatly prevented some catastrophic consequences such as falling down and thus reduced the variance of the rewards.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this work , the authors combine model-based value expansion(MVE) with model-free reinforcement learning and also take into account the high-order stochastic characteristics of the environment to make the value expansion algorithm in modeling error propagation of dynamics risk-averse. They propose a novel Hybrid-RL method, namely the Risk Averse Value Expansion(RAVE), that uses an ensemble of probabilistic dynamics models to generate imaginative rollouts and to model risk aversion of risks by estimating the lower confidence bound of the ensemble. They showed that by taking the risk (in terms of variance) of the dynamic propagation error into the account, RAVE can achieve comparable performance with other state-of-the-art baselines in benchmark experiments including MuJoCo and robo-school. Also they showed that RAVE reduced the variance of the return and thus prevented catastrophic consequences such as falling. I found this work interesting as the authors try to take the uncertainty of probabilistic dynamics model into account for estimating the value function and its confidence bounds in model-free RL. Utilizing such confidence interval of values, they can have a way of doing exploration while being risk-averse. While similar approaches of modeling the mean and variance of Q functions can be found in some existing work, such as boot-strapped DQN (Osband*16), and efficient exploration via Bayesian DQN (Azizzadenesheli*18), none of these work model the variance of the Q-function using the error propagation of the dynamics model. Through extensive experiments, it has also demonstrated the effectiveness of RAVE in terms of achieving good performance, approximation error in value estimation, as well as robustness to failure in standard mujoco benchmarks, which provides readers some detailed understanding on how this risk-averse uncertainty modeling/propagation technique helps in exploration. However, while i found this idea interesting, it appears to me that the current work is still quite preliminary (without theoretical concentration inequality based variance bounds for guiding exploration), and empirically, it would be great to compare this method with the aforementioned bootstrapping/bayesian based approaches. My other major comment is on claiming the proposed method to be *risk-averse*, because in RL, risk-averse methods are commonly known to not only optimize the expected return but also to provide guarantees to other moments of the return, such as variance of CVaR. However, while the method used variance of the expected value (due to the error propagation of the dynamics) for exploration, I am not seeing the risk-averse optimization criteria being studied here. Therefore calling the RL method risk-averse might be a little mis-leading.
assistant
In this work , the authors combine model-based value expansion(MVE) with model-free reinforcement learning and also take into account the high-order stochastic characteristics of the environment to make the value expansion algorithm in modeling error propagation of dynamics risk-averse. They propose a novel Hybrid-RL method, namely the Risk Averse Value Expansion(RAVE), that uses an ensemble of probabilistic dynamics models to generate imaginative rollouts and to model risk aversion of risks by estimating the lower confidence bound of the ensemble. They showed that by taking the risk (in terms of variance) of the dynamic propagation error into the account, RAVE can achieve comparable performance with other state-of-the-art baselines in benchmark experiments including MuJoCo and robo-school. Also they showed that RAVE reduced the variance of the return and thus prevented catastrophic consequences such as falling. I found this work interesting as the authors try to take the uncertainty of probabilistic dynamics model into account for estimating the value function and its confidence bounds in model-free RL. Utilizing such confidence interval of values, they can have a way of doing exploration while being risk-averse. While similar approaches of modeling the mean and variance of Q functions can be found in some existing work, such as boot-strapped DQN (Osband*16), and efficient exploration via Bayesian DQN (Azizzadenesheli*18), none of these work model the variance of the Q-function using the error propagation of the dynamics model. Through extensive experiments, it has also demonstrated the effectiveness of RAVE in terms of achieving good performance, approximation error in value estimation, as well as robustness to failure in standard mujoco benchmarks, which provides readers some detailed understanding on how this risk-averse uncertainty modeling/propagation technique helps in exploration. However, while i found this idea interesting, it appears to me that the current work is still quite preliminary (without theoretical concentration inequality based variance bounds for guiding exploration), and empirically, it would be great to compare this method with the aforementioned bootstrapping/bayesian based approaches. My other major comment is on claiming the proposed method to be *risk-averse*, because in RL, risk-averse methods are commonly known to not only optimize the expected return but also to provide guarantees to other moments of the return, such as variance of CVaR. However, while the method used variance of the expected value (due to the error propagation of the dynamics) for exploration, I am not seeing the risk-averse optimization criteria being studied here. Therefore calling the RL method risk-averse might be a little mis-leading.