summaries: The paper presents a new approach for model-based offline RL. Here, the model is not used to estimate the return directly using roll-outs, but to generate synthetic data that is then used to train the Q-function. In a way, the method is a combination of CQL and MOPO. What is new is the idea of using this synthetic data to, penalize extrapolation instead of using uncertainty estimation. This modification performs much better on the selected benchmarks. As benchmarks a selection of D4RL benchmarks is used (3 benchmarks) and two vision based benchmarks.  This paper proposed a new model-based offline RL algorithm, COMBO. The algorithm applies model-based learning to be able to generate random rollouts, and the generated data is combined with the given fixed dataset for value function training. The paper shows by using the generated rollouts as data for Conservative Q-learning provides better constraint and better empirical performance compared to the orignal CQL algorithm.  This paper introduces a Conservative Offline Model-Based policy Optimization RL algorithm, called COMBO, that learns a pessimistic model by enabling a lower bound optimization on the policy performance without requiring uncertainty quantification. COMBO employs an actor-critic method to learn the value function on both offline and synthetic datasets. The idea is to penalize Q-values learned from state-action pairs that are out-of-support of the offline dataset.   This work investigates model-based reinforcement learning with offline dataset. In particular, most of the work on mbrl with offline datasets requires uncertainty estimation to determine the out-of-distribution states and actions. This work eliminates the need to have uncertainty estimation by using the existing Conservative Q-Learning(CQL) algorithm for states and actions from the dataset as well as the rollouts generated from the dynamics using the policy being learned. They show that inclusion of new dataset from the generated rollouts and by being pessimistic about their values, they tend to be better than existing Model-based offline Reinforcement Learning which requires uncertainty estimation
abstractText: Model-based reinforcement learning (RL) algorithms, which learn a dynamics model from logged experience and perform conservative planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating conservatism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We empirically find that uncertainty estimation is not accurate and leads to poor performance in certain scenarios in offline model-based RL. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that trains a value function using both the offline dataset and data generated using rollouts under the model while also additionally regularizing the value function on out-of-support state-action tuples generated via model rollouts. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. Theoretically, we show that COMBO satisfies a policy improvement guarantee in the offline setting. Through extensive experiments, we find that COMBO attains greater performance compared to prior offline RL on problems that demand generalization to related but previously unseen tasks, and also consistently matches or outperforms prior offline RL methods on widely studied offline RL benchmarks, including image-based tasks.
authors.name: Tianhe Yu
authors.name: Aviral Kumar
authors.name: Rafael Rafailov
authors.name: Aravind Rajeswaran
authors.name: Sergey Levine
authors.name: Chelsea Finn
references.authors: Alekh Agarwal Nan Jiang Sham M Kakade
references.title: Reinforcement learning: Theory and algorithms
references.venue: CS Dept., UW Seattle,
references.authors: Rishabh Agarwal Dale Schuurmans Mohammad Norouzi
references.title: An optimistic perspective on offline reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Arthur Argenson Gabriel Dulac-Arnold
references.title: Model-based offline planning
references.venue: arXiv preprint arXiv:2008.05556,
references.authors: Kamyar Azizzadenesheli Emma Brunskill Animashree Anandkumar
references.title: Efficient exploration through bayesian deep q-networks
references.venue: In ITA,
references.authors: Ignasi Clavera Violet Fu Pieter Abbeel
references.title: Model-augmented actor-critic: Backpropagating through paths
references.venue: arXiv preprint arXiv:2005.08068,
references.authors: Thomas Degris Martha White Richard S Sutton
references.title: Off-policy actor-critic
references.venue: arXiv preprint arXiv:1205.4839,
references.authors: Frederik Ebert Chelsea Finn Sudeep Dasari Annie Xie Alex Lee Sergey Levine
references.title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control
references.venue: arXiv preprint arXiv:1812.00568,
references.authors: Damien Ernst Pierre Geurts Louis Wehenkel
references.title: Tree-based batch mode reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Chelsea Finn Sergey Levine
references.title: Deep visual foresight for planning robot motion
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Justin Fu Aviral Kumar Ofir Nachum George Tucker Sergey Levine
references.title: D4rl: Datasets for deep data-driven reinforcement learning, 2020
references.authors: Scott Fujimoto Shixiang Shane Gu
references.title: A minimalist approach to offline reinforcement learning
references.venue: arXiv preprint arXiv:2106.06860,
references.authors: Scott Fujimoto David Meger Doina Precup
references.title: Off-policy deep reinforcement learning without exploration
references.venue: arXiv preprint arXiv:1812.02900,
references.authors: Scott Fujimoto Herke Van Hoof David Meger
references.title: Addressing function approximation error in actor-critic methods
references.venue: arXiv preprint arXiv:1802.09477,
references.authors: Seyed Kamyar Seyed Ghasemipour Dale Schuurmans Shixiang Shane Gu
references.title: Emaq: Expected-max q-learning operator for simple yet effective offline and online rl
references.venue: In International Conference on Machine Learning,
references.authors: Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine
references.title: Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor
references.venue: arXiv preprint arXiv:1801.01290,
references.authors: Thomas Jaksch Ronald Ortner Peter Auer
references.title: Near-optimal regret bounds for reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Michael Janner Justin Fu Marvin Zhang Sergey Levine
references.title: When to trust your model: Model-based policy optimization
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Natasha Jaques Asma Ghandeharioun Judy Hanwen Shen Craig Ferguson Agata Lapedriza Noah Jones Shixiang Gu Rosalind Picard
references.title: Way off-policy batch deep reinforcement learning of implicit human preferences in dialog
references.venue: arXiv preprint arXiv:1907.00456,
references.authors: Natasha Jaques Judy Hanwen Shen Asma Ghandeharioun Craig Ferguson Agata Lapedriza Noah Jones Shixiang Shane Gu Rosalind Picard
references.title: Human-centric dialog training via offline reinforcement learning
references.authors: Ying Jin Zhuoran Yang Zhaoran Wang
references.title: Is pessimism provably efficient for offline rl
references.venue: In International Conference on Machine Learning,
references.authors: Gregory Kahn Adam Villaflor Pieter Abbeel Sergey Levine
references.title: Composable actionconditioned predictors: Flexible off-policy learning for robot navigation
references.venue: In Conference on Robot Learning,
references.authors: Dmitry Kalashnikov Alex Irpan Peter Pastor Julian Ibarz Alexander Herzog Eric Jang Deirdre Quillen Ethan Holly Mrinal Kalakrishnan Vincent Vanhoucke
references.title: Scalable deep reinforcement learning for vision-based robotic manipulation
references.venue: In Conference on Robot Learning,
references.authors: Rahul Kidambi Aravind Rajeswaran Praneeth Netrapalli Thorsten Joachims
references.title: Morel: Model-based offline reinforcement learning
references.venue: arXiv preprint arXiv:2005.05951,
references.authors: Ilya Kostrikov Rob Fergus Jonathan Tompson Ofir Nachum
references.title: Offline reinforcement learning with fisher divergence critic regularization
references.venue: In International Conference on Machine Learning,
references.authors: Aviral Kumar Justin Fu Matthew Soh George Tucker Sergey Levine
references.title: Stabilizing off-policy q-learning via bootstrapping error reduction
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Aviral Kumar Aurick Zhou George Tucker Sergey Levine
references.title: Conservative q-learning for offline reinforcement learning
references.venue: arXiv preprint arXiv:2006.04779,
references.authors: Sascha Lange Thomas Gabel Martin A. Riedmiller
references.title: Batch reinforcement learning
references.venue: In Reinforcement Learning,
references.authors: Romain Laroche Paul Trichelair Remi Tachet Des Combes
references.title: Safe policy improvement with baseline bootstrapping
references.venue: In International Conference on Machine Learning,
references.authors: Alex X. Lee Anusha Nagabandi Pieter Abbeel Sergey Levine
references.title: Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Byung-Jun Lee Jongmin Lee Kee-Eung Kim
references.title: Representation balancing offline modelbased reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Sergey Levine Aviral Kumar George Tucker Justin Fu
references.title: Offline reinforcement learning: Tutorial, review, and perspectives on open problems
references.venue: arXiv preprint arXiv:2005.01643,
references.authors: Yao Liu Adith Swaminathan Alekh Agarwal Emma Brunskill
references.title: Off-policy policy gradient with state distribution correction
references.authors: Yao Liu Adith Swaminathan Alekh Agarwal Emma Brunskill
references.title: Provably good batch reinforcement learning without great exploration
references.venue: arXiv preprint arXiv:2007.08202,
references.authors: Kendall Lowrey Aravind Rajeswaran Sham Kakade Emanuel Todorov Igor Mordatch
references.title: Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Ajay Mandlekar Fabio Ramos Byron Boots Silvio Savarese Li Fei-Fei Animesh Garg Dieter Fox
references.title: Iris: Implicit reinforcement without interaction at scale for learning control from offline robot manipulation data
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Tatsuya Matsushima Hiroki Furuta Yutaka Matsuo Ofir Nachum Shixiang Gu
references.title: Deployment-efficient reinforcement learning via model-based offline optimization
references.venue: arXiv preprint arXiv:2006.03647,
references.authors: Rémi Munos Csaba Szepesvari
references.title: Finite-time bounds for fitted value iteration
references.venue: J. Mach. Learn. Res.,
references.authors: Ofir Nachum Bo Dai Ilya Kostrikov Yinlam Chow Lihong Li Dale Schuurmans
references.title: Algaedice: Policy gradient from arbitrary experience
references.authors: Ian Osband Benjamin Van Roy
references.title: Why is posterior sampling better than optimism for reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Ian Osband John Aslanides Albin Cassirer
references.title: Randomized prior functions for deep reinforcement learning
references.authors: Yaniv Ovadia Emily Fertig Jie Ren Zachary Nado David Sculley Sebastian Nowozin Joshua V Dillon Balaji Lakshminarayanan Jasper Snoek
references.title: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift
references.venue: arXiv preprint arXiv:1906.02530,
references.authors: Xue Bin Peng Aviral Kumar Grace Zhang Sergey Levine
references.title: Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
references.authors: Marek Petrik Yinlam Chow Mohammad Ghavamzadeh
references.title: Safe policy improvement by minimizing robust baseline regret
references.venue: arXiv preprint arXiv:1607.03842,
references.authors: Doina Precup Richard S Sutton Sanjoy Dasgupta
references.title: Off-policy temporal-difference learning with function approximation
references.venue: In ICML, pages 417–424,
references.authors: Rafael Rafailov Tianhe Yu A. Rajeswaran Chelsea Finn
references.title: Offline reinforcement learning from images with latent space models
references.venue: ArXiv, abs/2012.11547,
references.authors: Paria Rashidinejad Banghua Zhu Cong Ma Jiantao Jiao Stuart Russell
references.title: Bridging offline reinforcement learning and imitation learning: A tale of pessimism
references.venue: arXiv preprint arXiv:2103.12021,
references.authors: Martin Riedmiller
references.title: Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method
references.venue: In European Conference on Machine Learning,
references.authors: Stephane Ross Drew Bagnell
references.title: Agnostic system identification for model-based reinforcement learning
references.venue: In ICML,
references.authors: Susan M Shortreed Eric Laber Daniel J Lizotte T Scott Stroup Joelle Pineau Susan A Murphy
references.title: Informing sequential clinical decision-making through reinforcement learning: an empirical study
references.venue: Machine learning,
references.authors: Noah Y Siegel Jost Tobias Springenberg Felix Berkenkamp Abbas Abdolmaleki Michael Neunert Thomas Lampe Roland Hafner Martin Riedmiller
references.title: Keep doing what worked: Behavioral modelling priors for offline reinforcement learning
references.venue: arXiv preprint arXiv:2002.08396,
references.authors: Avi Singh Albert Yu Jonathan Yang Jesse Zhang Aviral Kumar Sergey Levine
references.title: Cog: Connecting new skills to past experience with offline reinforcement learning
references.authors: Samarth Sinha Animesh Garg
references.title: S4rl: Surprisingly simple self-supervision for offline reinforcement learning
references.venue: arXiv preprint arXiv:2103.06326,
references.authors: R.S. Sutton A.G. Barto
references.title: Reinforcement Learning: An Introduction
references.authors: Richard S Sutton
references.title: Dyna, an integrated architecture for learning, planning, and reacting
references.venue: ACM Sigart Bulletin,
references.authors: Richard S Sutton A Rupam Mahmood Martha White
references.title: An emphatic approach to the problem of off-policy temporal-difference learning
references.venue: The Journal of Machine Learning Research,
references.authors: Adith Swaminathan Thorsten Joachims
references.title: Batch learning from logged bandit feedback through counterfactual risk minimization
references.venue: J. Mach. Learn. Res,
references.authors: Phillip Swazinna Steffen Udluft Thomas Runkler
references.title: Overcoming model bias for robust offline deep reinforcement learning
references.venue: arXiv preprint arXiv:2008.05533,
references.authors: L. Wang Wei Zhang Xiaofeng He H. Zha
references.title: Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation
references.venue: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
references.authors: Yifan Wu George Tucker Ofir Nachum
references.title: Behavior regularized offline reinforcement learning
references.venue: arXiv preprint arXiv:1911.11361,
references.authors: Yue Wu Shuangfei Zhai Nitish Srivastava Joshua Susskind Jian Zhang Ruslan Salakhutdinov Hanlin Goh
references.title: Uncertainty weighted actor-critic for offline reinforcement learning
references.venue: arXiv preprint arXiv:2105.08140,
references.authors: F. Yu H. Chen X. Wang Wenqi Xian Yingying Chen Fangchen Liu V. Madhavan Trevor Darrell
references.title: Bdd100k: A diverse driving dataset for heterogeneous multitask learning
references.venue: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Tianhe Yu Deirdre Quillen Zhanpeng He Ryan Julian Karol Hausman Chelsea Finn Sergey Levine
references.title: Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
references.venue: In Conference on Robot Learning,
references.authors: Tianhe Yu Garrett Thomas Lantao Yu Stefano Ermon James Zou Sergey Levine Chelsea Finn Tengyu Ma
references.title: Mopo: Model-based offline policy optimization
references.venue: arXiv preprint arXiv:2005.13239,
references.authors: Xianyuan Zhan Xiangyu Zhu Haoran Xu
references.title: Model-based offline planning with trajectory pruning
references.venue: arXiv preprint arXiv:2105.07351,
references.authors: Wenxuan Zhou Sujay Bajracharya David Held. Plas
references.title: Latent action space for offline reinforcement learning
references.venue: arXiv preprint arXiv:2011.07213,
sections.heading: 1 Introduction
sections.text: Offline reinforcement learning (offline RL) [30, 34] refers to the setting where policies are trained using static, previously collected datasets. This presents an attractive paradigm for data reuse and safe policy learning in many applications, such as healthcare [62], autonomous driving [65], robotics [25, 48], and personalized recommendation systems [59]. Recent studies have observed that RL algorithms originally developed for the online or interactive paradigm perform poorly in the offline case [14, 28, 26]. This is primarily attributed to the distribution shift that arises over the course of learning between the offline dataset and the learned policy. Thus, development of algorithms specialized for offline RL is of paramount importance to benefit from the offline data available in aformentioned applications. In this work, we develop a principled model-based offline RL algorithm that matches or exceeds the performance of prior offline RL algorithms in benchmark tasks.
  A major paradigm for algorithm design in offline RL is to incorporate conservatism or regularization into online RL algorithms. Model-free offline RL algorithms [15, 28, 63, 21, 29, 27] directly incorporate conservatism into the policy or value function training and do not require learning a dynamics model. However, model-free algorithms learn only on the states in the offline dataset, which can lead to overly conservative algorithms. In contrast, model-based algorithms [26, 67] learn a pessimistic dynamics model, which in turn induces a conservative estimate of the value function. By generating and training on additional synthetic data, model-based algorithms have the potential for
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  broader generalization and solving new tasks using the offline dataset [67]. However, these methods rely on some sort of strong assumption about uncertainty estimation, typically assuming access to a model error oracle that can estimate upper bounds on model error for any state-action tuple. In practice, such methods use more heuristic uncertainty estimation methods, which can be difficult or unreliable for complex datasets or deep network models. It then remains an open question as to whether we can formulate principled model-based offline RL algorithms with concrete theoretical guarantees on performance without assuming access to an uncertainty or model error oracle. In this work, we propose precisely such a method, by eschewing direct uncertainty estimation, which we argue is not necessary for offline RL.
  Our main contribution is the development of conservative offline model-based policy optimization (COMBO), a new model-based algorithm for offline RL. COMBO learns a dynamics model using the offline dataset. Subsequently, it employs an actor-critic method where the value function is learned using both the offline dataset as well as synthetically generated data from the model, similar to Dyna [57] and a number of recent methods [20, 67, 7, 48]. However, in contrast to Dyna, COMBO learns a conservative critic function by penalizing the value function in state-action tuples that are not in the support of the offline dataset, obtained by simulating the learned model. We theoretically show that for any policy, the Q-function learned by COMBO is a lower-bound on the true Q-function. While
  the approach of optimizing a performance lower-bound is similar in spirit to prior model-based algorithms [26, 67], COMBO crucially does not assume access to a model error or uncertainty oracle. In addition, we show theoretically that the Q-function learned by COMBO is less conservative than model-free counterparts such as CQL [29], and quantify conditions under which the this lower bound is tighter than the one derived in CQL. This is illustrated through an example in Figure 1. Following prior works [31], we show that COMBO enjoys a safe policy improvement guarantee. By interpolating model-free and model-based components, this guarantee can utilize the best of both guarantees in certain cases. Finally, in our experiments, we find that COMBO achieves the best performance on tasks that require out-of-distribution generalization and outperforms previous latent-space offline model-based RL methods on image-based robotic manipulation benchmarks. We also test COMBO on commonly studied benchmarks for offline RL and find that COMBO generally performs well on the benchmarks, achieving the highest score in 9 out of 12 MuJoCo domains from the D4RL [12] benchmark suite.
sections.heading: 2 Preliminaries
sections.text: Markov Decision Processes and Offline RL. We study RL in the framework of Markov decision processes (MDPs) specified by the tupleM = (S,A, T, r, µ0, γ). S,A denote the state and action spaces. T (s′|s,a) and r(s,a) ∈ [−Rmax, Rmax] represent the dynamics and reward function respectively. µ0(s) denotes the initial state distribution, and γ ∈ (0, 1) denotes the discount factor. We denote the discounted state visitation distribution of a policy π using dπM(s) := (1− γ) ∑∞ t=0 γ
  tP(st = s|π), where P(st = s|π) is the probability of reaching state s at time t by rolling out π in M. Similarly, we denote the state-action visitation distribution with dπM(s,a) := d π M(s)π(a|s). The goal of RL is to learn a policy that maximizes the return, or long term cumulative rewards: maxπ J(M, π) := 11−γE(s,a)∼dπM(s,a)[r(s,a)].
  Offline RL is the setting where we have access only to a fixed dataset D = {(s,a, r, s′)}, which consists of transition tuples from trajectories collected using a behavior policy πβ . In other words, the dataset D is sampled from dπβ (s,a) := dπβ (s)πβ(a|s). We defineM as the empirical MDP induced by the dataset D and d(s,a) as sampled-based version of dπβ (s,a). In the offline setting, the goal is to find the best possible policy using the fixed offline dataset.
  Model-Free Offline RL Algorithms. One class of approaches for solving MDPs involves the use of dynamic programming and actor-critic schemes [56, 5], which do not explicitly require the learning
  of a dynamics model. To capture the long term behavior of a policy without a model, we define the action value function as Qπ(s,a) := E [ ∑∞ t=0 γ
  t r(st,at) | s0 = s,a0 = a] , where future actions are sampled from π(·|s) and state transitions happen according to the MDP dynamics. Consider the following Bellman operator: BπQ(s,a) := r(s,a) + γE s′∼T (·|s,a),a′∼π(·|s′) [Q(s′,a′)], and its sample based counterpart: B̂πQ(s,a) := r(s,a) + γQ(s′,a′), associated with a single transition (s,a, s′) and a′ ∼ π(·|s′). The action-value function satisfies the Bellman consistency criterion given by BπQπ(s,a) = Qπ(s,a) ∀(s,a). When given an offline dataset D, standard approximate dynamic programming (ADP) and actor-critic methods use this criterion to alternate between policy evaluation [40] and policy improvement. A number of prior works have observed that such a direct extension of ADP and actor-critic schemes to offline RL leads to poor results due to distribution shift over the course of learning and over-estimation bias in the Q function [14, 28, 63]. To address these drawbacks, prior works have proposed a number of modifications aimed towards regularizing the policy or value function (see Section 6). In this work, we primarily focus on CQL [29], which alternates between:
  Policy Evaluation: The Q function associated with the current policy π is approximated conservatively by repeating the following optimization:
  Qk+1←argmin Q
  β ( Es∼D,a∼µ(·|s)[Q(s,a)]−Es,a∼D[Q(s,a)] ) + 1
  2 Es,a,s′∼D
  [( Q(s,a)−B̂πQk(s,a) ) 2 ] , (1)
  where µ(·|s) is a wide sampling distribution such as the uniform distribution over action bounds. CQL effectively penalizes the Q function at states in the dataset for actions not observed in the dataset. This enables a conservative estimation of the value function for any policy [29], mitigating the challenges of over-estimation bias and distribution shift.
  Policy Improvement: After approximating the Q function as Q̂π, the policy is improved as π ← argmaxπ′Es∼D,a∼π′(·|s) [ Q̂π(s,a) ] . Actor-critic methods with parameterized policies and Q
  functions approximate argmax and argmin in above equations with a few gradient descent steps.
  Model-Based Offline RL Algorithms. A second class of algorithms for solving MDPs involve the learning of the dynamics function, and using the learned model to aid policy search. Using the given dataset D, a dynamics model T̂ is typically trained using maximum likelihood estimation as: minT̂ E(s,a,s′)∼D [ log T̂ (s′|s,a) ] . A reward model r̂(s,a) can also be learned similarly if it is un-
  known. Once a model has been learned, we can construct the learned MDP M̂ = (S,A, T̂ , r̂, µ0, γ), which has the same state and action spaces, but uses the learned dynamics and reward function. Subsequently, any policy learning or planning algorithm can be used to recover the optimal policy in the model as π̂ = argmaxπ J(M̂, π). This straightforward approach is known to fail in the offline RL setting, both in theory and practice, due to distribution shift and model-bias [51, 26]. In order to overcome these challenges, offline model-based algorithms like MOReL [26] and MOPO [67] use uncertainty quantification to construct a lower bound for policy performance and optimize this lower bound by assuming a model error oracle u(s,a). By using an uncertainty estimation algorithm like bootstrap ensembles [43, 4, 37], we can estimate u(s,a). By constructing and optimizing such a lower bound, offline model-based RL algorithms avoid the aforementioned pitfalls like model-bias and distribution shift. While any RL or planning algorithm can be used to learn the optimal policy for M̂, we focus specifically on MBPO [20, 57] which was used in MOPO. MBPO follows the standard structure of actor-critic algorithms, but in each iteration uses an augmented dataset D ∪Dmodel for policy evaluation. Here, D is the offline dataset and Dmodel is a dataset obtained by simulating the current policy using the learned dynamics model. Specifically, at each iteration, MBPO performs k-step rollouts using T̂ starting from state s ∈ D with a particular rollout policy µ(a|s), adds the model-generated data to Dmodel, and optimizes the policy with a batch of data sampled from D ∪Dmodel where each datapoint in the batch is drawn from D with probability f ∈ [0, 1] and Dmodel with probability 1− f .
sections.heading: 3 Conservative Offline Model-Based Policy Optimization
sections.text: The principal limitation of prior offline model-based algorithms (discussed in Section 2) is the assumption of having access to a model error oracle for uncertainty estimation and strong reliance on heuristics of quantifying the uncertainty. In practice, such heuristics could be challenging for complex datasets or deep neural network models [44]. We argue that uncertainty estimation is not
  Algorithm 1 COMBO: Conservative Model Based Offline Policy Optimization Require: Offline dataset D, rollout distribution µ(·|s), learned dynamics model T̂θ , initialized policy and critic
  πϕ and Qψ . 1: Train the probabilistic dynamics model T̂θ(s′, r|s,a) = N (µθ(s,a),Σθ(s,a)) on D. 2: Initialize the replay buffer Dmodel ← ∅. 3: for i = 1, 2, 3, · · · , do 4: Collect model rollouts by sampling from µ and T̂θ starting from states inD. Add model rollouts toDmodel. 5: Conservatively evaluate πiϕ by repeatedly solving eq. 2 to obtain Q̂ πiϕ ψ using samples from D ∪Dmodel. 6: Improve policy under state marginal of df by solving eq. 3 to obtain πi+1ϕ . 7: end for
  imperative for offline model-based RL and empirically show that uncertainty estimation could be inaccurate in offline RL problems especially when generalization to unknown behaviors is required in Section 5.1.1. Our goal is to develop a model-based offline RL algorithm that enables optimizing a lower bound on the policy performance, but without requiring uncertainty quantification. We achieve this by extending conservative Q-learning [29], which does not require explicit uncertainty quantification, into the model-based setting. Our algorithm COMBO, summarized in Algorithm 1, alternates between a conservative policy evaluation step and a policy improvement step, which we outline below.
  Conservative Policy Evaluation: Given a policy π, an offline dataset D, and a learned model of the MDP M̂, the goal in this step is to obtain a conservative estimate of Qπ . To achieve this, we penalize the Q-values evaluated on data drawn from a particular state-action distribution that is more likely to be out-of-support while pushing up the Q-values on state-action pairs that are trustworthy, which is implemented by repeating the following recursion:
  Q̂k+1←argmin Q
  β ( Es,a∼ρ(s,a)[Q(s,a)]−Es,a∼D[Q(s,a)] ) + 1
  2 Es,a,s′∼df
  [( Q(s,a)−B̂πQ̂k(s,a) )2] . (2)
  Here, ρ(s,a) and df are sampling distributions that we can choose. Model-based algorithms allow ample flexibility for these choices while providing the ability to control the bias introduced by these choices. For ρ(s,a), we make the following choice: ρ(s,a) = dπ
  M̂ (s)π(a|s), where dπ M̂ (s) is the
  discounted marginal state distribution when executing π in the learned model M̂. Samples from dπ M̂ (s) can be obtained by rolling out π in M̂. Similarly, df is an f−interpolation between the offline dataset and synthetic rollouts from the model: dµf (s,a) := f d(s,a) + (1 − f) d µ
  M̂ (s,a),
  where f ∈ [0, 1] is the ratio of the datapoints drawn from the offline dataset as defined in Section 2 and µ(·|s) is the rollout distribution used with the model, which can be modeled as π or a uniform distribution. To avoid notation clutter, we also denote df := d µ f .
  Under such choices of ρ and df , we push down (or conservatively estimate) Q-values on state-action tuples from model rollouts and push up Q-values on the real state-action pairs from the offline dataset. When updating Q-values with the Bellman backup, we use a mixture of both the model-generated data and the real data, similar to Dyna [57]. Note that in comparison to CQL and other model-free algorithms, COMBO learns the Q-function over a richer set of states beyond the states in the offline dataset. This is made possible by performing rollouts under the learned dynamics model, denoted by dµ M̂ (s,a). We will show in Section 4 that the Q function learned by repeating the recursion in Eq. 2 provides a lower bound on the true Q function, without the need for explicit uncertainty estimation. Furthermore, we will theoretically study the advantages of using synthetic data from the learned model, and characterize the impacts of model bias.
  Policy Improvement Using a Conservative Critic: After learning a conservative critic Q̂π, we improve the policy as:
  π′ ← argmax π
  Es∼ρ,a∼π(·|s) [ Q̂π(s,a) ] (3)
  where ρ(s) is the state marginal of ρ(s,a). When policies are parameterized with neural networks, we approximate the argmax with a few steps of gradient descent. In addition, entropy regularization can also be used to prevent the policy from becoming degenerate if required [17]. In Section 4.2, we show that the resulting policy is guaranteed to improve over the behavior policy.
  Practical Implementation Details. Our practical implementation largely follows MOPO, with the key exception that we perform conservative policy evaluation as outlined in this section, rather than using uncertainty-based reward penalties. Following MOPO, we represent the probabilistic dynamics model using a neural network, with parameters θ, that produces a Gaussian distribution over the next state and reward: T̂θ(st+1, r|s,a) = N (µθ(st,at),Σθ(st,at)). The model is trained via maximum likelihood. For conservative policy evaluation (eq. 2) and policy improvement (eq. 3), we augment ρ with states sampled from the offline dataset, which shows more stable improvement in practice. It is relatively common in prior work on model-based offline RL to select various hyperparameters using online policy rollouts [67, 26, 3, 33]. However, we would like to avoid this with our method, since requiring online rollouts to tune hyperparameters contradicts the main aim of offline RL, which is to learn entirely from offline data. Therefore, we do not use online rollouts for tuning COMBO, and instead devise an automated rule for tuning important hyperparameters such as β and f in a fully offline manner. We search over a small discrete set of hyperparameters for each task, and use the value of the regularization term Es,a∼ρ(s,a)[Q(s,a)]−Es,a∼D[Q(s, s)] (shown in Eq. 2) to pick hyperparameters in an entirely offline fashion. We select the hyperparameter setting that achieves the lowest regularization objective, which indicates that the Q-values on unseen model-predicted state-action tuples are not overestimated. Additional details about the practical implementation and the hyperparameter selection rule are provided in Appendix B.1 and Appendix B.2 respectively.
sections.heading: 4 Theoretical Analysis of COMBO
sections.text: In this section, we theoretically analyze our method and show that it optimizes a lower-bound on the expected return of the learned policy. This lower bound is close to the actual policy performance (modulo sampling error) when the policy’s state-action marginal distribution is in support of the state-action marginal of the behavior policy and conservatively estimates the performance of a policy otherwise. By optimizing the policy against this lower bound, COMBO guarantees policy improvement beyond the behavior policy. Furthermore, we use these insights to discuss cases when COMBO is less conservative compared to model-free counterparts.
sections.heading: 4.1 COMBO Optimizes a Lower Bound
sections.text: We first show that training the Q-function using Eq. 2 produces a Q-function such that the expected off-policy policy improvement objective [8] computed using this learned Q-function lower-bounds its actual value. We will reuse notation for df and d from Sections 2 and 3. Assuming that the Q-function is tabular, the Q-function found by approximate dynamic programming in iteration k, can be obtained by differentiating Eq. 2 with respect to Qk (see App. A for details):
  Q̂k+1(s,a) = (B̂πQk)(s,a)− β ρ(s,a)− d(s,a) df (s,a) . (4)
  Eq. 4 effectively applies a penalty that depends on the three distributions appearing in the COMBO critic training objective (Eq. 2), of which ρ and df are free variables that we choose in practice as discussed in Section 3. For a given iteration k of Eq. 4, we further define the expected penalty under ρ(s,a) as:
  ν(ρ, f) := Es,a∼ρ(s,a) [ ρ(s,a)− d(s,a)
  df (s,a)
  ] . (5)
  Next, we will show that the Q-function learned by COMBO lower-bounds the actual Q-function under the initial state distribution µ0 and any policy π. We also show that the asymptotic Q-function learned by COMBO lower-bounds the actual Q-function of any policy π with high probability for a large enough β ≥ 0, which we include in Appendix A.2. LetM represent the empirical MDP which uses the empirical transition model based on raw data counts. The Bellman backups over the dataset distribution df in Eq. 2 that we analyze is an f−interpolation of the backup operator in the empirical MDP (denoted by BπM) and the backup operator under the learned model M̂ (denoted by B π M̂
  ). The empirical backup operator suffers from sampling error, but is unbiased in expectation, whereas the model backup operator induces bias but no sampling error. We assume that all of these backups enjoy concentration properties with concentration coefficient Cr,T,δ , dependent on the desired confidence value δ (details in Appendix A.2). This is a standard assumption in literature [31]. Now, we state our main results below. Proposition 4.1. For large enough β, we have Es∼µ0,a∼π(·|s)[Q̂π(s,a)] ≤ Es∼µ0,a∼π(·|s)[Qπ(s,a)], where µ0(s) is the initial state distribution. Furthermore, when ϵs is small, such as in the large
  sample regime, or when the model bias ϵm is small, a small β is sufficient to guarantee this condition along with an appropriate choice of f .
  The proof for Proposition 4.1 can be found in Appendix A.2. Finally, while Kumar et al. [29] also analyze how regularized value function training can provide lower bounds on the value function at each state in the dataset [29] (Proposition 3.1-3.2), our result shows that COMBO is less conservative in that it does not underestimate the value function at every state in the dataset like CQL (Remark 1) and might even overestimate these values. Instead COMBO penalizes Q-values at states generated via model rollouts from ρ(s,a). Note that in general, the required value of β may be quite large similar to prior works, which typically utilize a large constant β, which may be in the form of a penalty on a regularizer [36, 29] or as constants in theoretically optimal algorithms [23, 49]. While it is challenging to argue that that either COMBO or CQL attains the tightest possible lower-bound on return, in our final result of this section, we discuss a sufficient condition for the COMBO lower-bound to be tighter than CQL.
  Proposition 4.2. Assuming previous notation, let ∆πCOMBO := Es,a∼dM(s),π(a|s) [ Q̂π(s,a) ] and
  ∆πCQL := Es,a∼dM(s),π(a|s) [ Q̂πCQL(s,a) ] denote the average values on the dataset under the Qfunctions learned by COMBO and CQL respectively. Then, ∆πCOMBO ≥ ∆πCQL, if:
  Es,a∼ρ(s,a) [ π(a|s) πβ(a|s) ] − Es,a∼dM(s),π(a|s) [ π(a|s) πβ(a|s) ] ≤ 0. (∗)
  Proposition 4.2 indicates that COMBO will be less conservative than CQL when the action probabilities under learned policy π(a|s) and the probabilities under the behavior policy πβ(a|s) are closer together on state-action tuples drawn from ρ(s,a) (i.e., sampled from the model using the policy π(a|s)), than they are on states from the dataset and actions from the policy, dM(s)π(a|s). COMBO’s objective (Eq. 2) only penalizes Q-values under ρ(s,a), which, in practice, are expected to primarily consist of out-of-distribution states generated from model rollouts, and does not penalize the Q-value at states drawn from dM(s). As a result, the expression (∗) is likely to be negative, making COMBO less conservative than CQL.
sections.heading: 4.2 Safe Policy Improvement Guarantees
sections.text: Now that we have shown various aspects of the lower-bound on the Q-function induced by COMBO, we provide policy improvement guarantees for the COMBO algorithm. Formally, Proposition 4.3 discuss safe improvement guarantees over the behavior policy. building on prior work [46, 31, 29]. Proposition 4.3 (ζ-safe policy improvement). Let π̂out(a|s) be the policy obtained by COMBO. Then, if β is sufficiently large and ν(ρπ, f)− ν(ρβ , f) ≥ C for a positive constant C, the policy π̂out(a|s) is a ζ-safe policy improvement over πβ in the actual MDPM, i.e., J(π̂out,M) ≥ J(πβ ,M) − ζ, with probability at least 1− δ, where ζ is given by,
  O ( γf
  (1− γ)2 ) E s∼dπ̂outM [√ |A| |D(s)|DCQL(π̂out, πβ) ] ︸ ︷︷ ︸
  := (1)
  +O ( γ(1− f) (1− γ)2 ) DTV(M,M̂)︸ ︷︷ ︸
  := (2)
  −β C (1− γ)︸ ︷︷ ︸ := (3) .
  The complete statement (with constants and terms that grow smaller than quadratic in the horizon) and proof for Proposition 4.3 is provided in Appendix A.4. DCQL denotes a notion of probabilistic distance between policies [29] which we discuss further in Appendix A.4. The expression for ζ in Proposition 4.3 consists of three terms: term (1) captures the decrease in the policy performance due to limited data, and decays as the size of D increases. The second term (2) captures the suboptimality induced by the bias in the learned model. Finally, as we show in Appendix A.4, the third term (3) comes from ν(ρπ, f)− ν(ρβ , f), which is equivalent to the improvement in policy performance as a result of running COMBO in the empirical and model MDPs. Since the learned model is trained on the dataset D with transitions generated from the behavior policy πβ , the marginal distribution ρβ(s,a) is expected to be closer to d(s,a) for πβ as compared to the counterpart for the learned policy, ρπ . Thus, the assumption that ν(ρπ, f)− ν(ρβ , f) is positive is reasonable, and in such cases, an appropriate (large) choice of β will make term (3) large enough to counteract terms (1) and (2) that reduce policy performance. We discuss this elaborately in Appendix A.4 (Remark 3).
  Further note that in contrast to Proposition 3.6 in Kumar et al. [29], note that our result indicates the sampling error (term (1)) is reduced (multiplied by a fraction f ) when a near-accurate model is used to augment data for training the Q-function, and similarity, it can avoid the bias of model-based methods by relying more on the model-free component. This allows COMBO to attain the best-of-both model-free and model-based methods, via a suitable choice of the fraction f .
  To summarize, through an appropriate choice of f , Proposition 4.3 guarantees safe improvement over the behavior policy without requiring access to an oracle uncertainty estimation algorithm.
sections.heading: 5 Experiments
sections.text: In our experiments, we aim to answer the follow questions: (1) Can COMBO generalize better than previous offline model-free and model-based approaches in a setting that requires generalization to tasks that are different from what the behavior policy solves? (2) How does COMBO compare with prior work in tasks with high-dimensional image observations? (3) How does COMBO compare to prior offline model-free and model-based methods in standard offline RL benchmarks?
  To answer those questions, we compare COMBO to several prior methods. In the domains with compact state spaces, we compare with recent model-free algorithms like BEAR [28], BRAC [63], and CQL [29]; as well as MOPO [67] and MOReL [26] which are two recent model-based algorithms. In addition, we also compare with an offline version of SAC [17] (denoted as SAC-off), and behavioral cloning (BC). In high-dimensional image-based domains, which we use to answer question (3), we compare to LOMPO [48], which is a latent space offline model-based RL method that handles image inputs, latent space MBPO (denoted LMBPO), similar to Janner et al. [20] which uses the model to generate additional synthetic data, the fully offline version of SLAC [32] (denoted SLAC-off), which only uses a variational model for state representation purposes, and CQL from image inputs. To our knowledge, CQL, MOPO, and LOMPO are representative of state-of-the-art model-free and model-based offline RL methods. Hence we choose them as comparisons to COMBO. To highlight the distinction between COMBO and a naïve combination of CQL and MBPO, we perform such a comparison in Table 8 in Appendix C. For more details of our experimental set-up, comparisons, and hyperparameters, see Appendix B.
sections.heading: 5.1 Results on tasks that require generalization
sections.text: To answer question (1), we use two environments halfcheetah-jump and ant-angle constructed in Yu et al. [67], which requires the agent to solve a task that is different from what the behavior policy solved. In both environments, the offline dataset is
  collected by policies trained with original reward functions of halfcheetah and ant, which reward the robots to run as fast as possible. The behavior policies are trained with SAC with 1M steps and we take the full replay buffer as the offline dataset. Following Yu et al. [67], we relabel rewards in the offline datasets to reward the halfcheetah to jump as high as possible and the ant to run to the top corner with a 30 degree angle as fast as possible. Following the same manner, we construct a third task sawyer-door-close based on the environment in Yu et al. [66], Rafailov et al. [48]. In this task, we collect the offline data with SAC policies trained with a sparse reward function that only gives a reward of 1 when the door is opened by the sawyer robot and 0 otherwise. The offline dataset is similar to the “medium-expert“ dataset in the D4RL benchmark since we mix equal amounts of data collected by a fully-trained SAC policy and a partially-trained SAC policy. We relabel the reward such that it is 1 when the door is closed and 0 otherwise. Therefore, in these datasets, the offline RL methods must generalize beyond behaviors in the offline data in order to learn the intended behaviors. We visualize the sawyer-door-close environment in the right image in Figure 3 in Appendix B.4.
  We present the results on the three tasks in Table 1. COMBO significantly outperforms MOPO, MOReL and CQL, two representative model-based methods and one representative model-free methods respectively, in the halfcheetah-jump and sawyer-door-close tasks, and achieves an approximately 8%, 4% and 12% improvement over MOPO, MOReL and CQL respectively on the ant-angle task. These results validate that COMBO achieves better generalization results in practice by behaving less conservatively than prior model-free offline methods (compare to CQL, which doesn’t improve much), and does so more robustly than prior model-based offline methods (compare to MOReL and MOPO).
sections.heading: 5.1.1 Empirical analysis on uncertainty estimation in offline model-based RL
sections.text: To further understand why COMBO outperforms prior model-based methods in tasks that require generalization, we argue that one of the main reasons could be that uncertainty estimation is hard in these tasks where the agent is required to go further away from the data distribution. To test this intuition, we perform empirical evaluations to study whether uncertainty quantification with deep neural networks, especially in the setting of dynamics model learning, is challenging and could cause problems with uncertainty-based model-based offline RL methods such as
  MOReL [26] and MOPO [67]. In our evaluations, we consider maximum learned variance over the ensemble (denoted as Max Var) maxi=1,...,N ∥Σiθ(s,a)∥F (used in MOPO). We consider two tasks halfcheetah-jump and ant-angle. We normalize both the model error and the uncertainty estimates to be within scale [0, 1] and performs linear regression that learns the mapping between the uncertainty estimates and the true model error. As shown in Figure 2, on both tasks, Max Var is unable to accurately predict the true model error, suggesting that uncertainty estimation used by offline model-based methods is not accurate and might be the major factor that results in its poor performance. Meanwhile, COMBO circumvents challenging uncertainty quantification problem and achieves better performances on those tasks, indicating the effectiveness and the robustness of the method.
sections.heading: 5.2 Results on image-based tasks
sections.text: To answer question (2), we evaluate COMBO on two image-based environments: the standard walker (walker-walk) task from the the DeepMind Control suite [61] and a visual door opening environment with a Sawyer robotic arm (sawyer-door) as used in Section 5.1.
  For the walker task we construct 4 datasets: medium-replay (M-R), medium (M), medium-expert (ME), and expert, similar to Fu et al. [12], each consisting of 200 trajectories. For sawyer-door task we use only the medium-expert and the expert datasets, due to the sparse reward – the agent is rewarded only when it successfully opens the door. Both environments are visulized in Figure 3 in Appendix B.4. To extend
  COMBO to the image-based setting, we follow Rafailov et al. [48] and train a recurrent variational model using the offline data and use train COMBO in the latent space of this model. We present
  results in Table 2. On the walker-walk task, COMBO performs in line with LOMPO and previous methods. On the more challenging Sawyer task, COMBO matches LOMPO and achieves 100% success rate on the medium-expert dataset, and substantially outperforms all other methods on the narrow expert dataset, achieving an average success rate of 96.7%, when all other model-based and model-free methods fail.
sections.heading: 5.3 Results on the D4RL tasks
sections.text: Finally, to answer the question (3), we evaluate COMBO on the OpenAI Gym [6] domains in the D4RL benchmark [12], which contains three environments (halfcheetah, hopper, and walker2d) and four dataset types (random, medium, medium-replay, and medium-expert). We include the results in Table 3. The numbers of BC, SAC-off, BEAR, BRAC-P and BRAC-v are taken from the D4RL paper, while the results for MOPO, MOReL and CQL are based on their respective papers [67, 29]. COMBO achieves the best performance in 9 out of 12 settings and comparable result in 1 out of the remaining 3 settings (hopper medium-replay). As noted by Yu et al. [67] and Rafailov et al. [48], model-based offline methods are generally more performant on datasets that are collected by a wide range of policies and have diverse state-action distributions (random, medium-replay datasets) while model-free approaches do better on datasets with narrow distributions (medium, medium-expert datasets). However, in these results, COMBO generally performs well across dataset types compared to existing model-free and model-based approaches, suggesting that COMBO is robust to different dataset types.
sections.heading: 6 Related Work
sections.text: Offline RL [10, 50, 30, 34] is the task of learning policies from a static dataset of past interactions with the environment. It has found applications in domains including robotic manipulation [25, 38, 48, 54], NLP [21, 22] and healthcare [52, 62]. Similar to interactive RL, both model-free and model-based algorithms have been studied for offline RL, with explicit or implicit regularization of the learning algorithm playing a major role.
  Model-free offline RL. Prior model-free offline RL algorithms have been designed to regularize the learned policy to be “close“ to the behavioral policy either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], offline actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantification to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29, 55] or explicitly measured by direct state or action constraints [14, 36], KL divergence [21, 63, 69], Wasserstein distance, MMD [28] and auxiliary imitation loss [13]. Different from these works, COMBO uses both the offline dataset as well as model-generated data.
  Model-based offline RL. Model-based offline RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling. Such methods however rely either on uncertainty quantification of the learned dynamics model which can be difficult for deep network models [44], or on directly constraining the policy towards the behavioral policy similar to model-free algorithms [39]. In contrast, COMBO conservatively estimates the value function by penalizing it in out-of-support states generated through model rollouts. This allows COMBO to
  retain all benefits of model-based algorithms such as broad generalization, without the constraints of explicit policy regularization or uncertainty quantification.
sections.heading: 7 Conclusion
sections.text: In the paper, we present conservative offline model-based policy optimization (COMBO), a modelbased offline RL algorithm that penalizes the Q-values evaluated on out-of-support state-action pairs. In particular, COMBO removes the need of uncertainty quantification as widely used in previous model-based offline RL works [26, 67], which can be challenging and unreliable with deep neural networks [44]. Theoretically, we show that COMBO achieves less conservative Q values compared to prior model-free offline RL methods [29] and guarantees a safe policy improvement. In our empirical study, COMBO achieves the best generalization performances in 3 tasks that require adaptation to unseen behaviors. Moreover, COMBO is able scale to vision-based tasks and outperforms or obtain comparable results in vision-based locomotion and robotic manipulation tasks. Finlly, on standard D4RL benchmark, COMBO generally performs well across dataset types compared to prior methods Despite the advantages of COMBO, there are few challenges left such as the lack of an offline hyperparameter selection scheme that can yield a uniform hyperparameter across different datasets and an automatically selected f conditioned on the model error. We leave them for future work.
  Acknowledgments and Disclosure of Funding We thank members of RAIL and IRIS for their support and feedback. This work was supported in part by ONR grants N00014-20-1-2675 and N00014-21-1-2685 as well as Intel Corporation. AK and SL are supported by the DARPA Assured Autonomy program. AR was supported by the J.P. Morgan PhD Fellowship in AI.
title: COMBO: Conservative Offline Model-Based Policy Optimization