summaries: This paper aims to improve the performance of RL in sparse reward settings via the guidance of sub-optimal demonstrations. The idea proposed by this work is to modify TRPO to restrict its updates to remain close to the behavior policy that generated the offline dataset, while decaying this constraint over time to enable the RL policy to improve upon the behavior policy in an online fashion. The authors do so by minimizing the KL divergence between the current RL policy and the behavior policy while not moving too far away from the current RL policy. The method is evaluated on openAI gym style tasks as well as a real robot navigation task.

summaries: This paper presents ‘LOGO’, an extension of the TRPO algorithm which enables additional learning guidance from offline, sub-optimal (possibly incomplete observation) demonstration data. By annealing away the learning contribution from the sub-optimal guidance policy data during training (with a learning schedule and corresponding hyper-parameter), LOGO utilises this data for guidance, rather than directly imitating it. Furthermore, because LOGO utilises the trust-region methodology, the authors are able to provide a theoretical analysis and lower bound on performance improvement each episode. The method shows promising performance on several MuJoCo continuous control tasks, as well as in a Gazebo TurtleBot simulation, which is also able to be transferred to a real-world robot.

summaries: This paper considers the problem of reinforcement learning with sparse reward functions obtained from offline demonstration. The authors propose a trust region policy optimization based algorithm with offline demonstration data for guidance. The proposed LOGO algorithm is proved to be efficient by a theoretical analysis showing the lower bound on the performance improvement. On benchmark datasets, the proposed algorithm performs better than the state-of-the-art approaches. An illustration is shown by implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance.

summaries: In a regime where we have access to offline behavior data from a suboptimal policy (a heuristic, human demonstrations, etc) we can use that data and "trust region"-based methods (TRPO in this work) to nudge exploration in the right direction by keeping the learnable policy close to the behavioral one. While there are several works exploring this general idea, the particular way presented in this work is novel and achieves better results on a set of MuJoCo environment. The approach is also shown to work on a physical robot (Turtlebot).

abstractText: A major challenge in real-world reinforcement learning (RL) is the sparsity of reward feedback. Often, what is available is an intuitive but sparse reward function that only indicates whether the task is completed partially or fully. However, the lack of carefully designed, fine grain feedback implies that most existing RL algorithms fail to learn an acceptable policy in a reasonable time frame. This is because of the large number of exploration actions that the policy has to perform before it gets any useful feedback that it can learn from. In this work, we address this challenging problem by developing an algorithm that exploits the offline demonstration data generated by a sub-optimal behavior policy for faster and efficient online RL in such sparse reward settings. The proposed algorithm, which we call the Learning Online with Guidance Offline (LOGO) algorithm, merges a policy improvement step with an additional policy guidance step by using the offline demonstration data. The key idea is that by obtaining guidance from not imitating the offline data, LOGO orients its policy in the manner of the sub-optimal policy, while yet being able to learn beyond and approach optimality. We provide a theoretical analysis of our algorithm, and provide a lower bound on the performance improvement in each learning episode. We also extend our algorithm to the even more challenging incomplete observation setting, where the demonstration data contains only a censored version of the true state observation. We demonstrate the superior performance of our algorithm over state-of-the-art approaches on a number of benchmark environments with sparse rewards and censored state. Further, we demonstrate the value of our approach via implementing LOGO on a mobile robot for trajectory tracking and obstacle avoidance, where it shows excellent performance.

authors:
Joshua Achiam
David Held
Aviv Tamar
Pieter Abbeel

title: Constrained policy optimization

venue: In Proceedings of the 34th International Conference on Machine Learning,

authors:
Robin Amsters
Peter Slaets

title: Turtlebot 3 as a robotics education platform. In Robotics in Education - Current Research and Innovations

venue: Proceedings of the 10th RiE, Advances in Intelligent Systems and Computing,

authors:
Scott Fujimoto
David Meger
Doina Precup

title: Off-policy deep reinforcement learning without exploration

venue: In International Conference on Machine Learning,

authors:
Yang Gao
Huazhe Xu
Ji Lin
Fisher Yu
Sergey Levine
Trevor Darrell

title: Reinforcement learning from imperfect demonstrations

venue: arXiv preprint arXiv:1802.05313,

authors:
Ian J. Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron C. Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In Advances in Neural Information Processing Systems,

authors:
Çaglar Gülçehre
Tom Le Paine
Bobak Shahriari
Misha Denil
Matt Hoffman
Hubert Soyer
Richard Tanburn
Steven Kapturowski
Neil C. Rabinowitz
Duncan Williams
Gabriel Barth-Maron
Ziyu Wang
Nando de Freitas
Worlds Team

title: Making efficient use of demonstrations to solve hard exploration problems

venue: In International Conference on Learning Representations,

authors:
Abhishek Gupta
Vikash Kumar
Corey Lynch
Sergey Levine
Karol Hausman

title: Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

venue: In Conference on Robot Learning,

authors:
Tuomas Haarnoja
Aurick Zhou
Pieter Abbeel
Sergey Levine

title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

authors:
Todd Hester
Matej Vecerı́k
Olivier Pietquin
Marc Lanctot
Tom Schaul
Bilal Piot
Dan Horgan
John Quan
Andrew Sendonaris
Ian Osband
Gabriel Dulac-Arnold
John P. Agapiou
Joel Z. Leibo
Audrunas Gruslys

title: Deep q-learning from demonstrations

venue: In AAAI Conference on Artificial Intelligence,

authors:
Jonathan Ho
Stefano Ermon

title: Generative adversarial imitation learning

authors:
Mingxuan Jing
Xiaojian Ma
Wenbing Huang
Fuchun Sun
Chao Yang
Bin Fang
Huaping Liu

title: Reinforcement learning from imperfect demonstrations under soft expert guidance

authors:
Sham M. Kakade
John Langford

title: Approximately optimal approximate reinforcement learning

venue: In International Conference Machine Learning,

authors:
Bingyi Kang
Zequn Jie
Jiashi Feng

title: Policy optimization with demonstrations

authors:
Beomjoon Kim
Amir-massoud Farahmand
Joelle Pineau
Doina Precup

title: Learning from limited demonstrations

authors:
Nathan P. Koenig
Andrew Howard

title: Design and use paradigms for gazebo, an open-source multi-robot simulator

venue: In IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
Aviral Kumar
Justin Fu
Matthew Soh
George Tucker
Sergey Levine

title: Stabilizing off-policy q-learning via bootstrapping error reduction

venue: Advances in Neural Information Processing Systems,

authors:
Gabriele Libardi
Gianni De Fabritiis
Sebastian Dittert

title: Guided exploration with proximal policy optimization using a single demonstration

authors:
Timothy P. Lillicrap
Jonathan J. Hunt
Alexander Pritzel
Nicolas Heess
Tom Erez
Yuval Tassa
David Silver
Daan Wierstra

title: Continuous control with deep reinforcement learning

venue: In International Conference on Learning Representations ICLR,

authors:
Ashvin Nair
Bob McGrew
Marcin Andrychowicz
Wojciech Zaremba
Pieter Abbeel

title: Overcoming exploration in reinforcement learning with demonstrations

venue: In IEEE International Conference on Robotics and Automation (ICRA),

authors:
Ashvin Nair
Murtaza Dalal
Abhishek Gupta
Sergey Levine

title: Accelerating online reinforcement learning with offline datasets

venue: arXiv preprint arXiv:2006.09359,

authors:
Andrew Y. Ng
Stuart J. Russell

title: Algorithms for inverse reinforcement learning

venue: In International Conference on Machine Learning (ICML),

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga
Alban Desmaison
Andreas Kopf
Edward Yang
Zachary DeVito
Martin Raison
Alykhan Tejani
Sasank Chilamkurthy
Benoit Steiner
Lu Fang
Junjie Bai
Soumith Chintala

title: Pytorch: An imperative style, high-performance deep learning library

authors:
Aravind Rajeswaran
Vikash Kumar
Abhishek Gupta
Giulia Vezzani
John Schulman
Emanuel Todorov
Sergey Levine

title: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

venue: In Robotics: Science and Systems,

authors:
Stéphane Ross
Geoffrey J. Gordon
Drew Bagnell

title: A reduction of imitation learning and structured prediction to no-regret online learning

venue: In International Conference on Artificial Intelligence and Statistics,

title: Learning from demonstration

venue: Advances in neural information processing systems,

authors:
John Schulman
Sergey Levine
Pieter Abbeel
Michael Jordan
Philipp Moritz

title: Trust region policy optimization

authors:
Noah Siegel
Jost Tobias Springenberg
Felix Berkenkamp
Abbas Abdolmaleki
Michael Neunert
Thomas Lampe
Roland Hafner
Nicolas Heess
Martin Riedmiller

title: Keep doing what worked: Behavior modelling priors for offline reinforcement learning

venue: In International Conference on Learning Representations (ICLR),

authors:
Umar Syed
Michael Bowling
Robert E Schapire

title: Apprenticeship learning using linear programming

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

authors:
Mel Vecerik
Todd Hester
Jonathan Scholz
Fumin Wang
Olivier Pietquin
Bilal Piot
Nicolas Heess
Thomas Rothörl
Thomas Lampe
Martin Riedmiller

title: Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards

venue: arXiv preprint arXiv:1707.08817,

authors:
Yifan Wu
George Tucker
Ofir Nachum

title: Behavior regularized offline reinforcement learning

venue: arXiv preprint arXiv:1911.11361,

authors:
Yueh-Hua Wu
Nontawat Charoenphakdee
Han Bao
Voot Tangkaratt
Masashi Sugiyama

title: Imitation learning from imperfect demonstration

authors:
Brian D. Ziebart
Andrew L. Maas
J. Andrew Bagnell
Anind K. Dey

title: Maximum entropy inverse reinforcement learning

venue: RL algorithms

heading: 1 INTRODUCTION

text:
Reinforcement Learning (RL) is being considered for application to a variety of real-world settings that have very large state-action spaces, but even under the availability of accurate simulators determining how best to explore the environment is a major challenge. Oftentimes, the problem setting is such that intuitively obvious reward functions are quite sparse. Such rewards typically correspond to the attainment of a particular task, such as a robot attaining designated way points, with little fine-grain reward feedback on the intermediate steps taken to complete these tasks. The sparsity of rewards implies that existing RL approaches often do not make much headway into learning viable policies unless provided with carefully hand-tuned reward feedback by a domain expert.
Many of these problems correspond to systems where data has been gathered over time using an empirically determined (sub-optimal) behavior policy, which contains information important to help bootstrap learning. An additional caveat is that this behavior data might only contain measurements of a subset of the true state. Is it possible to design an algorithm that has principled policy optimization for efficient learning, while also being able to explore by utilizing the data derived from a behavior policy? A natural candidate to begin with is the framework of policy gradient algorithms (Schulman et al., 2015; Lillicrap et al., 2016; Haarnoja et al., 2018), which performs really well in the dense reward setting. Can such a policy gradient algorithm be aligned with a behavior policy for guided exploration that is critical in the sparse reward setting?
In this work, we propose a principled approach for using the well known trust region policy optimization (TRPO) algorithm (Schulman et al., 2015) with offline demonstration data for guidance in a sparse reward setting. Our choice of the TRPO approach is motivated by its analytical tractability and superior performance. Our key insight is that we can utilize the trust region approach while being guided by the behavior data by virtue of two steps. The first step is identical to traditional TRPO to generate a candidate policy. In the second step, the objective is to find a policy closest to the behavior policy, subject to it being in the trust region of the candidate generated in the first step. Thus, the second step ensures that the policy chosen is always guided by the behavior policy, but the level of alignment with the behavior policy can be reduced by shrinking the trust region over time. We call our approach Learning Online with Guidance Offline (LOGO) to capture this two-step procedure. This principled approach enables us to provide analytical performance guarantees while achieving superior performance in a number of benchmark problems.
Our main results are as follows: (i) LOGO can guarantee a performance improvement as long as the behavior policy is able to offer an advantage. Reducing the alignment with the behavior policy over time allows us to extract this advantage and then smoothly move beyond to find near-optimal policies, (ii) we provide a generalized version of the Performance Difference Lemma (Kakade & Langford, 2002) wherein the stage reward function can depend on the policy itself, and use this result to determine a surrogate objective function for step 2 of LOGO that is straightforward to evaluate. This allows us to implement LOGO in the manner of two TRPO-like steps, enabling us to leverage the TRPO code base, (iii) we show on standard MuJoCo environments that LOGO trained with sparse rewards can attain nearly the same performance as an optimal algorithm trained with dense rewards. We go further and show that this excellent performance carries over to waypoint tracking by a robot over Gazebo simulations and real-world TurtleBot experiments, (iv) finally, we show that LOGO can also be used in the case where the demonstration data only contains a censored version of the true state (incomplete state information) by simply adding a projection to the available subset of the state space in step 2 of the algorithm. Again, we provide supporting evidence via excellence in both MuJoCo simulations, as well as obstacle avoidance by a TurtleBot that is trained with a behavior policy without a Lidar (censored state), but is tested with it (full state).

heading: 1.1 RELATED WORK

text:
Our work is mainly related to two RL research areas: Imitation Learning (IL): The goal of an IL algorithm is to imitate an (expert) policy using the demonstration data generated by that policy. Behavior cloning (BC) is a simple IL approach where the expert policy is estimated from the demonstration data using supervised learning. BC algorithms, however, suffer from the problem of distribution shift (Ross et al., 2011). Inverse reinforcement learning (IRL) algorithms (Ng & Russell, 2000; Ziebart et al., 2008) estimate a reward function from the demonstration data and solve a forward RL problem using this reward function. Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) avoids the reward estimation problem by formulating the IL problem as a distribution matching problem, and provides an implicit reward for the RL algorithm using a discriminator (Goodfellow et al., 2014). Most IL algorithms do not use reward feedback from the environment, and hence are restricted to the performance of the policy that generated the demonstration data. Our approach is different from pure IL, and we leverage online RL with (sparse) reward feedback for efficient learning.
Learning from Demonstration (LfD): The key idea of the LfD algorithms is to use demonstration data to aid online learning (Schaal et al., 1997). Many works propose to exploit demonstration data by adding it to the replay buffer with a prioritized replay mechanism to accelerate learning (Hester et al., 2018; Vecerik et al., 2017; Nair et al., 2018). Rajeswaran et al. (2018) combine a policy gradient algorithm with demonstration data by using a mix of behavior cloning and online RL fine tuning. Nair et al. (2020) propose AWAC algorithm to accelerate online RL by leveraging large amounts of offline data with associated rewards. Different from this, LOGO does not need the reward observations. Moreover, we give provable guarantees on the performance of LOGO whereas AWAC does not have any such provable guarantee (further details are provided in Appendix G). Kim et al. (2013) propose a framework to integrate LfD and approximate policy iteration by formulating a coupled constraint convex optimization problem, where the expert demonstrations define a set of linear constraints. This approach is, however, limited to small problems with a discrete action space. The closest to our work is the PofD algorithm proposed by Kang et al. (2018). PofD modifies the reward function by taking a weighted combination of the sparse reward from the online interaction and an implicit
reward obtained from the demonstration data using a discriminator. Very different from this, we propose an intuitive and principled approach of using the offline demonstration data for guiding the online exploration during the initial phase of learning. Our two step approach enables us to provide a rigorous performance guarantee for the proposed LOGO algorithm, and to leverage trust region-based approaches to solve high dimensional problems with even sparser settings. We provide an extensive comparison between our algorithm and PofD in Section 5.
Offline RL: Recently, there has been many interesting works in the area of offline RL (Kumar et al., 2019; Fujimoto et al., 2019; Siegel et al., 2020; Wu et al., 2019a) which focus on learning a policy using only the offline data without any online learning or online fine-tuning. Different from these, LOGO is an online RL algorithm (further details are provided in Appendix G).

heading: 2 PRELIMINARIES AND PROBLEM SETTING

heading: 2.1 PRELIMINARIES

text:
A Markov Decision Process (MDP) is denoted as a tuple < S,A, R, P, γ >, where S is the state space, A is the action space, R : S ×A → R is the reward function, P : S ×A× S → [0, 1] is the transition probability function with P (s′|s, a) giving the probability of transitioning to state s′ when action a is taken at state s, and γ ∈ (0, 1) is the discount factor. A policy π is a mapping from S to probability distribution over A, with π(s, a) specifying the probability of taking action a in state s. A policy π can generate state-action trajectory τ , where τ = (s0, a0, s1, a1, . . .), s0 ∼ µ, at ∼ π(st, ·), st+1 ∼ P (·|st, at) and µ is the initial state distribution. The infinite horizon discounted return of policy π is defined as JR(π) = Eτ∼π [ ∑∞ t=0 γ
tR(st, at)]. The goal of a reinforcement learning algorithm is to learn the optimal policy π? = arg maxπ JR(π).
The value function of a policy π defined as V πR (s) = Eτ∼π [ ∑∞ t=0 γ
tR(st, at)|s0 = s], is the expected cumulative discounted reward obtained by following the policy π starting from the state s. The action-value function of a policy π is defined similarly as QπR(s, a) = Eτ∼π [ ∑∞ t=0 γ
tR(st, at)|s0 = s, a0 = a]. The advantage function is defined as AπR(s, a) = QπR(s, a) − V πR (s). The discounted state visitation distribution for the policy π, denoted as dπ, is defined as dπ(s) = (1− γ) ∑∞ t=0 γ
tP(st = s|π), where the probability is defined with respect to the randomness induced by π, P and µ.
Definitions and Notations: The Kullback-Leibler (KL) divergence between two distribution p and q is defined as DKL(p, q) = ∑ x p(x) log p(x) q(x) . The average KL divergence between two polices π1 and π2 with respect to dπ1 is defined as Dπ1KL(π1, π2) = Es∼dπ1 [DKL(π1(s, ·), π2(s, ·))]. The maximum KL divergence between two polices π1 and π2 is defined as DmaxKL (π1, π2) = maxs DKL(π1(s, ·), π2(s, ·)). The total variation (TV) distance between two distribution p and q is defined asDTV(p, q) = (1/2) ∑ x |p(x)−q(x)|. The average TV distance and the maximum TV distance between two polices π1 and π2 are defined as Dπ1TV(π1, π2) = Es∼dπ1 [DTV(π1(s, ·), π2(s, ·))] and DmaxTV (π1, π2) = maxs DTV(π1(s, ·), π2(s, ·)), respectively.

heading: 2.2 PROBLEM SETTING

text:
As described in the introduction, our goal is to develop an algorithm that can exploit offline demonstration data generated using a sub-optimal behavior policy for faster and efficient online reinforcement learning in a sparse reward setting. Formally, we assume that the algorithm has access to the demonstration data generated by a sub-optimal behavior policy πb. We first consider the setting in which the demonstration data has complete state observation. In this setting, the demonstration data has the form D = {τ i}ni=1, where τ i = (si1, ai1, . . . , siT , aiT ), τi ∼ πb. Later, we also propose an extension to the incomplete observation setting in which only a censored version of the true state is available in demonstration data. More precisely, instead of the complete state observation sit, the demonstration data will only contain s̃it = o(s i t), where o(·) is a projection to a lower dimensional subspace. We represent the incomplete demonstration data as D̃ = {τ̃ i}ni=1, where τ̃ i = (s̃i1, ai1, . . . , s̃iT , aiT ). We make the following assumption about the behavior policy πb.
Assumption 1. In the initial episodes of learning, Ea∼πb [A πk R (s, a)] ≥ β > 0,∀s, where πk is the learning policy employed by the algorithm in the kth episode of learning.
Intuitively, the above assumption implies that taking action according to πb will provide a higher advantage than taking actions according to πk because Ea∼πk [A πk R (s, a)] = 0. This is a reasonable assumption, since the behavior policies currently in use in many systems are likely to perform much better than an untrained policy. We also note that a similar assumption is made by Kang et al. (2018) to formalize the notion of a useful behavior policy. We emphasize that πb need not be the optimal policy, and πb could be such that JR(πb) < JR(π?). Our proposed algorithm learns a policy that performs better than πb through online learning with guided exploration.

heading: 3 ALGORITHM AND PERFORMANCE GUARANTEE

heading: 3.1 ALGORITHM

text:
In this section, we describe our proposed LOGO algorithm. Each iteration of our algorithm consists of two steps, namely a policy improvement step and a policy guidance step. In the following, πk denotes the policy after k iterations.
Step 1: Policy Improvement: In this step, the LOGO algorithm performs a one step policy improvement using the Trust Region Policy Optimization (TRPO) approach (Schulman et al., 2015). This can be expressed as
πk+1/2 = arg max π
Es∼dπk ,a∼π [AπkR (s, a)] s.t. D πk KL(π, πk) ≤ δ. (1)
The TRPO update finds the policy πk+1/2 that maximizes the objective while constraining this maximizing policy to be within the trust region around πk defined as {π : DπkKL(π, πk) ≤ δ}. The TRPO approach provides a provable guarantee on the performance improvement, as stated in Proposition 1. We omit the details as this is a standard approach in the literature.
Step 2: Policy Guidance: While the TRPO approach provides an efficient online learning strategy in the dense reward setting, it fails when the reward structure is sparse. In particular, it fails to achieve any significant performance improvement for a very large number of episodes in the initial phase of learning due to the lack of useful reward feedback that is necessary for efficient exploration. We propose to overcome this challenge by providing policy guidance using offline demonstration data after each step of the TRPO update. The policy guidance step is given as
πk+1 = arg min π
DπKL(π, πb) s.t. D max KL (π, πk+1/2) ≤ δk. (2)
Intuitively, the policy guidance step aids learning by selecting the policy πk+1 in the direction of the behavior policy πb. This is achieved by finding a policy that minimizes the KL divergence w.r.t. πb, but at the same time lies inside the trust region around πk+1/2 defined as {π : DmaxKL (π, πk+1/2) ≤ δk}. This trust region-based policy guidance is the key idea that distinguishes LOGO from other approaches. In particular, this approach gives LOGO two unique advantages over other state-of-the-art algorithms.
First, unlike imitation learning that tries to mimic the behavior policy by directly minimizing the distance (typically KL/JS divergence) between it and the current policy, LOGO only uses the behavior policy to guide initial exploration. This is achieved by starting the guidance step with a large value of the trust region δk, and gradually decaying it according to an adaptive update rule (specified in Appendix F) as learning progresses. This novel approach of trust region based policy improvement and policy guidance enables LOGO to both learn faster in the initial phase by exploiting demonstration data, and to converge to a better policy than the sub-optimal behavior policy.
Second, from an implementation perspective, the trust region based approach allows to approximate the objective by a surrogate function that is amenable to sample based learning (see Proposition 2). This allows us to implement LOGO in the manner of two TRPO-like steps, enabling us to leverage the TRPO code base. Details are given Section 4.

heading: 3.2 PERFORMANCE GUARANTEE

text:
We derive a lower bound on the performance improvement, JR(πk+1)− JR(πk) for LOGO in each learning episode. We analyze the policy improvement step and policy guidance step separately. We begin with the performance improvement due to the TRPO update (policy improvement step). The following result and its analysis are standard in the literature and we omit the details and the proof.
Proposition 1 (Proposition 1, (Achiam et al., 2017)). Let πk and πk+1/2 are related by (1). Then,
JR(πk+1/2)− JR(πk) ≥ − √ 2δγ R,k/(1− γ)2, (3)
where R,k = maxs,a |AπkR (s, a)|.
We now give the following result which can be used to obtain a lower bound on the performance improvement due to the policy guidance step. Lemma 1. Let πk+1/2 be a policy that satisfies Assumption 1. Then, for any policy π,
JR(π)− JR(πk+1/2) ≥ (1− γ)−1β − (1− γ)−1 R,k+1/2 √ 2DπKL(π, πb), (4)
Where R,k+1/2 = maxs,a |A πk+1/2 R (s, a)|.
Remark 1. Lemma 1 also gives an intuitive explanation for our proposed policy guidance step. The policies used in the initial phase of the learning can be far from the optimal. So, it is reasonable to assume that πk (and hence πk+1/2) satisfies Assumption 1 in the initial phase of learning. Then, minimizing DπKL(π, πb) can get a non-negative lower bound in (4), which will imply that the performance of πk+1 is better than πk+1/2. This is indeed the idea behind the policy guidance step.
Combining the results of Proposition 1 and Lemma 1, and with some more analysis, we get the following performance improvement guarantee for the LOGO algorithm. Theorem 1. Let πk and πk+1/2 are related by (1) and let πk+1/2 and πk+1 are related by 2. Let R,k and R,k+1/2 be as defined in Proposition 1 and Lemma 1, respectively. LetRmax = maxs,a |R(s, a)|. (i) If πk+1/2 satisfies Assumption 1, then
JR(πk+1)− JR(πk) ≥ − √
2δγ R,k (1− γ)2 + β (1− γ) − R,k+1/2 (1− γ)
√ 2DπKL(πk+1, πb). (5)
(ii) If πk+1/2 does not satisfy Assumption 1, then
JR(πk+1)− JR(πk) ≥ −( √ 2δγ R,k + 3Rmaxδk)/(1− γ)2. (6)
Remark 2. In the initial phase of learning when the baseline policy is better than the current policy, the policy guidance step can add a non-negative term to the standard lower bound obtained by the TRPO approach, as shown in (5). This indicates faster learning in the initial phase as compared to the naive TRPO approach. The policy improvement guarantee during the later phase of learning is given by (6), which shows that LOGO achieves similar performance guarantee as TRPO when δk = O( √ δ). We ensure this by decreasing the value of δk as the learning progresses. Thus, Theorem 1 clearly shows the key advantage of the LOGO algorithm achieved by the novel combination of the policy improvement step and the policy guidance step.

heading: 4 PRACTICAL ALGORITHM

text:
We first develop an approximation to the policy guidance step (2) that is amenable to sample-based learning and can scale to policies paramaterized by neural networks. This step involves minimizing DπKL(π, πb) under a trust region constraint. However, this is not easy to solve directly by a samplebased learning approach, because estimating it requires samples generated according to any possible π, which is clearly infeasible. To overcome this issue, inspired by the surrogate function idea used in the TRPO algoirthm (Schulman et al., 2015), we derive a surrogate function for DπKL(π, πb) that can be estimated using only the samples from the policy πk+1/2.
For deriving a surrogate function for DπKL(π, πb) that is amenable to sample-based learning, we first define the policy dependent reward function Cπ as Cπ(s, a) = log(π(s, a)/πb(s, a)). Using Cπ , we can also define the quantities JCπ (π̃), V π̃ Cπ , Qπ̃Cπ and A π̃ Cπ
for any policy π̃, exactly as defined in Section 2.1 by replacing R by Cπ . Using these notations, we now present an interesting result which we call the performance difference lemma for policy dependent reward function. Lemma 2. For any policies π and π̃,
JCπ (π)− JCπ̃ (π̃) = (1− γ)−1 Es∼dπ,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] + (1− γ) −1DπKL(π, π̃). (7)
Note that the above result has an additional term, (1− γ)−1DπKL(π, π̃), compared to the standard performance difference lemma (Kakade & Langford, 2002). In addition to being useful for analyzing our algorithm, we believe that the above result may also be of independent interest.
We now make an interesting observation that JCπ (π) = (1− γ)−1DπKL(π, πb) (proof is given in the Appendix), which can be used with Lemma 2 to derive the surrogate function given below.
Proposition 2. Let πk+1/2 be as given in (1). Then, for any policy π that lies in the trust region around πk+1/2 defined as {π : DmaxKL (π, πk+1/2) ≤ δk}, we have
DπKL(π, πb) ≤ αk + Es∼dπk+1/2 ,a∼π(s,·)[A πk+1/2 Cπ
k+1/2
(s, a)] + γ(1− γ)−1 π,k √ 2δk + δk, (8)
where αk = D πk+1/2 KL (πk+1/2, πb), π,k = maxs,a |A πk+1/2 Cπ
k+1/2
(s, a)|.
Now, instead of minimizing DπKL(π, πb), we will minimize the upper bound as a surrogate objective function. Note that only one term in this surrogate objective function depends on π, and that term can be estimated using only samples from πk+1/2. Thus, we will approximate the policy guidance step as
πk+1 = arg min π E s∼d π k+1/2 ,a∼π(s,·)[A
πk+1/2 Cπ
k+1/2
(s, a)] s.t. DmaxKL (π, πk+1/2) ≤ δk. (9)
Since DmaxKL is difficult to implement in practice, we will further approximate it by average KL divergence D
πk+1/2 KL (π, πk+1/2). We note that this is a standard approach used in the TRPO algorithm.
We can now put steps (1) and (9) together to obtain the full algorithm. However, as the policies are represented by large neural networks, solving them exactly is challenging. Hence, for sufficiently small δ, δk, we can further approximate the objective functions and constraints by a Taylor series expansion to obtain readily implementable update equations. This is a standard approach (Schulman et al., 2015; Achiam et al., 2017), and we only present the final result below.
Consider the class of policies {πθ : θ ∈ Θ} where θ is the parameter of the policy. Let θk be the parameter corresponding to policy πk. Then, the Taylor series expansion-based approximate solution of (1) and (9) yields the final form of LOGO as follows:
θk+1/2 = θk +
√ 2δ
gTk F −1 k gk
F−1k gk, θk+1 = θk+1/2 − √ 2δk
hTk L −1 k hk
L−1k hk, (10)
where gk = ∇θ Es∼dπk ,a∼πθ(s,·) [A πk R (s, a)], Fk = ∇2θ D πk KL(πθ, πk), and hk = ∇θ Es∼dπk+1/2 ,a∼πθ(s,·)[A πk+1/2 Cπ
k+1/2
(s, a)], Lk = ∇2θ D πk+1/2 KL (πθ, πk+1/2).
While it is straightforward to compute AπCπ for any policy when the form of the baseline policy πb is known, it is more challenging when only the demonstration data D generated according to πb is available. We overcome this challenge by training a discriminator using the demonstration dataD and the data Dk+1/2 generated by the policy πk+1/2 (Goodfellow et al., 2014; Ho & Ermon, 2016; Kang et al., 2018) that will approximate the policy dependent reward function Cπk+1/2 . Further details on training this discriminator and a concise form of the algorithm are given in Appendix D.

heading: 4.1 EXTENSION TO INCOMPLETE OBSERVATION SETTING

text:
We now discuss how to extend LOGO to the setting where the behavior policy data contains only incomplete state observations. For instance, consider the problem of learning a policy for a mobile robot to reach a target point without colliding with any obstacles. Collision avoidance requires sensing the presence of obstacles, which is typically achieved by camera/Lidar sensors. Learning an RL policy for this problem requires a high fidelity simulator that models various sensors and their interaction with the dynamics. Such high fidelity simulators are, however, typically slow and difficult to parallelize, and training an RL policy using such a simulator in a sparse reward environment can be very time consuming and computationally expensive. Often, it is much easier to train an RL policy for the trajectory tracking problem using only a simple kinematics simulator model which only has a lower dimensional state space compared to the original problem. In particular, such a kinematics simulator will only have the position and velocity of the robot as the state instead of the true state with
high dimensional camera/Lidar image. Can we use the demonstration data or behavior policy from this low dimensional simulator to accelerate the RL training in a high dimensional/fidelity simulator in sparse reward environments? We answer this question affirmatively using a simple idea to extend LOGO into such incomplete observation settings.
Since the behavior policy appears in LOGO only through a policy dependent reward function Cπ(s, a) = log(π(s, a)/πb(s, a)), we propose to replace this with a form that can handle the incomplete observation setting. For any state s ∈ S, let s̃ = o(s) be its projection to a lower dimensional state space S̃. As explained in the mobile robot example above, a behavior policy π̃b in the incomplete observation setting can be interpreted as mapping from S̃ to the set of probability distributions over A. We can then replace Cπ(s, a) in the LOGO algorithm with C̃π(s, a) defined as C̃π(s, a) = log(π(s, a)/π̃b(o(s), a)). When only the demonstration data with incomplete observation is available instead of the representation of the behavior policy, we propose to train a discriminator to estimate C̃π. Let D̃ = {τ̃ i}ni=1 be the demonstration with incomplete observation, where τ̃ i = (s̃i1, a i 1, . . . , s̃ i T , a i T ). Then we train a discriminator using D̃ and Dπ to estimate C̃π .

heading: 5 EXPERIMENTS

text:
We now evaluate LOGO from two perspectives: (i) Can LOGO learn near-optimally in a sparse reward environment when guided by demonstration data generated by a sub-optimal policy? (ii) Can LOGO retain near-optimal performance when guided by sub-optimal and incomplete demonstration data with sparse rewards? We perform an exhaustive performance analysis of LOGO, first through simulations under four standard (sparsified) environments on the widely used MuJoCo platform (Todorov et al., 2012). Next, we conduct simulations on the Gazebo simulator (Koenig & Howard, 2004) using LOGO for way-point tracking by a robot in environments with and without obstacles, with the only reward being attainment of way points. Finally, we transfer the trained models to a real-world TurtleBot robot (Amsters & Slaets, 2019) to demonstrate LOGO in a realistic setting.
0.04 0.02 0.00 0.02 0.04 0.05 0.00 0.05 LOGO PofD TRPO BC-TRPO GAIL DAPG Behavior Expert
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Samples 1e7
0
10 00
20 00
30 00
Av er
ag e
ep iso
di c
re wa
rd
Hopper-v2
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Samples 1e7
-20 00
-10 00
01 00
02 00
03 00
04 00
05 00
0
HalfCheetah-v2
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Samples 1e7
0
10 00
20 00
30 00
40 00
Walker-v2
0.0 0.5 1.0 1.5 2.0 Samples 1e5
0
20 00
40 00
60 00
80 00
10 00
0 InvertedDoublePendulum-v2
(a) Evaluation on MuJoCo with full offline observation.
In what follows, we present a summary of our experiments1 and results. We remark that LOGO is relatively easy to implement and train, since its two TRPO-like steps imply that we can utilize much of a TRPO code base.
1code base and a video of the TurtleBot experiments: https://github.com/DesikRengarajan/LOGO

heading: 5.1 MUJOCO SIMULATIONS

text:
In our first set of experiments, we consider four standard environments using the MuJoCo platform. We introduce sparsity by reducing the events at which reward feedback is provided. Specifically, for Hopper, HalfCheetah and Walker2d, we provide a reward of +1 at each time only after the agent moves forward over 2, 20, and 2 units from its initial position, respectively. For InvertedDoublePendulum, we introduce sparsity by providing a reward only at the end of the episode.
Besides LOGO, our candidate algorithms are as follows: (i) Expert: We train TRPO in the dense reward environment to provide the optimal baseline, (ii) Behavior: We use a partially trained expert that is still at a sub-optimal stage of learning to provide behavior data, (iv) GAIL: We use Generative Adversarial Imitation Learning (Ho & Ermon, 2016), which attempts to imitate the Behavior policy (iii) TRPO: We directly use TRPO in the sparse reward setting without any guidance from behavior data (iv) POfD: We use Policy Optimization from Demonstration (Kang et al., 2018) as a heuristic approach to exploiting behavior data. (v) BC-TRPO: We warm start TRPO by performing behavior cloning (BC) on the sub-optimal behavior data. (vi) DAPG: We use Demo Augmented Policy Gradient (Rajeswaran et al., 2018) which warm starts Natural Policy Gradient (NPG) algorithm using BC, and fine tunes it online using behavior data in a heuristic manner. Note that for all algorithms, we evaluate the final performance in the corresponding dense reward environment provided by OpenAI Gym, which provides a standardized way of comparing their relative merits.
Setting 1: Sparse Rewards. We compare the performance of our candidate algorithms in the sparse reward setting in Figure 1a, which illustrates their rewards during training. As expected, TRPO fails to make much meaningful progress during training, while GAIL can at best attain the same performance as the sub-optimal behavior policy. While BC-TRPO benefits from warm starting, it fails to learn later due to the absence of online guidance as in the case of LOGO. POfD and LOGO both use the behavior data to boot strap learning. POfD suffers from the fact that it is influenced throughout the learning process by the behavior data, which prevents it from learning the best policy. However, LOGO’s nuanced exploration using the demonstration data only as guidance enables it to quickly attain optimality. LOGO outperforms DAPG in all but one environment.
Setting 2: Sparse Rewards and Incomplete State. We next consider sparse rewards along with reducing the revealed state dimensions in the behavior data of Setting 1. These state dimensions are selected by eliminating state dimensions revealed until the expert TRPO with dense rewards starts seeing reduced performance. This ensures that we are pruning valuable information. Since GAIL, POfD and LOGO all utilize the behavior data, we use the approach of projecting the appropriate reward functions that depend on the behavior data into a lower dimensional state space described in Section 4.1. We emphasize that BC-TRPO and DAPG cannot be extended to this setting as they require full state information for warm starting (in BC-TRPO and DAPG) and online fine tuning (in DAPG). We see from Figure 1b that LOGO is still capable of attaining good performance, although training duration is increased. We will further explore the value of being able to utilize behavior data with such incomplete state information in robot experiments in the next subsection.

heading: 5.2 TURTLEBOT EXPERIMENTS

text:
We now evaluate the performance of LOGO in a real-world using TurtleBot, a two wheeled differential drive robot (Amsters & Slaets, 2019). We train policies for two tasks, (i) Waypoint tracking and (ii) Obstacle avoidance, on Gazebo, a high fidelity 3D robotics simulator.
Task 1: Waypoint Tracking: The goal is to train a policy that takes the robot to an arbitrary waypoint within 1 meter of its current position in an episode of 20 seconds. The episode concludes when the robot either reaches the waypont or the episode timer expires. The state space of the agent are its relative x, y, coordinates and orientation φ to the waypoint. The actions are its linear and angular velocities. The agent receives a sparse reward of +1 if it reaches the waypoint, and 0 otherwise. We created a sub-optimal Behavior policy by training TRPO with dense rewards on our own low fidelity kinematic model Python-based simulator with the same state and action space. While it shows reasonable waypoint tracking, the trajectories that it generates in Gazebo are inefficient, and its real-world waypoint tracking is poor (Figures 2 (d)-(f)). As expected, TRPO shows poor performance in this sparse reward setting and often does not attain the desired waypoint before the episode concludes, giving the impression of aimless circling seen in Figure 2(f). LOGO is able to effectively utilize the Behavior policy and shows excellent waypoint tracking seen in Figures 2 (d)-(f).
Task 2: Obstacle Avoidance: The goal and rewards are the same as in Task 1, with a penalty of −1 for collision with the obstacle, shown in Figure 2 (a)-(c). The complete state space is now augmented by a 2D Lidar scan in addition to coordinates and orientation described in Task 1. However, the Behavior policy is still generated via the low fidelity kinematic simulator without the obstacle, i.e., it is created on a lower dimensional state space. As seen in Figures 2 (g)-(i), this renders the Behavior policy impractical for Task 2, since it almost always hits the obstacle in both Gazebo and the realworld. However, it does possess information on how to track a waypoint, and when combined with the full state information, this nugget is utilized very effectively by LOGO to learn a viable policy as seen in Figures 2 (g)-(i). Further, TRPO in this sparse reward setting does poorly and often collides with the obstacle in real-world experiments as seen in Figure 2 (i).

heading: 6 CONCLUSION

text: In this paper, we studied the problem of designing RL algorithms for problems in which only sparse reward feedback is provided, but offline data collected from a sub-optimal behavior policy, possibly with incomplete state information is also available. Our key insight was that by dividing the training problem into two steps of (i) policy improvement and (ii) policy guidance using the offline data, each using the concept of trust region based policy optimization, we can both obtain principled policy improvement and desired alignment with the behavior policy. We designed an algorithm entitled LOGO around this insight and showed how it can be instantiated in two TRPO-like steps. We both proved analytically that LOGO can exploit the advantage of the behavior policy, as well as validated its performance through both extensive simulations and illustrative real-world experiments.

heading: 7 ACKNOWLEDGEMENT

text: This work was supported in part by the National Science Foundation (NSF) grants NSF-CRII-CPS1850206, NSF-CAREER-EPCN-2045783, NSF-CPS-2038963 and NSF-CNS 1955696, and U.S. Army Research Office (ARO) grant W911NF-19-1-0367. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring agencies.

heading: A USEFUL TECHNICAL RESULTS

text:
We will use the well known Performance Difference Lemma in our analysis (Kakade & Langford, 2002)
Lemma 3 (Performance Difference Lemma). For any two policies π and π̃,
JR(π)− JR(π̃) = (1− γ)−1 Es∼dπ,a∼π(s,·)[Aπ̃R(s, a)]
The following result gives a bound on the difference between discounted state visitation distributions of two policies in terms of their average total variation difference.
Lemma 4 (Lemma 3, (Achiam et al., 2017)). For any two policies π and π′,∥∥∥dπ − dπ′∥∥∥ 1 ≤ 2γ(1− γ)−1 DπTV(π, π′)
We will use the following well known result. Lemma 5. For any two policies π and π′, DπTV(π, π′) ≤ √ DπKL(π, π ′)/2. Proof. From Pinsker’s inequality, DTV(p, q) ≤ √ DKL(p, q)/2 for any two distributions p, q. Combining this with Jensen’s inequality, we get DπTV(π, π ′) ≤ √ DπKL(π, π ′)/2.
We will use the following result which is widely used in literature. We provide the proof for completeness.
Lemma 6. Given any policy π and a function fπ : S ×A → R,
Eτ∼π[ ∞∑ t=0 γtfπ(st, at)] = (1− γ)−1Es∼dπ,a∼π(s,·)[fπ(s, a)]

heading: Proof.

text:
Eτ∼π[ ∞∑ t=0 γtfπ(st, at)] = ∑ s,a ∞∑ t=0 γtP(st = s, at = a|π)fπ(s, a)
= ∑ s,a ∞∑ t=0 γtP(st = s|π)π(s, a)fπ(s, a) = ∑ s,a π(s, a)fπ(s, a) ∞∑ t=0 γtP(st = s|π)
= (1−γ)−1 ∑ s,a dπ(s)π(s, a)fπ(s, a) = (1− γ)−1Es∼dπ,a∼π(s,·)[fπ(s, a)].

heading: B PROOF OF THE RESULTS IN SECTION 3

heading: B.1 PROOF LEMMA 1

text:
Proof. Starting from the performance difference lemma,
JR(π)− JR(πk+1/2) = (1− γ)−1 Es∼dπ,a∼π(s,·)[A πk+1/2 R (s, a)] = (1− γ)−1 ∑ s dπ(s) ∑ a πb(s, a)A πk+1/2 R (s, a)
+ (1− γ)−1 ∑ s dπ(s) ∑ a (π(s, a)− πb(s, a))A πk+1/2 R (s, a)
(a) ≥ (1− γ)−1β + (1− γ)−1 ∑ s dπ(s) ∑ a (π(s, a)− πb(s, a))A πk+1/2 R (s, a)
(b) ≥ (1− γ)−1β − (1− γ)−1 R,k+1/2 2 DπTV(π, πb) (c) ≥ (1− γ)−1β − (1− γ)−1 R,k+1/2 √ 2DπKL(π, πb),
where (a) follows from the premise that πk+1/2 satisfies Assumption 1, (b) is obtained by denoting R,k+1/2 = maxs,a |A πk+1/2 R (s, a)|, and (c) follows from Lemma 5.

heading: B.2 PROOF THEOREM 1

text:
We will first prove the following result. Lemma 7. For any two policies π and π̃,
|JR(π)− JR(π̃)| ≤ 3Rmax
(1− γ)2 DmaxTV (π, π̃),
where Rmax = maxs,a |R(s, a)|.
Proof. We have, JR(π)− JR(π̃) = (1− γ)−1( ∑ s,a dπ(s)π(s, a)R(s, a)− ∑ s,a dπ̃(s)π̃(s, a)R(s, a))
= (1− γ)−1( ∑ s,a dπ(s)π(s, a)R(s, a)− ∑ s,a dπ̃(s)π(s, a)R(s, a))
+ (1− γ)−1( ∑ s,a dπ̃(s)π(s, a)R(s, a)− ∑ s,a dπ̃(s)π̃(s, a)R(s, a))
≤ (1− γ)−1Rmax ∥∥dπ − dπ̃∥∥ 1 + (1− γ)−1RmaxDmaxTV (π, π̃)
(a) ≤ (1− γ)−1Rmax2γ(1− γ)−1DmaxTV (π, π̃) + (1− γ)−1RmaxDmaxTV (π, π̃)
≤ 3Rmax (1− γ)2 DmaxTV (π, π̃),
where (a) follows from Lemma 4. The lower bound can also be obtained in a similar way. Combining both, we obtain the desired result.
Proof of Theorem 1. (i) If πk+1/2 satisfies Assumption 1, then (5) follows immediately by combining the results of Proposition 1 and Lemma 1. (ii) If πk+1/2 does not satisfy Assumption 1, we can directly bound JR(πk+1)− JR(πk+1/2) without invoking Lemma 1 (and hence the implicit assumption that moving towards πb improves the policy).
Use Lemma 7, we get
JR(πk+1)− JR(πk+1/2) ≥ − 3Rmax
(1− γ)2 DmaxTV (πk+1, πk+1/2) ≥ − 3Rmax (1− γ)2 δk, (11)
where the last inequality follows from the fact DmaxTV (πk+1, πk+1/2) ≤ δk according to the policy guidance step (2).
Now, combining the result of Proposition 1 and the above inequality, we obtain (6).

heading: C PROOF OF THE RESULTS IN SECTION 4

heading: C.1 PROOF OF LEMMA 2

text:
Proof. We follow the proof technique of the performance difference lemma (PDL) with some modifications to get the result. For any given initial state s,
V πCπ (s)− V π̃ Cπ̃ (s) = Eτ∼π|s0=s[ ∞∑ t=0 γtCπ(st, at)]− V π̃Cπ̃ (s)
= Eτ∼π|s0=s[ ∞∑ t=0 γt(Cπ(st, at) + V π̃ Cπ̃ (st)− V π̃ Cπ̃ (st))]− V π̃ Cπ̃ (s)
(a) = Eτ∼π|s0=s[ ∞∑ t=0 γt(Cπ(st, at) + γV π̃ Cπ̃ (st+1)− V π̃ Cπ̃ (st))]
=Eτ∼π|s0=s[ ∞∑ t=0 γt(Cπ̃(st, at) + γV π̃ Cπ̃ (st+1)− V π̃ Cπ̃ (st))]
+ Eτ∼π|s0=s[ ∞∑ t=0 γt(Cπ(st, at)− Cπ̃(st, at))], (12)
where (a) is obtained by rearranging the summation and telescoping. The first summation in (12) can now be handled as in the proof of the PDL, and we omit the details. So, we get
Eτ∼π[ ∞∑ t=0 γt(Cπ̃(st, at) + γV π̃ Cπ̃ (st+1)− V π̃ Cπ̃ (st))] = (1− γ) −1Es∼dπ,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] (13)
The second summation can be written as
Eτ∼π[ ∞∑ t=0 γt(Cπ(st, at)− Cπ̃(st, at))] = Eτ∼π[ ∞∑ t=0 γt(log π(st, at) πb(st, at) − log π̃(st, at) πb(st, at) )]
= Eτ∼π[ ∞∑ t=0 γt log π(st, at) π̃(st, at) ] (b) = (1− γ)−1 ∑ s dπ(s) ∑ a π(s, a) log π(st, at) π̃(st, at) = (1− γ)−1DπKL(π, π̃), (14)
where (b) is obtained by using Lemma 6.
Now, by taking expectation on the both sides of (12) w.r.t. the initial state distribution µ, and then substituting (13) and (14) there, we get the desired result.

heading: C.2 PROOF OF PROPOSITION 2

text:
Lemma 2 provides an approach to evaluate the infinite horizon return JCπ (π) of policy π using the infinite horizon return JCπ̃ (π̃) and advantage A π̃ Cπ̃
of policy π̃. Unfortunately, the RHS of (7) contains two terms that requires taking expectation w.r.t. dπ, which means that empirical estimation requires samples according to π (that are unavailable). This is a well known issue in the policy gradient literature and it is addressed by considering an approximation by replacing the expectation w.r.t. dπ by dπ̃ (Kakade & Langford, 2002; Schulman et al., 2015). We follow the same approach with minor modifications in the context of our problem.
We will first prove the following lemmas.
Lemma 8. For any two policies π and π̃,
JCπ (π)− JCπ̃ (π̃) ≤ 1
(1− γ) Es∼dπ̃,a∼π(s,·)[Aπ̃Cπ̃ (s, a)]
+
√ 2γ π̃ (1− γ)2 √ Dπ̃KL(π, π̃) +
1
(1− γ) DmaxKL (π, π̃),
where π̃ = maxs,a |Aπ̃Cπ̃ (s, a)|.
Proof. We will separately bound the two terms on the RHS of (7) in Lemma 2. Firstly, Es∼dπ,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] = ∑ s dπ(s) ∑ a π(s, a)Aπ̃Cπ̃ (s, a)
= ∑ s dπ̃(s) ∑ a π(s, a)Aπ̃Cπ̃ (s, a) + ∑ s (dπ(s)− dπ̃(s)) ∑ a π(s, a)Aπ̃Cπ̃ (s, a)
≤ Es∼dπ̃,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] + ∥∥dπ − dπ̃∥∥ 1 π̃
(a) ≤ Es∼dπ̃,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] + 2γ(1− γ) −1 π̃ D π̃ TV(π, π̃)
(b) ≤ Es∼dπ̃,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] + √ 2γ(1− γ)−1 π̃ √ Dπ̃KL(π, π̃), (15)
where (a) follows from Lemma 4 and (b) follows from Lemma 5.
Secondly, DπKL(π, π̃) ≤ DmaxKL (π, π̃). Using this fact and the inequality (15) in (7), we get the desired result.
We now make an interesting observation that JCπ (π) is a scaled version of the average KL divergence between π and πb. We formally state this result below.
Lemma 9. For any arbitrary policy π,
JCπ (π) = Eτ∼π[ ∞∑ t=0 γtCπ(st, at)] = (1− γ)−1DπKL(π, πb) (16)
Proof. We have,
JCπ (π) = Eτ∼π[ ∞∑ t=0 γtCπ(st, at)] (a) = (1− γ)−1 ∑ s,a dπ(s)π(s, a)Cπ(s, a)
= (1− γ)−1 ∑ s dπ(s) ∑ a π(s, a) log π(s, a) πb(s, a) = (1− γ)−1DπKL(π, πb),
where (a) follows from Lemma 6.
Proof of Proposition 2. Using the result of Lemma 9 in the inequality given by Lemma 8, for any policy π and π̃, we get DπKL(π, πb) ≤ Dπ̃KL(π̃, πb) + Es∼dπ̃,a∼π(s,·)[Aπ̃Cπ̃ (s, a)] + √ 2γ π̃ (1− γ) √ Dπ̃KL(π, π̃) +D max KL (π, π̃).
Now, replacing π̃ by πk+1/2, we get
DπKL(π, πb) ≤ αk + Es∼dπk+1/2 ,a∼π(s,·)[A πk+1/2 Cπ
k+1/2
(s, a)]
+
√ 2γ π,k
(1− γ)
√ D πk+1/2 KL (π, πk+1/2) +D max KL (π, πk+1/2), (17)
where αk = D πk+1/2 KL (πk+1/2, πb), π,k = maxs,a |A πk+1/2 Cπ
k+1/2
(s, a)|.
Now, for any π that lies in the trust region {π : DmaxKL (π, πk+1/2) ≤ δk}, we have D πk+1/2 KL (π, πk+1/2) ≤ DmaxKL (π, πk+1/2) ≤ δk. Using this in (17) gives the final result.

heading: D DETAILS OF THE PRACTICAL ALGORITHM

text:
As explained in Section 4, when πb is known, obtaining Cπ(s, a) = log(π(s, a)/πb(s, a)) and estimating AπCπ is straightforward. Hence, LOGO can perform the policy improvement and policy guidance step according to the update equations given in (10) by directly using πb. When the form of πb is unknown and we only have access to the demonstration data D generated according to πb, we have to estimate Cπ(s, a) using D and Dπ , where Dπ is the trajectory data generated according to π. Instead of estimating log(π(s, a)/πb(s, a)) directly, we make use of the one-to-one correspondence between a policy and its discounted state-action visitation distribution defined as ρπ(s, a) = dπ(s)π(s, a) (Syed et al., 2008). More precisely, we estimate log(ρπ(s, a)/ρπb(s, a)) using D and Dπ instead of estimating log(π(s, a)/πb(s, a)) directly. We can then use the powerful framework of
generative adversarial networks (GAN) (Goodfellow et al., 2014) to estimate this quantity. This can be achieved by training a discriminator function B : S ×A → [0, 1] as
max B
E(s,a)∼ρπb [logB(s, a)] + E(s,a)∼ρπ [log(1−B(s, a))]. (18)
We note that the GAN-based approach to estimate a distance metric between ρπ and ρπb is popular in the imitation learning literature (Ho & Ermon, 2016; Kang et al., 2018).
The optimal discriminator for the above problem is given byB∗(s, a) = ρ πb (s,a)
ρπb (s,a)+ρπ(s,a) (Goodfellow et al., 2014, Proposition 1). Hence, given the discriminator function B obtained after training to solve (18), we use Cπ(s, a) = − logB(s, a) in the LOGO algorithm, which provides an approximation to the quantity of interest. As shown in Section 5, this approximation yields excellent results in practice.
We summarize the LOGO algorithm below.
Algorithm 1 LOGO Algorithm 1: Initialization: Initial policy π0, Demonstration data D or behavior policy πb 2: for k = 0, 1, 2, .. do 3: Collect (s, a, r, s′) ∼ πk and store in Dπk 4: if πb is known then 5: Cπk(s, a) = log(πk(s, a)/πb(s, a)) 6: else 7: Train a discriminator B(s, a) according to (18) using D and Dπk 8: Cπk(s, a) = − logB(s, a) 9: end if 10: Estimate gk and Fk using Dπk
11: Perform policy improvement step: θk+1/2 = θk +
√ 2δ
gTk F −1 k gk
F−1k gk
12: Decay δk (according to the adaptive rule described in Appendix F) 13: Estimate hk and Lk using Cπk and Dπk+1/2
14: Perform policy guidance step: θk+1 = θk+ 12 − √ 2δk
hTk L −1 k hk
L−1k hk
15: end for

heading: E DETAILS OF TURTLEBOT EXPERIMENTS

text:
We evaluate the performance of LOGO in a real-world setting using TurtleBot, a two wheeled differential drive robot (Amsters & Slaets, 2019). We consider two different simulators for training our policy. The first one is a simple kinematics-based low fidelity simulator. The second one is a sophisticated physics-based Gazebo simulator (Koenig & Howard, 2004) with an accurate model of the TurtleBot.
We use the low fidelity simulator to get a sub-optimal Behavior policy by training TRPO with dense rewards. The details are given in Section E.1. Gazebo simulator is used for implementing the LOGO algorithm in a sparse reward setting with complete and incomplete observations. The details are given in Section E.2.

heading: E.1 LOW FIDELITY KINEMATICS SIMULATOR

text:
We design the low fidelity simulator using the OpenAI gym framework where the dynamics are governed by the following equations
xt+1 = xt + vt cos(θt)∆, yt+1 = yt + vt sin(θt)∆, θt+1 = θt + ωt,∆,
where xt, yt are the coordinates of the bot, vt is the linear velocity, ωt is the angular velocity, and θt is its yaw calculated from the quaternion angles, all calculated at time t. ∆ is the time discretization factor. We define the state st to be the normalized relative position w.r.t. the waypoint,
i.e., st = ((xt − xw)/G, (yt − yw)/G, θt − θwt ), where xw, yw are the target coordinates of the waypoint, and θwt is the target heading to the waypoint at time t. we define the action at to be the linear and angular velocities, i.e., at = [vt, ωt]. We discretize the action space into 15 actions.
We handcraft a dense reward function as follows,
rt =  +10 if |xt − xw| ≤ 0.05 and |yt − yw| ≤ 0.05, −1 if |xt| ≥ G or |yt| ≥ G, −0.166dt − 0.3184|θwt − θt| otherwise,
where dt is a combination of cross track and along track error defined as follows, dt = ( (xt − xw)2 + (yt − yw)2 ) sin2(θwt − θt) + |xt − xw|+ |yt − yw|.

heading: E.2 HIGH FIDELITY GAZEBO SIMULATOR

text:
Gazebo is a physics-based simulator with rendering capabilities and can be used for realistic simulation of the environment. It is configurable and can be used to model robots with multiple joint constraints, actuator physics, gravity and frictional forces and a wide range of sensors in indoor as well as outdoor settings. Gazebo facilitates close-to-real-world data collection from the robots and sensor models. Since it runs in real-time, it takes millions of simulation frames for an RL agent to learn simple tasks. Gazebo can also speed up simulation by increasing step size. This can however lead to loss of precision. For this project, we ran the training scheme on Gazebo in almost real time with a simulation step-size of ∆T = 0.001.
The Gazebo simulation setup consists of the differential drive robot (TurtleBot3 Burger) model spawned in either an empty space or a custom space with obstacles at a predefined location. A default coordinate grid is setup with respect to the base-link of the robot model. A ROS (Stanford Artificial Intelligence Laboratory et al.) framework is instantiated using a custom-built OpenAI Gym environment which acts as an interface between the proposed RL algorithm and the simulation model. ROS topics are used to capture the state update information of the robot asynchronously in a callback driven mechanism. For the purpose of our experiments we use the following ROS topics,
1. /odom (for xt, yt, θt): Odometry information based on the wheel encoder 2. /cmd vel (for vt, ωt): Linear and angular velocity commands 3. /scan (for obstacle avoidance): Scan values from the Lidar
The /odom and /scan topics are used to determine st while /cmd vel is used to output the required action at. The state space and action space for waypoint tracking task is similar to state and action space in section E.1. For obstacle avoidance tasks, we also include the distance values obtained form the Lidar as a part of the state.

heading: E.3 ROBOT PLATFORM: TURTLEBOT3

text: We use TurtleBot 3 (Amsters & Slaets, 2019), an open source differential drive robot equipped with a RP Lidar as our robotic platform for real-world experiments. We use ROS as the middleware to setup the communication framework. The bot transmits its state (position and Lidar information) information over a network to a intel NUC, which transmits back the corresponding action according to the policy being executed.

heading: E.4 TRAINING: WAYPOINT TRACKING

text:
In navigation problems, we have a global planner that uses a high level map of the bot’s surroundings for planning a trajectory using waypoints a unit-meter distance from each other, while the goal of the bot is to achieve these waypoints. To obtain a waypoint tracking scheme, we first train a behavior policy on the low fidelity simulator using TRPO to reach a waypoint approximately one unit distance (1m in the real world) away from its initial position. In each training episode, the bot is reset to the origin and a waypoint is randomly chosen as xw ∼ uniform([−1, 1]), yw = 1. The episode is terminated if the robot reaches the waypoint, or if it crosses the training boundary or exceeds the maximum episode length. The behavior policy obtained after training in the low fidelity simulator
is then used in the LOGO algorithm training on Gazebo. LOGO is trained in Gazebo with sparse sparse rewards, where a reward of +1 is provided if the bot reaches the waypoint, and 0 otherwise. We evaluate the trained policy in Gazebo and the real world, by providing it a series of waypoints to track in order to reach its final goal.

heading: E.5 TRAINING: OBSTACLE AVOIDANCE

text:
We train our bot for obstacle avoidance in Gazebo using the behavior policy described in the section above. Our goal is to use the skills of waypoint navigation from the behavior policy to guide and learn the skills of obstacle avoidance. The state space includes the Lidar scan values in addition to the state space described in section E.1. The /scan provides 360 values, each of these indicate the distance to the nearest object in a 1◦ sector. For the purpose of our experiments, we use the minimum distance in each 60◦ sector. This reduces the Lidar data to 6 values. We train our algorithm on Gazebo with a fixed obstacle for random waypoints. In each training episode, the bot is reset to the origin and a waypoint is generated similar to the previous section. The episode is terminated if same conditions in the previous section are satisfied or if a collision with the obstacle occurs. We demonstrate the performance of our algorithms both in Gazebo as well as the real-world.
F IMPLEMENTATION DETAILS
We implement all the algorithms in this paper using PyTorch (Paszke et al., 2019). For all our experiments, we have a two layered (128× 128) fully connected neural network with tanh activation functions to parameterize our policy and value functions. We use a learning rate of 3 × 10−4, a discount factor γ = 0.99, and TRPO parameter δ = 0.01. We decay the influence of the behavior policy by decaying δk. We start with δ0, and we do not decay δk for the first Kδ iterations. For k > Kδ, we geometrically decay δk as δk ← αδk, whenever the average return in the current iteration is greater than the average return in the past 10 iterations. The rest of the hyperparameters for MuJoCo simulations, Gazebo simulation, and real-world experiments are given in table 1. In table 2 we provide details on the demonstration data collected using the behavior policy.
We implement LOGO based on a publicly available TRPO code base. We implement PofD, TRPO, BC-TRPO, and GAIL by modifying the same code base as used by LOGO. We run DAPG using code base and hyperparameters provided by the authors. For consistency, we run all algorithms using the same batch size.

heading: G RELATED WORK

text:
Offline RL: Recently, there have been many interesting works in the area of offline RL which use offline data to learn a policy. In particular, offline RL algorithms such as BEAR (Kumar et al., 2019), BCQ (Fujimoto et al., 2019), ABM (Siegel et al., 2020), and BRAC (Wu et al., 2019a) focus on learning a policy using only the offline data without any online learning or online fine-tuning. The key idea of these offline RL algorithms is to learn a policy that is ‘close’ to the behavior policy that generated the data via imposing some constraints, to overcome the problem of distribution shift. This approach, however, often results in conservative policies, and hence it is difficult to guarantee significant performance improvement over the behavior policy. Moreover, standard offline RL algorithms are not immediately amenable to online fine-tuning, as observed in Nair et al. (2020). LOGO is different from the offline RL algorithms in the following two key aspects. First, unlike the offline RL algorithms mentioned before, LOGO is an online RL algorithm which uses the offline demonstration data for guiding the online exploration during the initial phase of learning. By virtue of this clever online guidance, LOGO is able to converge to a policy that is significantly superior than the sub-optimal behavior policy that generated the demonstration data. Second, offline RL algorithms typically require large amount of state-action data with the associated rewards. LOGO only requires a small amount demonstration data since it is used only for the guiding the online exploration. Moreover, LOGO requires only the state-action observation and does not need the associated reward data. In many real-world application applications, it may be possible to get state-action demonstration data from human demonstration or using a baseline policy. However, it is difficult to assign reward values to these observation.
Advantage Weighted Actor-Critic (AWAC) algorithm (Nair et al., 2020) propose to accelerate online RL by leveraging offline data. This work is different from our approach in four crucial aspects. (i) AWAC requires offline data with associated rewards whereas LOGO requires only the state-action observations (not the reward data). In many real-world applications, it may be possible to get stateaction demonstration data from human demonstration or using a baseline policy. However, it may be difficult to assign reward values to these observations, especially in the sparse reward setting. (ii) AWAC explicitly mentions that it leverages large amounts of offline data whereas LOGO relies on small amount of sub-optimal demonstration data. (iii) LOGO gives a novel and theoretically sound approach using the double trust region structure that provides provable guarantees on its performance. AWAC algorithm does not give any such provable guarantees. (iv) LOGO can be easily extended to the setting with incomplete state information where as AWAC is not immediately amenable to such extension.
There are also many recent works on addressing multiple aspects of IL and LfD, including combining IL and RL for long horizon tasks (Gupta et al., 2019), learning from imperfect (Gao et al., 2018; Jing et al., 2020; Wu et al., 2019b) and incomplete (Libardi et al., 2021; Gülçehre et al., 2020) demonstrations. Their approaches, performance guarantees and experimental settings are different from that of our problem and proposed solution approach.
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Samples 1e7
0
10 00
20 00
30 00
Av er
ag e
ep iso
di c
re wa
rd
Hopper-v2
= 0.85 = 0.9
= 0.95 = 0.97
= 0.99 = 1
Behavior Expert
(a) Sensitivity of α
0.0 0.5 1.0 1.5 2.0 2.5 3.0 Samples 1e7
0
10 00
20 00
30 00
Av er
ag e
ep iso
di c
re wa
rd
Hopper-v2
K = 1 K = 50
K = 100 K = 500
K = 1000 K = 1500
Behavior Expert
(b) Sensitivity of Kδ
H SENSITIVITY ANALYSIS OF α AND Kδ
We run LOGO for different values of α and Kδ by keeping the other parameters fixed on the Hopperv2 environment. We observe from Figure 3a that the performance of LOGO is not sensitive to perturbations in α. When α = 1, we do not decay the influence of the behavior data, this results in a performance close to the behavior data which is inline with our intuition. We observe from Figure 3b that LOGO is not sensitive to perturbations in Kδ as well. We observe that the performance of LOGO is similar for Kδ = 1, 50, 100. This because we only decay δk whenever the average return in the current iteration is greater than the average return in the past 10 iterations. We further note that Kδ controls the iteration from which we begin decaying the influence of the behavior data, this can be clearly seen for Kδ = 500, 1000 where the performance rises as soon as the decaying begins.