summaries: This paper proposes Augmented Policy Cloning (APC) approach to improve the data efficiency of expert behavior by augmenting expert trajectory data with virtual, perturbed states and the expert actions in these virtual states. This method is simple by applying the existing image-based data augmentation method, but it can increase the data efficiency of policy cloning and transfer high-DoF behaviors. The proposed method outperforms BC and Naive ABC. The paper explores a novel approach to using expert policies in RL context. The main idea is to enrich expert demonstrations by perturbing a state by Gaussian noise and querying expert policy to produce an action from the perturbed state. These additional state-action pairs are used in the policy learning algorithm. The authors show experimentally that the proposed Augmented Policy Cloning (APC) can significantly improve the quality of cloned policy, compared to Behaviour Cloning. This paper proposed an data-augmentation technique -- "augmented policy cloning (APC)" to enable efficient policy learning from parametric experts. It achieve a high level efficiency in transferring knowledge from an expert to a student policy for high Degrees of Freedom environment. The augmentation is introduce not only for the states but also to the actions of underlying policy. The paper proposes a policy cloning method that transfers an expert to a student. In the considered setting, the expert policy is accessible at all times, while there is no access to the environment to perform RL. The proposed method leverages data augmentation to successfully train the student policy. This paper proposes to study the non-interactive imitation learning setting where access to the parametric expert policy is available. The paper proposes to augment the logged expert data’s states with noise, and then determine then sample corresponding actions from the expert’s policy. These perturbed states and corresponding action are then use to train the agent. They show this method outperforms Behavior cloning and an approach where the logged states are perturbed but the corresponding actions are kept fixed on imitation learning tasks and RL tasks where IL is used for warmstarting. 
abstractText: We present a simple, yet powerful data-augmentation technique to enable data1 efficient learning from parametric experts. Whereas behavioral cloning refers to 2 learning from samples of an expert, we focus here on what we refer to as the policy 3 cloning setting which allows for offline queries of an expert or expert policy. This 4 setting arises naturally in a number of problems, especially as a component of 5 other algorithms. We achieve a very high level of data efficiency in transferring 6 behavior from an expert to a student policy for high Degrees of Freedom (DoF) 7 control problems using our augmented policy cloning (APC) approach, which 8 combines conventional image-based data augmentation to build invariance to 9 image perturbations with an expert-aware offline data augmentation approach that 10 induces appropriate feedback-sensitivity in a region around expert trajectories. We 11 show that our method increases data-efficiency of policy cloning, enabling transfer 12 of complex high-DoF behaviors from just a few trajectories, and we also show 13 benefits of our approach in the context of algorithms in which policy cloning is a 14 constituent part. 15
references.authors: Nicolas Heess
references.title: Information asymmetry in kl-regularized rl, 2019
references.authors: 339 Teh
references.title: Behavior priors for efficient reinforcement learning
references.venue: arXiv preprint arXiv:2010.14274,
sections.text: We present a simple, yet powerful data-augmentation technique to enable data-1 efficient learning from parametric experts. Whereas behavioral cloning refers to2 learning from samples of an expert, we focus here on what we refer to as the policy3 cloning setting which allows for offline queries of an expert or expert policy. This4 setting arises naturally in a number of problems, especially as a component of5 other algorithms. We achieve a very high level of data efficiency in transferring6 behavior from an expert to a student policy for high Degrees of Freedom (DoF)7 control problems using our augmented policy cloning (APC) approach, which8 combines conventional image-based data augmentation to build invariance to9 image perturbations with an expert-aware offline data augmentation approach that10 induces appropriate feedback-sensitivity in a region around expert trajectories. We11 show that our method increases data-efficiency of policy cloning, enabling transfer12 of complex high-DoF behaviors from just a few trajectories, and we also show13 benefits of our approach in the context of algorithms in which policy cloning is a14 constituent part.15
  1 Introduction16
  In various control and reinforcement learning settings, there is a need to transfer behavior from an17 expert policy to a student policy. Broadly, when only samples from the expert policy are available, the18 standard approach is to employ a version of regression from states to actions. This class of approaches19 for producing a policy is known as behavioral cloning [Pomerleau, 1989, Michie and Sammut, 1996].20 Behavioral cloning is quite flexible and supports the setting where the expert trajectories come from a21 human teleoperating the relevant system directly, as well as various settings where the trajectories are22 sampled from other controllers, which themselves may have been trained or scripted. However, for23 any of the settings where the expert policy is actually available, rather than just samples from the24 expert, it is reasonable to suspect that sampling random rollouts from the expert policy followed by25 performing behavioral cloning is not the optimally efficient approach for transferring behavior from26 the expert to the student. Once a trajectory has been sampled via an expert rollout, there is actually27 additional information available that can be ascertained in the neighborhood of the trajectory, without28 having to perform an additional rollout, via the local feedback properties of the expert.29
  In this work, we refer to this setting, where we want to transfer from an expert policy to a student30 policy, while assuming the expert policy can be queried, as policy cloning. Naturally, there is still31 often an incentive to reduce the total number of rollouts, which may require actually collecting data32 in an unsafe or costly fashion, especially for real-world control problems. As such, there is an aim to33 characterize any efficiency that can be gained in learning from small numbers of rollouts without as34 much concern for how many offline queries are required of the expert policy. If one has primarily35 encountered behavioral cloning in the context of learning from human demonstrations, policy cloning,36
  Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
  with an available expert policy may seem contrived. However, policy cloning naturally arises in many37 settings. For example, we may have multiple experts that we wish to consolidate into a single neural38 network policy or there may be memory considerations that motivate compressing a large expert39 network into a smaller model with similar behavior. Perhaps most natural are settings in which an40 expert policy is costly or slow to execute, for example due to running a compute intensive procedure41 such as model predictive control (MPC) on specialized hardware (e.g. GPU); in such settings, the42 aim is to transfer expert behavior to a parametric student policy that amortizes the cost. DAGGER is43 one well known approach for efficiently transferring behavior from an expert to a student under these44 kinds of constraints [Ross et al., 2011]. In a separate setting, the expert may be suboptimal and the45 student needs to learn from expert while also being able to exceed the expert performance, perhaps46 by continuing to learn from a task via RL. This problem has been described as kickstarting in one47 incarnation [Schmitt et al., 2018], but also can arise when learning from behavioral priors [Tirumala48 et al., 2020], [Galashov et al., 2019], as also happens, for example, in Distral [Teh et al., 2017].49
  To improve data-efficiency in supervised settings generally, including in behavioral cloning settings,50 it is reasonable to consider data augmentation. Data augmentation refers to applying perturbations to51 a finite training dataset to effectively amplify its diversity, usually in the hopes of producing a model52 that is invariant to the class of perturbations performed. For example, in the well studied problem53 of object classification from single images, it is known that applying many kinds of perturbation54 should not affect the object label, so a model can be trained with many input perturbations all yielding55 the same output [Shorten and Khoshgoftaar, 2019]. This setting is fairly representative, with data56 augmentation usually intended to make the model “robust" to nuisance perturbations of the input.57 This class of image-perturbation has also been recently demonstrated to be effective in the context of58 control problems in the offline RL setting [Yarats et al., 2021, Laskin et al., 2020].59
  Critically, for control problems it is not the case that the action should be invariant to the input state.60 Or rather, while it does make sense for a control policy to be invariant to certain classes of sensor61 noise, an important class of robustness is that the policy is appropriately feedback-responsive. This is62 to say that for small perturbations of the state of the control system, the optimal action is different63 in precisely the way that the expert implicitly knows. This has been recognized and exploited in64 previous research that has distilled feedback-control plans into controllers [Mordatch and Todorov,65 2014, Mordatch et al., 2015, Merel et al., 2019]. A similar intuition also underlies schemes which66 inject noise into the expert during rollouts to sample more comprehensively the space of how the67 expert recovers from perturbations [Laskey et al., 2017, Merel et al., 2019].68
  In this work, we leverage this insight to develop a highly efficient policy cloning approach that69 makes use of both classes of data augmentation. For a high-DoF control problem that operates only70 from state (humanoid run task from DeepMind control suite [Tunyasuvunakool et al., 2020]), we71 demonstrate the feasibility of policy cloning that employs state-based data augmentation with expert72 querying to transfer the feedback-sensitive behavior of the expert in a region around a small number73 of rollouts. Then on a more difficult high-DoF control problem that involves both state-derived and74 egocentric image observations (humanoid running through corrdiors task from DeepMind control75 suite [Tunyasuvunakool et al., 2020]), we combine the state-based expert-aware data augmentation76 with a separate image augmentation intended to induce invariance to image perturbations. Essentially77 our expert-aware data augmentation involves applying random perturbations to the state-derived78 observations, and training the student to match the expert-queried optimal action at each perturbed79 state, thereby gaining considerable knowledge from the expert without performing excessive rollouts80 simply to cover the state space around existing trajectories. Our approach compares favorably to81 sensible baselines, including the naive approach of attempting to perform behavioral cloning with82 state perturbations, which seeks to induce invariance (as proposed in [Laskin et al., 2020]) rather than83 feedback-sensitivity to state-derived observations.84
  In the presentation that follows, we will describe the problem setting (Section 2) as well as our85 approach (Section 3), describe the domains we employ and present our initial experiments (Section 4),86 show that our augmented policy cloning approach works well when used as a component of other87 algorithms like DAGGER and kickstarting (Section 5), and finally close with a discussion (Section 6).88
  2 Problem description89
  2.1 Expert-driven learning90
  We start by introducing a notion of expert-driven learning that will be used throughout the paper.91 At first, we present a general form of the expert-driven objective and then introduce a few concrete92 examples. We consider a standard Reinforcement Learning (RL) problem. We present the domain as93 an MDP with continuous states for simplicity, however the problem definition is similar for a POMDP94 with observations derived from the state. Formally, we describe the MDP in terms of a continuous state95 space S ∈ Rn (for some n > 0), an action space A, transition dynamics p(s′|s, a) : S ×A → p(S),96 and a reward function r : S × A → R. Let Π be a set of parametric policies, i.e. of mappings97 πθ : S → p(A) from the state space S to the probability distributions over actions A, where θ ∈ Rm98 for some m > 0. For simplicity of the notation, we omit the parameter in front of the policy, i.e.99 π = πθ and optimizing over the set of policies would be equivalent to the optimizing over a set of100 parameters. A reinforcement learning problem consists in finding such a policy π that it maximizes101 the expected discounted future reward:102
  J(π) = Ep(τ) [∑ t γtr(at|st) ] , (1)
  where p(τ) = p(s0) ∏ t p(at|st)p(st+1|st, at) is a trajectory distribution. We assume the existence103 of an expert policy πE(a|s). This policy could be used to simplify the learning of a new policy on the104 same problem. Formally, we construct a new learning objective which aims to maximize the expected105 reward of the problem in hand as well as to clone the expert policy:106
  J(π, πE) = αJ(π)− λD(π, πE), (2) where D is some measure of distance of π from πE and α ≥ 0, λ ≥ 0 are parameters measuring107 importance of both objectives. In most of the applications, α ∈ {0, 1} and λ ≥ 0 represents a relative108 importance of cloning an expert policy with respect to the RL objective.109
  2.2 Behavioral cloning (BC)110
  One important instance of the objective (2) with α = 0, λ = 1 is behavioral cloning. In this case, the111 measure of distance is defined as:112
  DBC(π, πE) = −E(a,s)∈BE [log π(a|s)] (3) Here, BE = {(si, ai), i = 1, . . . , N}, N > 0 is a fixed dataset containing expert data. Minimizing113 the objective (3) is be equivalent to maximizing the likelihood of the expert data under the policy π.114 The action in eqn. (3) can be replaced by πE(s) for deterministic policies or by the mean or the mode115 for stochastic policies (e.g., by the mean µE(s) for Gaussian policies πE(·|s) = N (µE(s), σE(s))).116
  2.3 DAGGER117
  Performance of Behavioral Cloning (BC) can be limited due to the fixed dataset, since the resulting118 policy may fail to generalize to states outside the training distribution. A different approach, known119 in the literature as DAGGER [Ross et al., 2011] was proposed to overcome this limitation. In this120 setting, the expert is queried in states visited by the student, thus reducing distribution shift. In our121 notation, this corresponds to α = 0, λ = 1 in eqn. (2) and the measure of distance is defined as:122
  DDAGGER(π, πE) = −Epβ(τ)[log π(a ′ t|st)], (4)
  where pβ(τ), β ∈ [0, 1] is a trajectory distribution where actions are sampled according to the mixture123 policy between a student and an expert:124
  p(a|s) = βπ̃(a|s) + (1− β)πE(a|s), (5) The action a′t in eqn. (4) is resampled from the expert policy for the state st, i.e., a ′ t ∼ πE(·|st).125 As in Section 2.2, for stochastic experts this action can be replaced by the mean or mode of the126 distribution in eqn. (4). The policy π̃(a|s) corresponds to a frozen version of student policy π so127 that the gradient ∇πDDAGGER(π, πE) is not taken with respect to p(a|s). Note that even though, in128 eqn. (4) we collect data from the environment, the setting nevertheless corresponds to pure imitation129 learning since expected reward is not directly maximized.130
  2.4 Kickstarting131
  In eqn. (2), we combine both maximization of expected task reward and minimization of distance to132 the expert. In literature, it is known as Kickstarting [Schmitt et al., 2018]. In this case, in the objective133 from eqn. (2), α = 1, and λ ≥ 0. As the measure of distance, we use the cross-entropy from expert134 to a student, similarly to [Schmitt et al., 2018]:135
  J(π, πE) = J(π)− λEp(τ) [ −EπE(a|s) log π(a|s) ] (6)
  where p(τ) is a trajectory distribution, where actions are sampled according to the student policy136 π(·|s). Usually, in the Kickstarting setting, the expert is sub-optimal and the goal is to train a policy137 that eventually outperforms the expert. Thus, it is customary to reduce λ over the course of training.138 Yet, for simplicity, in our experiments we keep this coefficient fixed.139
  3 Augmented policy cloning140
  The previous section has demonstrated that the objective corresponding to the cloning behavior from141 the parametric expert policy could arise in multiple scenarios. In this section we propose a new and142 simple method which can significantly improve the data efficiency of the approaches described in143 Section 2. We explain the basic idea for BC, but its generalization to other expert-driven learning144 approaches described in Section 2 is straightforward. In Section 5 we show results for these problems.145
  When optimizing the objective (3), for every state s ∈ DE from the expert trajectories dataset, we146 consider a small Gaussian state perturbation:147
  δs ∼ N (0, σ2s) (7)
  which produces a new virtual state:148 s′ = s+ δs (8)
  Then, for this state we query the expert and obtain a new action149
  a′ ∼ πE(·|s+ δs) (9)
  We then augment the dataset DE with these new pairs of virtual states and actions. More explicitly150 the idea can be expressed in terms of the following objective:151
  D(π, πE)APC = E(a,s)∈BE [log π(a|s) + Eδs∼N (0,σ2s),a′∼πE(·|s+δs) log π(a ′|s+ δs)] (10)
  We call this approach Augmented Policy Cloning (APC) as it queries the expert policy to augment152 the training data. This approach is different from a naive data-augmentation technique, where a new153 state would be generated, but associated with the original action (and not a new one). It therefore154 allows to build policies which are feedback-responsive with respect to the expert. We formulate APC155 algorithm for BC in Algorithm 1.156
  4 Core Results: Evaluation of Augmented Policy Cloning157
  4.1 Domains158
  To study how our method performs on complex control domains, we consider two complex, high-DoF159 continuous control tasks involving control of a physically simulated humanoid body. Both domains160 are implemented using the MuJoCo physics engine [Todorov et al., 2012] and are available in the161 dm_control repository [Tunyasuvunakool et al., 2020]. The first task is the standard control suite162 Run task, where the Humanoid body needs to run at a target speed and observations are based on163 proprioception. The second task is the Walls task which requires the same Humanoid body to run164 along a corridor and avoid walls, using both proprioception and egocentric vision as observations.165 Both of these problems are rather challenging insofar as they require stabilization and locomotion166 control of a relatively complex humanoid body with 21 actuated DoFs, in one case using vision to167 guide the movement. Note these environments are related to the domains that have been proposed168 for use in offline RL benchmarks [Gulcehre et al., 2020]; however, the experiments we perform in169 this work require availability of the expert policy, so we do not use offline data, but instead train170 new experts and perform experiments in the very low data regime. For more details, please refer to171 Section 1.1 in Supplementary Material.172
  Algorithm 1 Augmented Policy Cloning (APC) Parametric student policy: πθ Initial parameters: θ0 expert policy: πE Dataset BE = {(si, ai), i = 1, . . . , N}, N > 0 of expert state-action pairs State perturbation noise σs Learning rate α Number of augmented samples: M Number of gradient updates: K Size of a batch: L for k=1,. . . ,K do
  Sample a batch of pairs {(ai, si)}Li=1 ∼ BE For each state si, sample M perturbations δsj ∼ N (0, σs), j = 1, . . . ,M Construct M virtual states s′i,j = si + δsj , i = 1, . . . , L, j = 1, . . . ,M Resample new actions from expert a′i,j ∼ πE(·|s′i,j) For Gaussian experts, the action ai = µE(si) and the new actions are a′i,j = µE(s ′ i,j) Compute the empirical negative log-likelihood: L = − [ log πθk(ai|si) + 1M ∑M j=1 log πθk(a ′ i,j |s′i,j)
  ] Update the parameters θk+1 = θk − α∇θL
  end for
  For each task, we train expert policies to convergence using the MPO algorithm Abdolmaleki et al.173 [2018]. Since the expert policy essentially saturates task performance, for each task, we keep three174 partially trained experts such that we can assess the ability of the kickstarting approach to outperform175 sub-optimal experts. We refer to the different experts as Low, achieving approximatively 25 % of the176 optimal policy reward, Medium, achieving around 50 % of the performance and High, corresponding177 to the converged policy. Each expert is represented by a Gaussian policy. For more details, please178 refer to Section 1.2 in Supplementary Material.179
  4.2 Applying Augmented Policy Cloning180
  First, we evaluate the performance of APC in fitting a fixed dataset of expert trajectories. In order181 to study the data efficiency of the method, we construct datasets containing different numbers of182 expert trajectories. The expert policies are represented by conditionally Gaussian distributions,183 i.e. πE(·|s) = N (µE(s), σ(s)). Thus, to assess the robustness of our method to expert noise we184 produce trajectories using the experts’ mean but adding different levels of (homoscedastic) zero-mean185 Gaussian noise σE .186 a ∼ N (µE(s), σE) Note that in addition to policy noise σE which is introduced when sampling trajectories, initial pose187 and environment layout (for the Walls task) are also sampled randomly for each episode. We consider188 4 levels of expert policy noise: Deterministic, which uses the Gaussian mean for the action, Low,189 with σE = 0.2, Medium σE = 0.5 and High σE = 1.0.190
  For the APC method, we rely on Algorithm 1. For baselines, we consider BC algorithm from eqn. (3)191 as well as a simple modification of BC, where we apply, similar to APC, state perturbations as in192 eqn. (7) and eqn. (8), but we do not produce a new action from the expert. We call this approach193 Naive Augmented Behavior Cloning (Naive ABC) which essentially corresponds to robustification194 of the student policies with respect to state perturbation and is similar in spirit to standard data-195 augmentation approaches. For vision-based tasks, we consider random crop augmentations of size196 48x48 (downsampled from the input image of 64x64), similar to Laskin et al. [2020]. When the197 image augmentations are used we add "with image" to the method name. On top of that, we consider198 a variant, where only image augmentation is used, which we call Naive ABC (image only). For199 all methods, as an action in the objective from eqn. (3), we use an expert mean µE(s). We train200 all approaches to convergence (300K learning iterations on Walls and 13M learning iterations on201 Run). Each learning iteration corresponds applying gradients to 64 trajectories, each containing202 10 time steps. After each learning iteration, we evaluate the policy on both a validation set (50203 random instances of the environment) and a test set (150 random instances of the environments).204
  We apply early-stopping based on the validation set performance to select the best model and205 report corresponding performance on the test set. For more details, please refer to Section 1.3 in206 Supplementary Material. As an additional evaluation, we test robustness of the obtained policies to a207 fixed amount of noise during execution. For a learned student policy π(·|s) = N (µ(s), σ(s)), we208 evaluate it by executing an action:209 a ∼ N (µ(s), σ), where σ is the fixed amount of student noise. We consider similar noise magnitudes as for the210 expert. For APC and Naive ABC, we sweep over state perturbation noise levels and choose the ones211 performing the best on the validation set. For APC, we use σs = 0.1 for Run and σs = 1.0 for Walls.212 For Naive ABC, we use σs = 0.001 for Run and σs = 0.01 for Walls. The ablation experiments213 over noise levels for APC and Naive ABC are presented in Section 2 of the Supplementary Material214 (Figure 1 and Figure 2).215
  The first of results, in Figure 1 (left) demonstrates the increased data efficiency of APC over BC and216 Naive ABC in terms of number of trajectories. The noise level of the expert and student are fixed to217 Low for the ease of comparison. We also see that Naive ABC performs similarly to BC. To further218 push the limits of data efficiency, we conducted a variant where a dataset contains only 1 full trajectory219 (1000 timesteps for Run and around 2k timesteps for Walls) along with multiple short trajectories220 (200 time-step only). This dramatically reduces the amount of expert data available to learn from.221 However, we hypothesise that in the environments considered, much of the diversity of the trajectories222 arises due to initial state variation. This setting might arise in domains where execution is costly,223 such as robotics applications. In such setting we might have a few longer trajectories along with a224 patchwork of shorter trajectories covering more diverse parts of the state space. The results for this225 experiment are given in figure 1 (right). Again, we see that APC is significantly more efficient than226 BC and Naive ABC. Interesting to note that APC picks up quite a high performance after observing227
  10 (1 full and 9 short) trajectories for Run task and 100 (1 full and 99 short) trajectories for the Walls228 task. In Section 3 in Supplementary material, we provide additional results for Walls task when we229 use image-based perturbations.230
  In the next experiment, in order to understand how robust our method to noise, we study the impact231 of different levels of student and expert noises on performance. For each run, we use a dataset of232 100 trajectories. The results are given in figure 2, where each column corresponds to a different233 level of expert noise, and the X-axis represents different levels of student noise. At first, we observe234 that APC is consistently more robust than BC and Naive ABC for any level of expert and student235 noise. On top of that, we can notice that for any fixed level of expert noise, the performance degrades236 when a student noise increases. Finally, we see that for higher noise levels of expert, the learned237 student performs better in the high noise regime. It is consistent with the intuition - training on noisy238 trajectories leads to a more robust policy. Overall, APC leads the most robust policy.239
  5 Additional Results: Augmented Policy Cloning as a subroutine240
  5.1 DAGGER with data augmentation241
  As described in Section 2.3, DAGGER [Ross et al., 2011] is a more sophisticated approach where242 data is collected from the real environment by executing a policy from eqn. (5), which is a mixture243 between a student and an expert. In this section we study how data augmentation approaches affect244 the data efficiency of the DAGGER algorithm.245
  We consider similar baselines for both tasks as in the previous section. For an expert policy that has246 been pre-trained via MPO [Abdolmaleki et al., 2018], we perform online rollouts for two values of247 the expert-student mixing coefficient, β = 0 and β = 0.3 (see eqn. 5). Since both student and expert248 are Gaussian distributions, instead of using a log π in eqn. (4), we could use a state-conditional cross249 entropy from an expert to a student,H[πE(·|s)||π(·|s)]. Empirically, we found that it worked better250 than using log π. We demonstrate a comparison in Section 4 in Supplementary Material. We run251 the experiments in a data-restricted setup such that for every collected trajectory (10 time-steps), we252 apply 10 gradient steps, using a replay-buffer to store the past experience. Additional experimental253 details are given in Section 1.4 in Supplementary Material. Results are shown in Figure 3. We see254 that APC and its vision variant outperform BC and Naive ABC similarly to the behavior cloning255 experiments. While we observe that image augmentation can help, we see that the primary advantage256 comes from the state-based augmentation for APC. For the Run task, we observe that all DAGGER257 methods achieve slightly lower performance than an expert policy. We speculate that this is due to258 insufficient coverage of the state space during training.259
  5.2 Kickstarting with data augmentation260
  1.0 e9 2.0 e9 3.0 e9 4.0 e9
  0
  250
  500
  750
  Ep iso
  de re
  wa rd
  Run, High
  Method APC BC Naive ABC APC with image Naive ABC (image only) Naive ABC with image MPO
  1.0 e9 2.0 e9 3.0 e9 4.0 e9
  Run, Medium
  1.0 e9 2.0 e9 3.0 e9 4.0 e9
  Run, Low
  0.0 e9
  0.5 e9
  1.0 e9
  0
  250
  500
  750
  Walls, High
  0.0 e9
  0.5 e9
  1.0 e9
  Walls, Medium
  0.0 e9
  0.5 e9
  1.0 e9
  Environment steps
  Walls, Low
  Figure 4: Kickstarting results. On the X-axis we show the number of environment steps. On the Y-axis we report averaged across 3 seeds episodic reward achieved by the student. We report confidence intervals in the shaded areas. For Run task, the confidence intervals are very small and are not visible. Each row indicates a task, whereas a column corresponds to the expert type. Dashed black line shows the expert performance.
  A similar in spirit approach is kickstarting Schmitt et al. [2018], where we solve an RL task as well261 as cloning the expert policy. Similarly to previous section, we apply APC in kickstarting on the262 cross entropy term in eqn. (6). We use 3 types of expert policy as described in Section 4.1. We run263 experiments using a distributed setup with 64 acting policies and 1 learner, querying the batches264 of trajectories (of size 10) from a replay buffer. On top of running BC methods, we also report the265 performance of MPO Abdolmaleki et al. [2018] learning from scratch on the task of interest. All266 details are given in Section 1.5 in Supplementary Material. The results are given in figure 4.267
  We observe that APC performs better than Naive ABC on Run task and similarly on Walls task. Both268 approaches perform better than BC and learning from scratch. We hypothesise that the reason of not269 seeing a consistent advantage could be due to two factors. As we are in a high-data regime, since270 there is no limit on relative acting / learning ratio, and acting policies are not restricted to collect271 trajectories, it is unclear whether data-augmentation should help. In addition, we use reward signal272 which makes the impact of expert cloning less important. Note that the resulting agent is less data273 efficient in these experiments; this is because we do not control the relative ratio between acting274 and learning (i.e., no rate-limiting on the learner, due to instability of kickstarting experiments when275 rate-limiting was explored). Furthermore, unlike in kickstarting Schmitt et al. [2018], we do not276 use an annealing schedule of λ to make the experiments simpler, but we still observe that a fixed277 coefficient helps to kickstart an experiment and outperform an expert policy. On top of that, we see278 that image-based augmentation have less of impact in this setting.279
  6 Discussion280
  Many expert-driven learning approaches actually have access to an expert that can be queried;281 however, this opportunity is rarely exploited fully. In this work we demonstrated a general scheme for282 more efficient transfer of expert behavior by augmenting expert trajectory data with virtual, perturbed283 states as well as the expert actions in these virtual states. This data augmentation technique is widely284 applicable and we demonstrated that it improves data efficiency when used in place of behavioral285 cloning both in the offline setting or when behavioral cloning is used as a step within DAGGER or286 kickstarting.287
  Critically, data efficiency is generally very important in realistic applications, where new data288 acquisition cost could be high. In particular, settings involving deployment of policies in the real289 world, such as robotics applications, may benefit from an ability to efficiently transfer expert policy290 behavior from one neural network to another (for compression or execution speed reasons), or to291 combine behavior from multiple experts into a single neural network. While overall, we consider the292 present work to be fairly basic research with limited ethical impact, insofar as our approach decreases293 the amount of data which needs to be collected through processes which could potentially be unsafe294 or costly, there is a potential positive social value.295
  The limitations of our approach consist in the reliance on the ability to query expert policy for the296 perturbed states which reduces the amount of applications where the method could be used. Another297 limitation is the reliance on the continuous state spaces. In discrete state spaces, it is unclear whether298 a small perturbation in state would result in a valid action from an expert.299
  In future work, we plan to explore how our proposed augmentation technique can be leveraged in the300 context of KL-regularized RL with behavior priors.301
  References302 Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin303 Riedmiller. Maximum a posteriori policy optimisation, 2018.304
  Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan305 Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and306 Nicolas Heess. Information asymmetry in kl-regularized rl, 2019.307
  Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna,308 Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. Rl unplugged:309 A collection of benchmarks for offline reinforcement learning. Advances in Neural Information310 Processing Systems, 33, 2020.311
  Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for312 robust imitation learning. In Conference on robot learning, pages 143–156. PMLR, 2017.313
  Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.314 Reinforcement learning with augmented data, 2020.315
  Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne,316 Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con-317 trol, 2019.318
  Donald Michie and Claude Sammut. Behavioural Clones and Cognitive Skill Models, page 387–395.319 Oxford University Press, Inc., USA, 1996. ISBN 019853860X.320
  Igor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory321 optimization. In Robotics: Science and Systems, volume 4, 2014.322
  Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive323 control of diverse complex characters with neural networks. Advances in Neural Information324 Processing Systems, 28:3132–3140, 2015.325
  Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Technical report,326 Carnegie-Mellon, 1989.327
  Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and328 structured prediction to no-regret online learning, 2011.329
  Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M.330 Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali331 Eslami. Kickstarting deep reinforcement learning, 2018.332
  Connor Shorten and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning.333 Journal of Big Data, 6, 07 2019. doi: 10.1186/s40537-019-0197-0.334
  Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia335 Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning.336 arXiv preprint arXiv:1707.04175, 2017.337
  Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu,338 Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye339 Teh, et al. Behavior priors for efficient reinforcement learning. arXiv preprint arXiv:2010.14274,340 2020.341
  Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.342 In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033.343 IEEE, 2012.344
  Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom345 Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for346 continuous control. Software Impacts, 6:100022, 2020.347
  Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep348 reinforcement learning from pixels. In 9th International Conference on Learning Representations,349 ICLR, volume 2021, 2021.350
  7 NeuRIPS 2021 checklist351
  • Do the main claims made in the abstract and introduction accurately reflect the pa-352 per’s contributions and scope?353 Yes, we believe that the claims in the abstract and introduction are accurately reflected by354 the paper’s contributions and scope.355
  • Have you read the ethics review guidelines and ensured that your paper conforms to356 them?357 Yes.358
  • Did you discuss any potential negative societal impacts of your work?359 Yes. We included this in the Discussion section.360
  • Did you describe the limitations of your work?361 Yes, we discuss these in the "Discussion" section.362 • Did you state the full set of assumptions of all theoretical results363
  Not applicable.364 • Did you include complete proofs of all theoretical results365
  Not applicable.366 • Did you include the code, data, and instructions needed to reproduce the main exper-367 imental results ?368 We included complete details in the supplementary material which could be used to reproduce369 the experimental results.370
  • Did you specify all the training details?371 Yes, in the main paper and supplementary material.372 • Did you report error bars?373
  We did. For every plot we reported the mean and standard deviations averaged across a374 number of independent random evaluations.375
  • Did you include the amount of compute and the type of resources used?376 Yes, we did, in the supplementary material.377 • If your work uses existing assests, did you cite the creators?378
  Yes. We use DeepMind control suite tasks and we cite the appropriate publications.379 • Did you mention the license of the assets?380
  We did not explicitly mention the licence as it is clear from the reference.381 • Did you include any new assets either in the supplementary material os as a URL?382
  No.383 • Did you discuss whether and how consent was obtained from people whose data you’re384 using / curating?385 Since the assets and references are in the public access, no consent was required.386
  • Did you discuss whether the data you are using / curating contains personally identifi-387 able information or offensive content?388 The assets which we used did not include any personally identifiable information nor389 offensive content.390
  • Did you include the full text of instructions given to participants and screenshots, if391 applicable?392 Not applicable.393
  • Did you describe any potential participant risks, with links to Institutional Review394 Board (IRB) approvals, if applicable395 Not applicable.396
  • Did you include the estimated hourly wage paid to participants and the total amount397 spent on participant compensation?398 Not applicable.399
title: Data augmentation for efficient learning from parametric experts