summaries: The paper claims that it identifies a fundamental issue in model-based reinforcement learning methods. The issue is called objective mismatch, which arises when one objective is optimized (for example, model learning objective) without taking into consideration of another objective (for example policy optimization). The author shows several experiments to illustrate the issue and proposes a method to mitigate it by assigning priorities to samples when training the model.

summaries: This paper discusses the old problem of mismatch between the ultimate reward obtained after optimizing a  decision (planning or control) over a probabilistic model (of dynamics) and  the training  objective for the model (log-likelihood). Experiments highlight that the NLL and reward can be very poorly correlated, that improvements in NLL initially improve reward but can later degrade it, and that models with similar NLLs can lead to very different rewards. A  reweighting trick is proposed and summarily evaluated.

abstractText: Model-based reinforcement learning (MBRL) has been shown to be a powerful framework for data-efficiently learning control of continuous tasks. Recent work in MBRL has mostly focused on using more advanced function approximators and planning schemes, leaving the general framework virtually unchanged since its conception. In this paper, we identify a fundamental issue of the standard MBRL framework – what we call the objective mismatch issue. Objective mismatch arises when one objective is optimized in the hope that a second, often uncorrelated, metric will also be optimized. In the context of MBRL, we characterize the objective mismatch between training the forward dynamics model w.r.t. the likelihood of the one-step ahead prediction, and the overall goal of improving performance on a downstream control task. For example, this issue can emerge with the realization that dynamics models effective for a specific task do not necessarily need to be globally accurate, and vice versa globally accurate models might not be sufficiently accurate locally to obtain good control performance on a specific task. In our experiments, we study this objective mismatch issue and demonstrate that the likelihood of the one-step ahead prediction is not always correlated with downstream control performance. This observation highlights a critical flaw in the current MBRL framework which will require further research to be fully understood and addressed. We propose an initial method to mitigate the mismatch issue by reweighting dynamics model training. Building on it, we conclude with a discussion about other potential directions of future research for addressing this issue.

authors:
P. Abbeel
M. Quigley
A.Y. Ng

title: Using inaccurate models in reinforcement learning

venue: In International Conference on Machine Learning (ICML), pages

authors:
B. Amos
I. Jimenez
J. Sacks
B. Boots
J.Z. Kolter

title: Differentiable MPC for end-to-end planning and control

venue: In Neural Information Processing Systems,

authors:
E. Banijamali
R. Shu
M. Ghavamzadeh
H. Bui
A. Ghodsi

title: Robust locally-linear controllable embedding

venue: arXiv preprint arXiv:1710.05373,

authors:
S. Bansal
R. Calandra
T. Xiao
S. Levine
C.J. Tomlin

title: Goal-driven dynamics learning via Bayesian optimization

venue: In IEEE Conference on Decision and Control (CDC),

title: Dynamic programming and optimal control, volume 1

venue: Athena scientific Belmont, MA,

title: Applied optimal control: optimization, estimation and control

authors:
K. Chua
R. Calandra
R. McAllister
S. Levine

title: Deep reinforcement learning in a handful of trials using probabilistic dynamics models

venue: In Advances in Neural Information Processing Systems,

authors:
M. Deisenroth
C.E. Rasmussen

title: Pilco: A model-based and data-efficient approach to policy search

venue: In Proceedings of the 28th International Conference on machine learning

authors:
P. Donti
B. Amos
J.Z. Kolter

title: Task-based end-to-end model learning in stochastic optimization

authors:
C. Gelada
S. Kumar
J. Buckman
O. Nachum
M.G. Bellemare

title: Deepmdp: Learning continuous latent space models for representation learning

authors:
D. Ghosh
A. Gupta
S. Levine

title: Learning actionable representations with goal-conditioned policies

venue: arXiv preprint arXiv:1811.07819,

authors:
I. Goodfellow
J. Shlens
C. Szegedy

title: Explaining and harnessing adversarial examples

venue: In International Conference on Learning Representations (ICLR),

authors:
D. Hafner
T. Lillicrap
I. Fischer
R. Villegas
D. Ha
H. Lee
J. Davidson

title: Learning latent dynamics for planning from pixels

venue: arXiv preprint arXiv:1811.04551,

authors:
B. Ichter
M. Pavone

title: Robot motion planning in learned latent spaces

venue: IEEE Robotics and Automation Letters,

authors:
M. Janner
J. Fu
M. Zhang
S. Levine

title: When to trust your model: Model-based policy optimization

venue: arXiv preprint arXiv:1906.08253,

authors:
N. Jiang
A. Kulesza
S. Singh
R. Lewis

title: The dependence of effective planning horizon on model accuracy

venue: In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems,

authors:
R. Jonschkowski
D. Rastogi
O. Brock

title: Differentiable particle filters: End-to-end learning with algorithmic priors

venue: arXiv preprint arXiv:1805.11122,

authors:
J. Joseph
A. Geramifard
J.W. Roberts
J.P. How
N. Roy

title: Reinforcement learning with misspecified model classes

venue: IEEE International Conference on Robotics and Automation,

authors:
N.R. Ke
A. Singh
A. Touati
A. Goyal
Y. Bengio
D. Parikh
D. Batra

title: Learning dynamics model in reinforcement learning by incorporating the long term future

title: Optimal control theory: an introduction

venue: Courier Corporation,

authors:
D.J. Mankowitz
N. Levine
R. Jeong
A. Abdolmaleki
J.T. Springenberg
T. Mann
T. Hester
M. Riedmiller

title: Robust reinforcement learning for continuous control with model misspecification

authors:
Ð. Miladinović
M.W. Gondal
B. Schölkopf
J.M. Buhmann
S. Bauer

title: Disentangled state space representations

venue: arXiv preprint arXiv:1906.03255,

authors:
A. Nagabandi
I. Clavera
S. Liu
R.S. Fearing
P. Abbeel
S. Levine
C. Finn

title: Learning to adapt in dynamic, real-world environments through meta-reinforcement learning

venue: arXiv preprint arXiv:1803.11347,

authors:
M. Okada
L. Rigazio
T. Aoshima

title: Path integral networks: End-to-end differentiable optimal control

venue: arXiv preprint arXiv:1706.09597,

authors:
M. Pereira
D.D. Fan
G.N. An
E. Theodorou

title: Mpc-inspired neural network policies for sequential decision making

venue: arXiv preprint arXiv:1802.05803,

authors:
S. Singh
S.M. Richards
V. Sindhwani
J.-J.E. Slotine
M. Pavone

title: Learning stabilizable nonlinear dynamics with contraction-based regularization, 2019

authors:
J. Sorg
R.L. Lewis
S.P. Singh

title: Reward design via online gradient ascent

authors:
A. Srinivas
A. Jabri
P. Abbeel
S. Levine
C. Finn

title: Universal planning networks

venue: arXiv preprint arXiv:1804.00645,

authors:
R.S. Sutton
A.G. Barto

title: Reinforcement learning: An introduction

venue: MIT press,

authors:
C. Szegedy
W. Zaremba
I. Sutskever
J. Bruna
D. Erhan
I. Goodfellow
R. Fergus

title: Intriguing properties of neural networks

venue: arXiv preprint arXiv:1312.6199,

title: Self-correcting models for model-based reinforcement learning

venue: In Thirty-First AAAI Conference on Artificial Intelligence,

title: Learning the reward function for a misspecified model

venue: arXiv preprint arXiv:1801.09624,

authors:
A. Tamar
G. Thomas
T. Zhang
S. Levine
P. Abbeel

title: Learning from the hindsight plan – episodic MPC improvement

venue: In IEEE International Conference on Robotics and Automation (ICRA),

authors:
S. Trimpe
A. Millane
S. Doessegger
R. D’Andrea

title: A self-tuning lqr approach demonstrated on an inverted pendulum

venue: IFAC Proceedings Volumes,

authors:
T. Wang
J. Ba

title: Exploring model-based planning with policy networks

venue: arXiv preprint arXiv:1906.08649,

authors:
M. Watter
J. Springenberg
J. Boedecker
M. Riedmiller

title: Embed to control: A locally linear latent dynamics model for control from raw images

venue: In Advances in neural information processing systems,

authors:
W. Whitney
R. Fergus

title: Understanding the asymptotic performance of model-based rl methods

authors:
G. Williams
N. Wagener
B. Goldfain
P. Drews
J.M. Rehg
B. Boots
E.A. Theodorou

title: Information theoretic MPC for model-based reinforcement learning

venue: In International Conference on Robotics and Automation (ICRA),

authors:
M. Zhang
S. Vikram
L. Smith
P. Abbeel
M.J. Johnson
S. Levine

title: Solar: Deep structured latent representations for model-based reinforcement learning

venue: arXiv preprint arXiv:1808.09105,

authors:
K. Zhou
J.C. Doyle
K. Glover

title: Robust and optimal control, volume 40

venue: Prentice hall New Jersey,

heading: 1 INTRODUCTION

text:
Model-based reinforcement learning (MBRL) is a popular approach for learning to control nonlinear systems that cannot be expressed analytically (Bertsekas, 1995; Sutton and Barto, 2018; Deisenroth and Rasmussen, 2011; Williams et al., 2017). MBRL techniques achieve the state of the art performance for continuous-control problems with access to a limited number of trials (e.g. Chua et al. (2018); Wang and Ba (2019)) and in controlling systems given only visual observations with no observations of the original system’s state (e.g. Hafner et al. (2018); Zhang et al. (2018)). MBRL approaches typically learn a forward dynamics model that predicts how the dynamical system will evolve when a set of control signals are applied. This model is classically fit with respect to the maximum likelihood of a set of trajectories collected on the real system, and then used as part of a control algorithm to be executed on the system (e.g., model-predictive control).
In this paper, we highlight a fundamental problem in the standard MBRL learning scheme: the objective mismatch issue. The learning of the forward dynamics model is decoupled from the subsequent controller and policy that it induces through the optimization of two different objective functions – negative log-likelihood (or its equivalent for deterministic models, e.g., the RMSE) of the single- or multi-step look-ahead prediction for the dynamics model, and task performance (i.e., reward) for the policy optimization. While the use of negative log-likelihood (NLL) for system identification is an historically accepted objective, it results in optimizing an objective that does not necessarily correlate to the actual performance. The contributions of this paper are to: 1) identify and formalize the problem of objective mismatch in MBRL; 2) examine the signs of and the effects of objective mismatch on simulated control tasks; 3) propose a initial mechanism to mitigate objective mismatch; 4) discuss the impact of objective mismatch on existing MBRL and outline future directions to address this problem.

heading: 2 MODEL-BASED REINFORCEMENT LEARNING

text:
Algorithm 1: Model-based RL Data: Initialize D from random actions while Improving do
Train model pθ(st+1|st, at) on D Collect data D′ with controller using pθ in real environment Aggregate dataset D = D ∪D′ We now outline the MBRL formulation used in the paper. At time t, we denote the state st ∈ Rds , the actions at ∈ Rda , and the reward r(st, at). We say that the MBRL agent acts in an environment governed by a state transition distribution p(st+1|st, at). We denote a parametric model to approximate this distribution with pθ(st+1|st, at). MBRL follows the general approach of an agent acting in its environment, learning a model of said environment, and then leveraging the model to act more effectively, summarized in Alg. 1. While iterating over parametric control policies, the agent collects measurements of state, action, next-state and forms a dataset D = {(sn, an, s′n)}Nn=1, where N is the sum of the length of all episodes. With the dynamics data D, the agent learns the environment in the form of a neural network forward dynamics model, learning an approximate dynamics pθ. This dynamics model is leveraged by a controller that takes in the current state st and returns an action sequence at:t+T maximizing the expected reward Eπθ(st) ∑t+T i=t r(si, ai), where T is the predictive horizon and πθ(st) is the set of state transitions induced by the model pθ. In our paper, we primarily use probabilistic networks designed to minimize the NLL of the predicted parametric distribution pθ, denoted as P , or ensembles of probabilistic networks denoted PE, and compare to deterministic networks minimizing the mean squared error (MSE), denoted D or DE. Unless otherwise stated we use the models as in PETS (Chua et al., 2018) with an expectation-based trajectory planner and a cross-entropy-method (CEM) optimizer.

heading: 3 OBJECTIVE MISMATCH AND ITS CONSEQUENCES

text:
The Origin of Objective Mismatch: The Subtle Differences between MBRL and System Identification Many ideas and concepts in model-based RL are rooted in the field of optimal control and system identification (Bertsekas, 1995; Zhou et al., 1996; Kirk, 2012; Bryson, 2018; Sutton and Barto, 2018). In system identification, the main idea is to use a two-step process where we first generate on the robot (optimal) elicitation trajectories τ to fit a dynamics model (typically analytical), and subsequently we apply this model to a specific task. This particular scheme has multiple assumptions: 1) the data collected cover the entire state-action space (done by appropriate elicitation trajectories); 2) the presence of large (virtually infinite) amount of data; 3) the global nature of the model resulting from the system identification process (which should be generalizable to a large range of tasks). With these assumptions, the general idea of system
identification is effectively to collect large amount of data covering the whole state-space to create
once a global model that is sufficiently accurate that we can at deployment time specify any desired task, and still obtain good performance.
When adopting the idea of learning the dynamics model used in optimal control for MBRL, it is important to consider if these assumptions still hold. The assumption of virtually infinite data is visibly in tension with the explicit goal of MBRL which is to reduce the number of interactions with the environment by being “smart” about the sampling of new trajectories. In fact, in MBRL the offline data collection performed via elicitation trajectories is largely replaced by on-policy sampling (with the exception of some initial motor babbling (Chua et al., 2018)) in order to explicitly reduce the need to collect large amount of data. Moreover – in practice – in the MBRL setting the data will not usually cover the entire state-action space, since they are generated by a policy trying to optimize one specific task. In conjunction with the use of non-parametric models (i.e., not analytical model), this results in learned models that are strongly biased towards capturing the distribution of the data, and that are only locally accurate. Nonetheless, this is at first sight not an issue since the common MBRL setting consider the learning aimed at solving only one specific task, and rarely test for generalization capabilities of the learned dynamics.
In practice, we can now see how the assumptions and goals of system identification are in contrast with the ones of MBRL. Understanding these differences and the downstream effects on algorithmic approach is crucial to design new families of MBRL algorithms.
Objective Mismatch During the MBRL process of iteratively learning a controller, the reward signal from the environment is diluted by the training of a forward dynamics model with a independent metric, as showing in Fig. 1. In our experiments, we highlight that the minimization of some network training cost does not hold a strong correlation to maximization of episode reward. As dynamic environments becoming increasingly complex in dimensionality, the assumptions of collected data distributions become weaker and over-fitting to different data poses an increased risk.
Formally, the problem of objective mismatch appears as two de-coupled optimization problems repeated over many cycles of Alg. 1, shown in Eq. (1a,b), which could be at the cost of minimizing the final reward. This loop becomes increasingly difficult to analyze as the dataset used for model training changes with each experimental trial – a step that is needed to include new data from previously unexplored states. In this paper we characterize the problems introduced by the interaction of these two optimization problems, but avoid to consider the interactions added by the changes in data distribution during the learning process, as this would significantly increase the complexity of the analysis. In addition, we discuss potential solutions, but do not make claims about the best way to do so, which is left for future work.
Training: argmin θ N∑ i=1 log pθ(s ′ i|si, ai), Control: argmax at:t+T Eπθ(st) t+T∑ i=t r(si, ai) (1a,b)

heading: 4 IDENTIFYING OBJECTIVE MISMATCH

text:
In this section, we experimentally study the issue of objective mismatch in MBRL to answer the following questions: 1) Does the distribution of models obtained from running a MBRL algorithm show a strong correlation between NLL and reward? 2) Are there signs of sub-optimality in the dynamics models training process that could be limiting performance? 3) What model differences are reflected in reward but not in NLL?
Experimental Setting In our experiments, we use two popular RL benchmark tasks: the cartpole and half cheetah. For more details on these tasks see Chua et al. (2018). For our cartpole experiments, we aggregate dynamics models at each episode from 50 runs of PETS (20 trials per run) paired with the on-policy episode reward achieved, giving the number of models Mcp = 1000. Our half cheetah experiments consist of 8 runs (300 trials per run) of PETS, tabulating the reward with the dynamics models again, yielding Mhc = 2400. All dynamics models are of depth 3, hidden width 500, and use the tanh activation function. With a large set of dynamics models in the feasible space of on-policy models, we then used a series of control datasets to evaluate the NLL of the dynamics models versus performance. The datasets we use are designed to investigate how different assumptions made in MBRL trickle down into final performance. We start with expert datasets to
test if on-policy performance is only linked to having adequately explored the environment. As a baseline, we compare the expert data to datasets collected by standard data aggregation procedures when learning. Finally, a dataset representing the entire state space is generated as a grid over the state-space or by sampling from reasonable values and measuring the transitions in order to test how complete dynamics models relates to model accuracy and final performance. Specifically, the cartpole datasets are as follows: an expert selection of episodes data with r > 179 (2, 400 points), a standard on-policy dataset aggregated from the end of a PETS run (3, 780 points), and a grid of datapoints to approximate the full system dynamics (16, 807 points). For half cheetah we compared three similar datasets: an expert dataset from a PETS run of half cheetah using the true dynamics for planning (3, 000 points), an on-policy dataset aggregated from PETS (90, 900 points), and an uniform sampled dataset attempting to approximate the full system dynamics (200, 000 points).

heading: 4.1 EXPLORATION OF MODEL LOSS VS EPISODE REWARD SPACE

text:
In the standard MBRL framework, it is assumed that there is a clear proportional relationship between model testing loss and on-policy performance. We here show that this is a strong assumption even in simple domains, and on specific on-policy data. The relationships between model accuracy on data representing the full environment space and reward show no concrete trend in Fig. 3c,f. The simplicity of the cartpole environment results in quick learning and a concentration of networks around peak reward. The distribution of rewards versus log-likelihood (LL) is shown in Fig. 3a-c, where there is a trend of increased reward as LL decreases, but there is substantial variance and points of disagreement. This bi-model distribution on the half cheetah expert data set, shown in most clearly Fig. 3d, relates to a failure mode in early half cheetah trials where the agent enters a unrecoverable state. The contrast between Fig. 3e and Fig. 3d,f shows a substantial difference in the transitions represented within the datasets. The multiple modes by which half cheetah runs during different stages of learning warrants more investigation into how new data should be incorporated into a training dataset while maintaining the knowledge of older data.
These results show that objective mismatch is not preventing MBRL from functioning, which would appear as model loss being fully dissociated with performance and the plots shown lacking any coherent shape. Rather, the mismatch we show likely is a ceiling on the performance of current MBRL algorithms by reducing the correlation between model accuracy and evaluation reward. The distributions in this section of on-policy data in Fig. 3b,e show a noisy trend of higher reward with better model loss and not the guarantee of improvement that is expected when training a better model.

heading: 4.2 MODEL LOSS VS EPISODE REWARD DURING TRAINING

text:
This section explores the relationship between how model training impacts performance at the per-epoch level. These experiments also shed light onto the impact of the strong dataset assumptions outlined in Sec. 3. Figures 4 and 5 are generated by re-evaluating the controller leveraging a dynamics model at each training epoch. As a dynamics model is trained, there are two key inflection points - the first is the training epoch where episode reward is maximized, and the second is when error on the validation set is minimized. When training on data from the entire state-space, our experiments show that the reward is maximized at a drastically different time than when validation loss is minimized. These experiments are focused on showing the disconnect between three standard practices in MBRL a) the assumption that the dynamics data can express large portions of the state-space when collected on policy, b) the idea that simple neural networks can satisfactorily capture complex dynamics, c) and the practice that model training is a simple optimization problem disconnected from reward. For the cartpole environment with the grid dataset, Fig. 4 shows that the performance of the controller degrades for P , PE models and fails to adequately learn with the D, DE models. For cartpole, training on on-policy data replicates results of the trials (> 175 reward), shown in Fig. 5b while the expert dataset fails to achieve any learned performance (< 20 reward) because it lacks a robust enough dataset for stable prediction generation during stochastic action sampling. Fig. 5 highlights how the trained models are able to represent other datasets that they are trained on,
which points to the need for fine-tuning of that assumption on datasets made at the outset between the state space and that collected on-policy. There is no information indicating the on-policy dataset will lead to a complete dynamics understanding because on-policy and grid show little correlation in evaluation loss. When training the gird on cartpole, the fact that the on-policy data diverges before
the reward decreasing is encouraging as objective mismatch may be preventable in simple tasks, but the problem becomes increasingly complex as the dynamics data distribution becomes large and multi-modal, as in half cheetah. Similar experiments on half cheetah data failed to achieve a matching reward to the full trials because the training process in PETS involves incremental network with dataset aggregation. When training on a single set of dynamics data for many epochs the controller did not achieve comparable reward to networks incrementally trained on new data after each episode.

heading: 4.3 DECOUPLING MODEL LOSS FROM CONTROLLER PERFORMANCE

text:
When two dynamics models evaluated on a control dataset have similar validation loss, there can be a large variance in the reflected control policy. In this section, we explore how differences in dynamics models are qualitatively reflected in control policies to show the reader that a accurate dynamics model does not guarantee performance.
Adversarial attack on model performance We performed an adversarial attack (Goodfellow et al., 2015; Szegedy et al., 2013) on a neural network dynamics model so that it attains a good likelihood but poor reward to continue illustrating objective mismatch. We start with a dynamics model that achieves high likelihood and high reward and tweak the parameters so that it continues achieving high likelihood but has a low reward. We fine-tune the network’s last layer to minimize the episode reward, with a large penalty if the model validation likelihood drops below the original value. We use a zerothorder optimizer (CMA-ES) because the reward is generally non-differentiable. As a starting point for this experiment we sampled a P dynamics model from the last trial of a PETS run
on cartpole. This model achieves reward of 176 and has a NLL of −4.827 on it’s on-policy training dataset. Using CMA-ES, we reduced the on-policy reward of the model to 98 – a 45% decrease in performance compared to the original model – while slightly improving the NLL to −4.845; the CMA-ES convergence over population iterations is shown in Fig. 7 and the difference between the two models is visualized in Fig. 6. Fine tuning of all model parameters would be even more likely to find sub-optimal performing controllers with low model loss because the output layer consists of about 1% of the total model parameters. This experiment shows that the model parameters that achieve a low model loss inhabit a broader space than the subset that also achieves high reward.
Sampling models with similar NLLs, different rewards To better understand the objective mismatch, we also compared how a difference of model loss can impact a control policy. We sampled models with similar NLL’s and extremely different rewards from Fig. 3d-e and visualized the chosen optimal action sequences along an expert trajectory. The control policies and dynamics models appear to be converging to different regions of state spaces. In these visualizations, there is not a emphatic
reason why the models achieved different reward, so further study is needed to quantify the impact of model differences. The interpretability of the difference between models and controllers will be important to solving the objective-mismatch issue.

heading: 5 ADDRESSING OBJECTIVE MISMATCH DURING MODEL TRAINING

text:
Tweaking dynamics model training can partially mitigate the problem of objective mismatch. While keeping the NLL minimization standard in MBRL, the trainer can prioritize state transitions associated with an expert trajectory over tuples further from expert, and dynamics model will use its capacity to learn relevant tasks more quickly. We show improved sample efficiency on Cartpole via re-weighting network training by a measure of Euclidean distance in the state-action space.
Given a element of a state space (si, ai), we can quantify the distance of any two tuples {(si, ai, s′i), i = 1, 2} as d = ||[s1, a1] − [s2, a2]||. With this distance, we re-weight the loss, l(y), of points further from an expert policy to be lower, so that points in the expert trajectory get a weight ω(y) = 1, and points at the edge of the grid dataset used in Sec. 4 get a weight ω(y) = 0 (in our experiments, multiple monotonic functions from 0 to 1 had similar results). With this notion of distance and weighting, we used the expert dataset discussed in Sec. 4 as a distance baseline. We generated a base dataset consisting of 25, 000, 000 tuples of (s, a, s′) by uniformly sampling across the state and action space of cartpole. We taxonomized this dataset by taking the minimum orthogonal distance, d∗, from each of the points to the 200 element dataset from one expert trajectory that achieved a reward of 180. To create different datasets that range from near-expert to nearly uniform across the state space, we vary the distance bound, , and number of points, S, trained on. For each sampling bound, , we sample 5 different datasets such that d∗i < from our distance-tabulated random dataset, and for each dataset (S, ) we trained 5 P models, giving 25 episodes from PETS algorithm on the cartpole task for each point in the heatmap shown in Fig. 8. This simple form of re-weighting the neural network loss, shown in Eq. (2a,b,c), demonstrated an improvement in sample efficiency to learn the cartpole task, as seen in the left halves of each plot in Fig. 8. Developing an iterative method to re-weight samples in an online training method has the potential to further improve the sample efficiency of MBRL baselines.
Weighting ω(y) = ed
∗(y) e Standard l(ŷ, y) Re-weight l(ŷ, y) · ω(y) (2a,b,c)

heading: 6 DISCUSSION, RELATED WORK, AND FUTURE WORK

text:
Objective mismatch seriously impacts the performance of MBRL. Our experiments have gone deeper into this fragility in the context of the state-of-the-art MBRL algorithms from an analysis perspective rather than from a solution perspective. Beyond the re-weighting of the NLL presented in Sec. 5, here we summarize and discuss the key pieces in the community starting to address this issue.
Learning the dynamics model to optimize the task performance Most relevant to our work are research directions on controllers that directly connect the reward signal back to terms of the controller. In theory, this exactly solves the model mismatch problem we discuss in this paper but in practice the current approaches are borderline intractable and have proven difficult to scale to complex systems. One popular way to do this is by designing systems that are fully differentiable so that the task reward can be backpropagated all the way to the dynamics. This has been investigated with differentiable MPC (Amos et al., 2018), Path Integral control (Okada et al., 2017; Pereira et al., 2018), and stochastic optimization (Donti et al., 2017). Universal Planning Networks (Srinivas et al., 2018) propose a differentiable planner that unrolls gradient descent steps over the action space of a planning network. An alternative explored in Bansal et al. (2017) is to use a zero-order optimizer instead (Bayesian optimization specifically) which can be used to learn locally linear dynamics to maximize the controller’s performance without having to compute gradients explicitly. Shaping the cost or reward Closely related are methods that, instead of learning a dynamics model optimizing task performance, fix the dynamics model and fine-tune the cost function of a controller to optimize, e.g., imitation loss Trimpe et al. (2014); Talvitie (2018); Tamar et al. (2017). Add heuristics to the dynamics model structure or training process to make control easier If it is infeasible or intractable to shape the cost or dynamics components of a controller, adding heuristics to the structure or training process of the dynamics model is reasonable and can give state-of-the-art results in many settings. One challenge in these heuristics is that they may be unstable and difficult to fix or improve when they do not work in new environments. These heuristics can manifest in the form of learning a latent space that is locally linear, e.g., in Embed to Control and related methods (Watter et al., 2015; Banijamali et al., 2017), by enforcing that the model makes long-horizon predictions (Ke et al., 2019), ignoring parts of the state space that actions can not control (Ghosh et al., 2018), detecting and correcting when a predictive model steps off the manifold of reasonable states (Talvitie, 2017), adding reward signal prediction on top of the latent space Gelada et al. (2019), adding noise when training transitions Mankowitz et al. (2019). among other approaches Jonschkowski et al. (2018); Ke et al. (2019); Miladinović et al. (2019); Ichter and Pavone (2019); Singh et al. (2019). Using incorrect models Instead of fixing the issues directly in the model or cost parameters, one can assume that the approximations are always incorrect but to use them anyway and potentially try to correct them Abbeel et al. (2006); Sorg et al. (2010); Joseph et al. (2013); Nagabandi et al. (2018). Add inductive biases to the controller Prior knowledge and inductive biases can be added to the controller in the form of hyper-parameters such as the horizon length, or by penalizing unreasonable control sequences by using, e.g., a slew rate penalty. These heuristics can significantly improve the performance if done correctly but can be difficult to tune. Jiang et al. (2015) use complexity theory to justify using a short planning horizon with an approximate model to reduce the the class of induced policies. Whitney and Fergus (2018) empirically show that long horizons are better for some tasks. Continuing Experiments Our experiments represent an initial exploration into the challenges of objective mismatch in MBRL, but more analysis is needed. Our exploration in Sec. 4.2 is limited to cartpole due to computational challenges of training with large dynamics datasets and Sec. 4.3 could be strengthened by defining quantitative comparisons in controller performance other than episode reward. Additionally, the effects of model mismatch should be quantified in other state of the art algorithms in MBRL such as MBPO (Janner et al., 2019) and POPLIN (Wang and Ba, 2019).

heading: 7 CONCLUSION

text: This paper identifies, formalizes and analyzes the issue of objective mismatch in MBRL. This fundamental disconnect in objectives between the training of the model w.r.t. the likelihood, and the overall task reward emerges from the subtle differences at the origins of MBRL with assumptions that have not been fully adapted. Experimental results highlighted the negative effects that objective mismatch have on the performance of a current state of the art MBRL algorithm. Our results are an exposition of current limitations and an encouragement to investigate nuance in current practices. In providing a first insight on the issue of objective mismatch in MBRL, we hope future work will more deeply examine these underlying assumptions. With our re-weighting method to mitigate mismatch, which also shows improvements in sample efficiency, we share our more thoughts about promising directions of future research to address this issue. We believe that fundamentally understanding and addressing the objective mismatch issue will contribute to improving the performance of MBRL.

heading: A EFFECT OF DATASET DISTRIBUTION WHEN LEARNING

text:
Learning speed can be slowed by many factors in dataset distribution, such as adding additional irrelevant transitions. When extra transitions from a specific area of the state space are included in the training set, the dynamics model will spend increased expression on these transitions. NLL of the model will be biased down as it learns this data, but it will reduce the learning speed as new, more relevant transitions are added to the training set.
Running cartpole random data collection with a short horizon of 10 steps (while forcing initial babbling state to always be 0), for 20, 200,400 and 2000 babbling rollouts (that sums up to 200, 2000, 4000 and 20000 transitions in the dataset finally shows some regression in the learning speed for runs with more useless data in the motor babbling. This data highlights the
importance of careful exploration vs exploitation tradeoffs, or changing how models are trained to be selective with data.

heading: B TASK GENERALIZATION IN SIMPLE ENVIRONMENTS

text:
In this section, we compare the performance of a model trained on data for the standard cartpole task (x position goal at 0) to policies attempting to move the cart to different positions in the x-axis. Fig. 10 is a learning curve of PETS with a PE model using the CEM optimizer. Even though performance levels out, the NLL continues to decrease as the dynamics models accrue more data. With more complicated systems, such as halfcheetah, the reward of different tasks verses global likelihood of the model would likely be more interesting (especially with incremental model training) - we will investigate this in future work. Below, we show that the dynamics model generalizes well to tasks close to zero (both positive in (Fig. 11b) and negative positions (Fig. 11a), but performance
drops off in areas the training set does not cover as well.
Below the learning curves in Fig. 12, we include snapshots of the distributions of training data used for these models at different trials, showing how coverage relates to reward in cartpole. It is worth investigating how many points can be removed from the training set while maintaining peak performance on each task.

heading: C WAYS MODEL MISMATCH CAN HARM THE PERFORMANCE OF A CONTROLLER

text:
Model mismatch between fitting the likelihood and optimizing the task’s reward manifests itself in many ways. Here we highlight two of them and in Sec. 6 we discuss how related work connects in with these issues.
Long-horizon rollouts of the model may be unstable and inaccurate. Time-series or dynamics models that are unrolled for long periods of time easily diverge from the true prediction and can easily step into predicting future states that are not on the manifold of reasonable trajectories. Taking these faulty dynamics models and using them as a smaller part of a controller that optimizes some cost function under a poor approximation to the dynamics. Issues can especially manifest if, e.g., the approximate dynamics do not properly capture stationarity properties necessary for the optimality of the true physical system being modeled.
Non-convex and non-smooth models may make the control optimization problem challenging The approximate dynamics might have bad properties that make the control optimization problem much more difficult than on the true system, even when the true optimal action sequence is optimal under the approximate model. This is especially true when using neural network as they introduce non-linearities and non-smoothness that make many classical control approaches difficult.


summaries: This paper presents an adversarial attack based on the feature representations at different layers given the classes. Instead of only looking at the final layers, class samples in intermediate feature space information is used to attack and increase transferability at the same time. Then the noise is optimized to perturb the input so that the a specific wrong output will be more likely to chosen. The paper is clear and well-written and different interesting experiments support the claims.

summaries: The paper proposes a new adversarial attack for the targeted blackbox model Unlike previous approaches which use the output layer possibly with some additional terms and regularization, the proposed approaches only rely on intermediate features. In fact, the adversarial example is based on a single intermediate layer. The adversarial examples are built by training, for each target class, a binary classifier for the class based only on the features of that layer.

abstractText: Almost all current adversarial attacks of CNN classifiers rely on information derived from the output layer of the network. This work presents a new adversarial attack based on the modeling and exploitation of class-wise and layer-wise deep feature distributions. We achieve state-of-the-art targeted blackbox transfer-based attack results for undefended ImageNet models. Further, we place a priority on explainability and interpretability of the attacking process. Our methodology affords an analysis of how adversarial attacks change the intermediate feature distributions of CNNs, as well as a measure of layer-wise and class-wise feature distributional separability/entanglement. We also conceptualize a transition from task/data-specific to model-specific features within a CNN architecture that directly impacts the transferability of adversarial examples.

authors:
Nicholas Carlini
David A. Wagner

title: Towards evaluating the robustness of neural networks

venue: In IEEE Symposium on Security and Privacy,

authors:
Shuyu Cheng
Yinpeng Dong
Tianyu Pang
Hang Su
Jun Zhu

title: Improving black-box adversarial attacks with a transfer-based

venue: prior. CoRR,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Fei-Fei Li

title: Imagenet: A large-scale hierarchical image database

venue: In CVPR,

authors:
Yinpeng Dong
Fangzhou Liao
Tianyu Pang
Hang Su
Jun Zhu
Xiaolin Hu
Jianguo Li

title: Boosting adversarial attacks with momentum

authors:
Nicholas Frosst
Nicolas Papernot
Geoffrey E. Hinton

title: Analyzing and improving representations with the soft nearest neighbor loss

venue: In ICML,

authors:
Ian J. Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: In ICLR,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Gao Huang
Zhuang Liu
Laurens van der Maaten
Kilian Q. Weinberger

title: Densely connected convolutional networks

authors:
Nathan Inkawhich
Wei Wen
Hai Li
Yiran Chen

title: Feature space perturbations yield more transferable adversarial examples

venue: In CVPR. IEEE Computer Society,

authors:
Alexey Kurakin
Ian J. Goodfellow
Samy Bengio

title: Adversarial machine learning at scale

venue: In ICLR. OpenReview.net,

authors:
Alexey Kurakin
Ian J. Goodfellow
Samy Bengio
Yinpeng Dong
Fangzhou Liao
Ming Liang
Tianyu Pang
Jianfeng Zhu
Xiaolin C. Hu
Cihang Xie
Jianyu Wang
Zhishuai Zhang
Zhou Ren
Alan Loddon Yuille
Sangxia Huang
Yao Zhao
Yuzhe Zhao
Zhonglin Han
Junjiajia Long
Yerkebulan Berdibekov
Takuya Akiba
Seiya Tokui
Motoki Abe

title: Adversarial attacks and defences

venue: competition. arXiv,

authors:
Yixuan Li
Jason Yosinski
Jeff Clune
Hod Lipson
John E. Hopcroft

title: Convergent learning: Do different neural networks learn the same representations

authors:
Ji Lin
Chuang Gan
Song Han

title: Defensive quantization: When efficiency meets robustness

authors:
Yanpei Liu
Xinyun Chen
Chang Liu
Dawn Song

title: Delving into transferable adversarial examples and black-box attacks

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Seyed-Mohsen Moosavi-Dezfooli
Alhussein Fawzi
Pascal Frossard

title: Deepfool: A simple and accurate method to fool deep neural networks

authors:
Nina Narodytska
Shiva Prasad Kasiviswanathan

title: Simple black-box adversarial attacks on deep neural networks

venue: In CVPR Workshops,

title: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples

authors:
Nicolas Papernot
Patrick D. McDaniel
Ian J. Goodfellow
Somesh Jha
Z. Berkay Celik
Ananthram Swami

title: Practical black-box attacks against machine learning

venue: In AsiaCCS,

authors:
Sara Sabour
Yanshuai Cao
Fartash Faghri
David J. Fleet

title: Adversarial manipulation of deep representations

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

authors:
Daniel Smilkov
Nikhil Thorat
Been Kim
Fernanda B. Viégas
Martin Wattenberg

title: Smoothgrad: removing noise by adding

venue: noise. arXiv,

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian J. Goodfellow
Rob Fergus

authors:
Florian Tramèr
Nicolas Papernot
Ian J. Goodfellow
Dan Boneh
Patrick D. McDaniel

title: The space of transferable adversarial

venue: examples. arXiv,

authors:
Florian Tramèr
Alexey Kurakin
Nicolas Papernot
Ian J. Goodfellow
Dan Boneh
Patrick D. McDaniel

title: Ensemble adversarial training: Attacks and defenses

authors:
Cihang Xie
Yuxin Wu
Laurens van der Maaten
Alan L. Yuille
Kaiming He

title: Feature denoising for improving adversarial robustness

authors:
Jason Yosinski
Jeff Clune
Yoshua Bengio
Hod Lipson

title: How transferable are features in deep neural networks

venue: In NIPS,

authors:
Matthew D. Zeiler
Rob Fergus

title: Visualizing and understanding convolutional networks

venue: Lecture Notes in Computer Science,

authors:
Wen Zhou
Xin Hou
Yongjun Chen
Mengyun Tang
Xiangqi Huang
Xiang Gan
Yong Yang

title: Transferable adversarial perturbations

text:
1 INTRODUCTION
Most recent adversarial attack literature has focused on empirical demonstrations of how classifiers can be fooled by the addition of quasi-imperceptible noise to the input (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini & Wagner, 2017; Moosavi-Dezfooli et al., 2016; Madry et al., 2018; Kurakin et al., 2017). However, adversarial attacks may be leveraged in other constructive ways to provide insights into how deep learning models learn data representations and make decisions. In this work, we propose a new blackbox transfer-based adversarial attack that outperforms state-of-the-art methods for undefended ImageNet classifiers. Importantly, this work provides a broad exploration into how different Deep Neural Network (DNN) models build feature representa-
tions and conceptualize classes. The new attack methodology, which we call the Feature Distribution Attack (FDA), leverages class-wise and layer-wise deep feature distributions of a substitute DNN to generate adversarial examples that are highly transferable to a blackbox target DNN.
One perspective on adversarial attacks is that adversarial noise is a direction in which to “move” the natural data. In standard attacks which directly use the classification output, the noise points in the direction of the nearest decision boundary at the classification layer (Tramèr et al., 2017). In this work, our crafted noise points in a direction that makes the data “look like” a sample of another class in intermediate feature space. Intuitively, if we can alter the representation in a layer whose features are representative of the data for the given task, but not specific to the model, the adversarial example may transfer better (to unobserved architectures) than attacks derived from logit-layer information.
Figure 1(top) illustrates the feature distribution modeling of a DNN, which is the core mechanism of the attack. f is a pre-trained substitute whitebox model to which we have full access. The true target blackbox model is not shown, but we only assume limited query access and that it has been
trained on ImageNet-1k (Deng et al., 2009). Adversarial examples are then generated on the whitebox model and transferred to the blackbox model. The novelty of the attack comes from the explicit use of class-wise and layer-wise feature distributions. In Figure 1(top), an auxiliary Neural Network (NN) gl,c learns p(y = c|fl(x)), which is the probability that the layer l features of the whitebox model, extracted from input image x, belong to class c. The attack uses these learned distributions to generate targeted (or untargeted) adversarial examples by maximizing (or minimizing) the probability that the adversarial example is from a particular class’s feature distribution (Figure 1(bottom)). We also use these learned distributions to analyze layer-wise and model-wise transfer properties, and to monitor how perturbations of the input change feature space representations. Thus, we gain insights on how feature distributions evolve with layer depth and architecture.

heading: 2 RELATED WORK

text:
In blackbox attacks (Narodytska & Kasiviswanathan, 2017; Su et al., 2017; Papernot et al., 2017; Tramèr et al., 2017; Inkawhich et al., 2019; Dong et al., 2018; Zhou et al., 2018), knowledge of the target model is limited. In this work, the target model is blackbox in the sense that we do not have access to its gradients and make no assumptions about its architecture (Madry et al., 2018; Cheng et al., 2019). A popular blackbox technique is transfer-based attacks, in which adversarial examples are constructed on the attackers’ own whitebox model and transferred to the target model. Papernot et al. (2016; 2017) develop special methods for training the attackers’ whitebox model to approximate the target model’s decision boundaries. In this work, we only use models that have been trained under standard configurations for ImageNet-1k (Deng et al., 2009). Tramèr et al. (2018) and Liu et al. (2017) bolster transferability by generating adversarial examples from an ensemble of whitebox models, which helps the noise not overfit a single model architecture. Our methods also discourage overfitting of the generating architecture, but we instead leverage feature space perturbations at the appropriate layer. In the 2017 NeurIPS blackbox attack competition (Kurakin et al., 2018), the winning method (Dong et al., 2018) used momentum in the optimization step, which helped to speed up the convergence rate and de-noise the gradient directions as to not be overly specific to the generating architecture. We also use this approach. Finally, Tramèr et al. (2017) analyze why transferability occurs and find that well-trained models have similar decision boundary structures. We also analyze transferability, but in the context of how adversarial examples change a model’s internal representations, rather than only making observations at the output layer.
While all of the above methods generate adversarial examples using information from the classification layer of the model, there have been a few recent works delving into the feature space of DNNs for both attacks and defenses. Sabour et al. (2016) show that in whitebox settings, samples can be moved very close together while maintaining their original image-domain representations. Zhou et al. (2018) regularize standard untargeted attack objectives to maximize perturbations of (all) intermediate feature maps and increase transferability. However, their primary objective is untargeted and still based on classification output information. Also, the authors do not consider which layers are affected and how the regularization alters the intermediate representations. Inkawhich et al. (2019) show that driving a source sample’s feature representation towards a target sample’s representation at particular layers in deep feature space is an effective method of targeted transfer attack. However, the method is targeted only and relies on the selection of a single (carefully selected) sample of the target class. Also, the attack success rate on ImageNet was empirically low. This work describes a more robust attack, with significantly better performance on ImageNet, and provides a more detailed analysis of layer-wise transfer properties. For adversarial defenses, Xie et al. (2019), Frosst et al. (2019), and Lin et al. (2019) consider the effects of adversarial perturbations in feature space but do not perform a layer-wise analysis of how the internal representations are affected.

heading: 3 ATTACK METHODOLOGY

text:
We assume to have a set of training data and a pre-trained model f from the same task as the target blackbox model (i.e. the ImageNet-1k training set and a pre-trained ImageNet model). To model the feature distributions for f , we identify a set of classes C = {c1, ..., cK} and a set of layers L = {l1, ..., lN} that we are keen to probe. For each layer in L, we train a small, binary, one-versusall classifier g for each of the classes in C, as shown in Figure 1 (top). Each binary classifier is given a unique set of parameters, and referred to as an auxiliary model gl,c. The output of an auxiliary model represents the probability that the input feature map is from a specific class c ∈ C. Thus, we
say that gl,c(fl(x)) outputs p(y = c|fl(x)), where fl(x) is the layer l feature map of the pre-trained model f given input image x.
Once trained, we may leverage the learned feature distributions to create both targeted and untargeted adversarial examples. Here, we focus mostly on targeted attacks, which are considered a harder problem (especially in the blackbox transfer case when we do not have access to the target model’s gradients) (Kurakin et al., 2018; Sharma et al., 2018). Discussion of untargeted attacks is left to Appendix C. Recall, the goal of a targeted attack is to generate an adversarial noise δ that when added to a clean sample x of class ysrc, the classification result of x+ δ is a chosen class ytgt. The key intuition for our targeted methods is that if a sample has features consistent with the feature distribution of class c at some layer of intermediate feature space, then it will likely be classified as class c. Although not shown in the objective functions for simplicity, for all attacks the adversarial noise δ is constrained by an `p norm (i.e. ||δ||p ≤ ), and the choice of layer l and target class label ytgt are chosen prior to optimization.
FDA We propose three targeted attack variants. The most straightforward variant, called FDA, finds a perturbation δ of the “clean” input image x that maximizes the probability that the layer l features are from the target class ytgt distribution:
max δ p(y = ytgt|fl(x+ δ)). (1)
We stress that unlike standard attacks that use output layer information to directly cross decision boundaries of the whitebox, our FDA objective leverages intermediate feature distributions which do not implicitly describe these exact boundaries.
FDA+ms In addition to maximizing the probability that the layer l features are from the target class distribution, the FDA+ms variant also considers minimizing the probability that the layer l features are from the source class ysrc distribution (ms = minimize source):
max δ λp(y = ytgt|fl(x+ δ))− (1− λ)p(y = ysrc|fl(x+ δ)). (2)
Here, λ ∈ (0, 1) weights the contribution of both terms and is a fixed positive value.
FDA+fd Similarly, the FDA+fd variant maximizes the probability that the layer l features are from the target class distribution while also maximizing the distance of the perturbed features from the original features (fd = feature-disruption):
max δ p(y = ytgt|fl(x+ δ)) + η ‖fl(x+ δ)− fl(x)‖2 ‖fl(x)‖2 . (3)
In other words, the feature-disruption term, with a fixed η ∈ R+, prioritizes making the layer l features of the perturbed sample maximally different from the original sample.
The additional terms in FDA+ms and FDA+fd encourage the adversarial sample to move far away from the starting point, which may intuitively help in generating (targeted) adversarial examples. Also, notice that FDA+ms requires the modeling of both the source and target class distributions, whereas the others only require the modeling of the target class distribution.
Optimization Procedure. The trained auxiliary models afford a way to construct a fully differentiable path for gradient-based optimization of the objective functions. Specifically, to compute FDA adversarial noise from layer l, we first build a composite model using the truncated whitebox model fl and the corresponding layer’s auxiliary model gl,c=ytgt for the target class ytgt, as shown in Figure 1(bottom). The loss is calculated as the Binary Cross Entropy (BCELoss) between the predicted p(y = ytgt|fl(x)) and 1. Thus, we perturb the input image in the direction that will minimize the loss, in turn maximizing p(y = ytgt|fl(x)). For optimization, we employ iterative gradient descent with momentum, as the inclusion of a momentum term in adversarial attacks has proven effective (Inkawhich et al., 2019; Dong et al., 2018). See Appendix D for more details.

heading: 4 EXPERIMENTAL SETUP

text:
ImageNet models. For evaluation we use popular CNN architectures designed for the ImageNet-1k (Deng et al., 2009) classification task: VGG-19 with batch-normalization (VGG19) (Simonyan & Zisserman, 2015), DenseNet-121 (DN121) (Huang et al., 2017), and ResNet-50 (RN50) (He et al.,
2016). All models are pre-trained and found in the PyTorch Model Zoo. Note, our methods are in no way specific to these particular models/architectures. We also emphasize transfers across different architectures rather than showing results between models from the same family.
Layer decoding scheme. Given a pre-trained model, we must choose a set of layers L to probe. For each model we subsample the layers such that we probe across the depth. For notation we use relative layer numbers, so layer 0 of DN121 (DN121l=0) is near the input layer and DN121l=12 is closer to the classification layer. For all models, the deepest layer probed is the logit layer. Appendix A decodes the notation for each model.
Auxiliary model training. We must also choose a set of classes C that we are interested in modeling. Recall, the number of auxiliary models required for a given base model is the number of layers probed multiplied by the number of classes we are interested in modeling. Attempting to model the feature distributions for all 1000 ImageNet classes for each layer is expensive, so we instead choose to run the majority of tests with a set of 10 randomly chosen classes (which are meant to be representative of the entire dataset): 24:“grey-owl”, 99:“goose”, 245:“bulldog”, 344:“hippo”, 471:“cannon”, 555:“fire-truck”, 661:“Model-T”, 701:“parachute”, 802:“snowmobile”, 919:“streetsign”. Thus, for each layer of each model we train 10 auxiliary classifiers, one for each class. After identifying high performing attack settings, we then produce results for all 1000 classes.
The architecture of all auxiliary models is the same, regardless of model, layer, or class. Each is a 2-hidden layer NN with a single output unit. There are 200 neurons in each hidden layer and the number of input units matches the size of the input feature map. To train the auxiliary models, unbiased batches from the whole ImageNet-1k training set are pushed through the truncated pretrained model (fl), and the extracted features are used to train the auxiliary model parameters.
Experimental procedure. Since we have three pre-trained models, there are 6 blackbox transfer scenarios to evaluate (no self-transfers). We use the ImageNet-1K validation set as the test dataset. Because FDA+ms requires both the source and target class distributions, for the targeted attack evaluations we only use source samples from the 10 trained classes, and for each sample, target each of the other 9 classes. For baseline attacks, we use targeted random-start Projected Gradient Descent (tpgd) (Madry et al., 2018; Kurakin et al., 2018), targeted NeurIPS2017 competition winning momentum iterative method (tmim) (Dong et al., 2018), and the Activation Attack (AA) (Inkawhich et al., 2019). Further, all targeted adversarial examples are constrained by `∞ = 16/255 as described in (Dong et al., 2018; Kurakin et al., 2018). As experimentally found, λ = 0.8 in (2) and η = 1e-6 in (3). Finally, as measured over the initially correctly classified subset of the test dataset (by both the whitebox and blackbox models), attack success is captured in two metrics. Error is the percentage of examples that the blackbox misclassifies and Targeted Success Rate (tSuc) is the percentage of examples that the blackbox misclassifies as the target label.

heading: 5 EMPIRICAL RESULTS

heading: 5.1 10-CLASS IMAGENET RESULTS

text:
The primary axis of interest is how attack success rate varies with the layer depth from which the feature distributions are attacked. Figure 2 shows the transfer results between all pairs of whitebox and blackbox models. Each plot shows a metric of attack success versus relative layer depth of the generated attack. The notation DN121 → VGG19 indicates adversarial examples were generated with a DN121 whitebox model and transferred to a VGG19 blackbox model.
Similar to Inkawhich et al. (2019), transferability trends for FDAs from a given whitebox model appear blackbox model agnostic (e.g. the shape of the curves from DN121→ RN50 are the same as DN121→ VGG19). This is a positive property, as once the optimal transfer layer for a whitebox model is found, evidence shows it will be the same for any blackbox model architecture. Further, the most powerful transfers come from perturbations of intermediate features, rather than perturbations of classification layer information. Another global trend is that in tSuc, FDA+fd performs best, FDA performs worst, FDA+ms is in-between, and all FDAs significantly outperform the other baselines. In the error metric, FDA+fd is best in early layers, FDA+ms is best in later layers, and FDA routinely under-performs the AA baseline. Although all attacks are targeted, it is relevant to report error as it is still an indication of attack strength. Also, it is clearly beneficial for the targeted attack objective to include a term that encourages the adversarial example to move far away from its starting place (FDA+ms & FDA+fd) in addition to moving toward the target region.
We now compare performance across whitebox models. For a DN121 whitebox, FDA+fd from DN121l=7 is the optimal targeted attack with an average tSuc of 34%. For DN121→ RN50, this attack outperforms the best baseline by 14% and 32% in error and tSuc, respectively. For a VGG19 whitebox, FDA+fd from V GG19l=5 is the optimal targeted attack with an average tSuc of 2.3%. For VGG19→ RN50, this attack outperforms the best baseline by 15% and 2% in error and tSuc, respectively. For a RN50 whitebox, FDA+fd from RN50l=8 is the optimal targeted attack with an average tSuc of 18%. For RN50→ DN121, this attack outperforms the best baseline by 27% and 17% in error and tSuc, respectively.

heading: 5.2 1000-CLASS IMAGENET RESULTS

text: Recall, due to the computational complexity of training one auxiliary model per class per layer per model, we ran the previous experiments using 10 randomly sampled ImageNet-1k classes. In reality, this may be a realistic attack scenario because an adversary would likely only be interested in attacking certain source-target pairs. However, to show that the 10 chosen classes are not special, and the previously identified optimal transfer layers are still valid, we train all 1000 class auxiliary models for DN121l=7 and RN50l=8. We exclude VGG19 because of its inferior performance in previous tests. Table 1 shows results for the four transfer scenarios. Attack success rates are all averaged over four random 10k splits and the standard deviation of all measurements is less than 1%. In these tests, for each source sample, a random target class is chosen. As expected, the 1000-class results closely match the previously reported 10-class results.

heading: 6 ANALYSIS OF TRANSFER PROPERTIES

text:
We now investigate why a given layer and/or whitebox model is better for creating transferable adversarial examples. We also explore the hypothesis that the layer-wise transfer properties of a
DNN implicate the transition of intermediate features from task/data-specific to model-specific. Intuitively, early layers of DNNs trained for classification may be working to optimally construct a task/data-specific feature set (Zeiler & Fergus, 2014; Yosinski et al., 2014). However, once the necessary feature hierarchy is built to model the data, further layers may perform extra processing to best suit the classification functionality of the model. This additional processing may be what makes the features model-specific. We posit that the peak of the tSuc curve for a given transfer directly encodes the inflection point from task/data-specific to model-specific features in the whitebox model. Instinctively, to achieve targeted attack success, the layer at which the attacks are generated must have captured the concepts of the classes for the general task of classification, without being overly specific to the architecture. Thus, layers prior to the inflection point may not have solidified the class concepts, whereas layers after the inflection point may have established the class concepts and are further processing them for the model output. This may also be considered an extension of the convergent learning theory of Li et al. (2016) and general-to-specific theory of Yosinski et al. (2014).

heading: 6.1 INTERMEDIATE DISRUPTION

text:
One way to measure why and how adversarial attacks work is to observe how the intermediate representations change as a result of perturbations to the input. Our trained auxiliary models afford a novel way to monitor the effects of such perturbations in deep feature space. To measure how much a layer’s features have changed as a result of a (targeted) adversarial perturbation, we define layerwise disruption as the difference between the target class probability before and after perturbation, as measured in layer l of model f : disruption = p(y = ytgt|fl(x+ δ))− p(y = ytgt|fl(x)). Figure 3 shows the average disruption caused in each transfer scenario, using both logit-based (tmim) and feature-based (FDA+fd) adversarial attacks. Each row plots the disruption versus layer depth from a single whitebox model to each blackbox model (e.g. the top row results from DN121 → VGG19 and DN121 → RN50 transfers). Each line represents the average disruption caused by some adversarial attack, where all FDAs are FDA+fd. The first column of plots shows the impact of each attack on the whitebox model’s feature distributions while the second and third columns shows impacts on the blackbox models’ feature distributions.
It appears FDAs generated from early layers (e.g. DN121l=0, V GG19l=0, RN50l=0) disrupt features the most in early layers and less so in deeper layers. Therefore, a sample resembling class ytgt in an early layer does not mean it will ultimately be classified as ytgt. However, recall from Figure 2 that attacks from early layers create very powerful untargeted adversarial examples (error). This indicates that early layer perturbations are amplified as they proceed through the model (Lin
et al., 2019), just not in a class-specific manner. Next, as expected, attacks that use information from the last layers of a whitebox model (e.g. tmim, DN121l=13, V GG19l=9, RN50l=12) create the largest disruption in the last layers of the whitebox, but not necessarily at the last layers of the blackbox models. However, the optimal transfer attacks (DN121l=7, V GG19l=5, RN50l=8) have high disruption all throughout the models, not just at the last layer. This is further evidence that perturbations of classification-layer features are overly model-specific and perturbations of optimal transfer-layer features are more specific to the data/task. Finally, notice that the maximum disruption caused in any blackbox model layer from VGG19 whitebox transfers is around 40% (row 2). For the DN121 and RN50 whitebox models, the maximum disruption is around 80%. This may explain VGG19 being an inferior whitebox model to transfer from, as the perturbation of intermediate VGG features does not in-turn cause significant disruption of blackbox model features.

heading: 6.2 AUXILIARY MODEL CORRELATION WITH FULL MODEL

text:
Another point of analysis is to investigate the correlation/discrepancy between the auxiliary models at a given layer and the output of the whitebox model. This may also indicate a transition from task/data-specific to model-specific features. We discuss discrepancy as an indication of how different the auxiliary model outputs are from the whitebox model outputs. Then correlation is the inverse of discrepancy so that when the auxiliary model outputs align well with the whitebox model outputs, the discrepancy is low and correlation is high. To evaluate discrepancy at a given layer l for input x, we aggregate the logit values (i.e. pre-sigmoid/softmax) for each class in C, as measured by the auxiliary models gl,c and the whitebox model f , into separate vectors. Then, a softmax (smax) operation is performed on each vector to establish two proper probability distributions over the classes in C. Discrepancy is then defined as the Kullback-Leibler divergence (DKL) between the two distributions, or discrepancy = DKL ( smax( [ gl,c(fl(x)) ]
∀c∈C)
∥∥ smax([f(x)[c]]∀c∈C)). Here,
f(x)[c] is the class c logit value from the whitebox model f given input x. Figure 4 shows the layer-wise auxiliary model correlations with the whitebox model outputs as measured from the average discrepancy over 500 input samples of classes in C. Note, the shapes of the curves are more informative than the actual values. Also, VGG19 layers have been shifted in notation by +4 so that layer depth 13 is the logit layer of each model. As expected, the auxiliary models in early layers have little correlation with the model output, while auxiliary models in later layers have high correlation with the model output. Importantly, the optimal transfer layers (?) mark a transition in the trendlines after which correlation increases sharply. This effect may directly explain why layers after the optimal layer are suboptimal, because the auxiliary models become highly correlated with the model output and begin to overfit the architecture. Since the auxiliary models are not highly-correlated with the output layer at the optimal-transfer-layers, we may surmise that the captured features are still mostly task/data specific.

heading: 6.3 AUXILIARY MODEL SALIENCY

text:
For a more qualitative analysis, we may inspect the auxiliary model saliency maps. Given an image of class ysrc, we visualize in Figure 5 what is salient to the ysrc auxiliary models at several DN121 layer depths using SmoothGrad (Smilkov et al., 2017) (see Appendix E for additional saliency examples for RN50). Notice, an observable transition occurs at the high-performing transfer layers from Figure 2 (DN121l=5,7). The salient regions move from large areas around the whole image (e.g. DN121l=0,3) to specific regions that are also salient in the classification layer (DN121l=13). The saliency and correlation transitions together show that the well-transferring layers have learned similar salient features as the classification layer while not being overly correlated with the model output. Therefore, perturbations focused on these salient regions significantly impact the final classification without being too specific to the generating architecture.
6.4 CLASS DISTRIBUTION SEPARABILITY
Finally, our trained auxiliary models afford new ways of measuring class-wise feature entanglement/separability for the purpose of explaining transfer performance. We adopt the definition of entanglement from Frosst et al. (2019) which states that highly entangled features have a “lack of separation of class manifolds in representation space,” and define separability as the inverse of entanglement. One way to measure the separability between class distributions in a layer using the auxiliary models is to gauge how far a sample has to “move” to enter a region of high class confidence. We define intra-class distance as the distance a sample has to move to enter a highconfidence region of its source class’s distribution. Similarly, we define inter-class distance as the distance a sample has to move to enter a high-confidence region of a target class distribution, where ysrc 6= ytgt. Then, separability in a layer is
the difference between average inter-class and intra-class distances. In practice, for a given sample, given model, chosen target class, and chosen layer, we iteratively perturb the sample using FDA with a small step size (e.g. 5e-5) until the confidence of the target distribution auxiliary network is over a threshold (e.g. 99.9%). The number of perturbation steps it takes to reach the confidence threshold encodes the distance of the sample to the targeted class’s distribution.
Results for the three whitebox models are shown in Figure 6, where the vertical axis is the separability in units of perturbation steps, and the horizontal axis is the layer-depth of each model. We see that there is some separability in all layers, for all models, indicating that even features very close to the input layer are somewhat class-specific. Further, VGG19’s features are much less separable than DN121 and RN50, indicating why VGG19 may have performed much worse as a whitebox model in the transferability tests. In the same vein, DN121 has generally the most separated features which further indicates why it may be a superior whitebox model. Intuitively, if a model/layer has highly class-separable feature distributions, FDA attacks may be more transferable because there is less ambiguity between the target class’s distribution and other class distributions during generation.

heading: 7 CONCLUSIONS

text: We present a new targeted blackbox transfer-based adversarial attack methodology that achieves state-of-the-art success rates for ImageNet classifiers. The presented attacks leverage learned classwise and layer-wise intermediate feature distributions of modern DNNs. Critically, the depth at which features are perturbed has a large impact on the transferability of those perturbations, which may be linked to the transition from task/data-specific to model-specific features in an architecture. We further leverage the learned feature distributions to measure the entanglement/separability of class manifolds in the representation space and the correlations of the intermediate feature distributions with the model output. Interestingly, we find the optimal attack transfer layers have feature distributions that are class-specific and highly-separable, but are not overly-correlated with the whitebox model output. We also find that highly transferable attacks induce large disruptions in the intermediate feature space of the blackbox models.

heading: ACKNOWLEDGMENTS

text: The research was supported in part by AFRL (FA8750-18-2-0057), DARPA, DOE, NIH, NSF and ONR.

heading: A. LAYER DECODING

text:
Table 2 is the layer number look-up-table that corresponds to the layer notation used in the paper. DenseNet-121 (DN121), VGG19bn (VGG), and ResNet-50 (RN50) appear because they are the model architectures used for the main results. The DN121 notation follows the implementation here: https://github.com/pytorch/vision/blob/master/torchvision/ models/densenet.py. In english, layer 0 shows that the output of the truncated model comes from the 2nd denseblock of the 2nd denselayer. Layer 11 means the output of the truncated model comes from the 14th denseblock in the 4th denselayer. Layer 13 indicates the output comes from the final FC layer of the model.
The VGG model does not have denseblocks or dense layers so we use another notation. In the implementation at https://github.com/pytorch/vision/blob/master/ torchvision/models/vgg.py, the VGG19bn model is constructed from the layer array: [64, 64,′M ′, 128, 128,′M ′, 256, 256, 256, 256,′M ′, 512, 512, 512, 512,′M ′, 512, 512, 512, 512, ′M ′, FC1, FC2, FC3], and we follow this convention in the table. In the array, each number corresponds to a convolutional layer with that number of filters, the M’s represent max-pooling layers, and the FCs represent the linear layers at the end of the model. Notice, in these tests we do not consider the first 11 layers of VGG19 as they were shown to have very little impact on classification when perturbed.
The RN50 notation follows the implementation here: https://github.com/pytorch/ vision/blob/master/torchvision/models/resnet.py. As designed, the model has 4 layer groups with [3,4,6,3] Bottlenecks in each, respectively. Thus, layer 0 means the output of the truncated model comes from the 1st Bottleneck of layer group 2. Layer 12 means the output comes from the 3rd Bottleneck of layer group 4, and layer 13 means the output comes from the final FC layer (i.e. output layer) of the model.

heading: B. FULL TARGETED TRANSFER RESULTS

text: Figure 7 shows the full targeted attack transfer results from which the Figure 2 were extracted. These full results include two additional metrics of attack success. Untargeted Transfer Rate (uTR) is the rate at which examples that fool the whitebox also fool the blackbox (encodes likelihood of misclassification). Targeted Transfer Rate (tTR) is the rate at which successful targeted examples on the whitebox are also successful targeted examples on the blackbox (encodes likelihood of targeted misclassification). Error and tSuc are described in Section 4.

heading: C. UNTARGETED FEATURE DISTRIBUTION ATTACKS

text:
The goal of an untargeted attack is to generate an adversarial noise δ that when added to a clean sample x of class ysrc, the classification result of x + δ is not ysrc. The key intuition for feature distribution-based untargeted attacks is that if a sample’s features are made to be outside of the feature distribution of class ysrc at some layer of intermediate feature space, then it will likely not be classified as ysrc.
uFDA The first untargeted attack variant is uFDA which is described as
min δ p(y = ysrc|fl(x+ δ)).
uFDA minimizes the probability that the layer l features of the perturbed sample x + δ are from the source class ysrc distribution. Unlike the targeted samples which drive towards high confidence regions of a target class feature distribution, this objective drives the sample towards low confidence regions of the source class feature distribution.
uFDA+fd The second untargeted variant uFDA+fd is described as
min δ pl(y = ysrc|fl(x+ δ))− η ‖fl(x+ δ)− fl(x)‖2 ‖fl(x)‖2 .
uFDA+fd also carries a feature disruption term so that the objective drives the perturbed sample towards low confidence regions of the source class feature distribution and maximal distance from the original sample’s feature representation.
fd-only The final untargeted attack fd-only is described as
max δ ‖fl(x+ δ)− fl(x)‖2 ‖fl(x)‖2 .
Notice, fd-only is simply the feature disruption term and is a reasonable standalone untargeted attack objective because making features maximally different may intuitively cause misclassification.
To test attack success, we generate untargeted adversarial examples from both DN121 and RN50 whiteboxes and test transfers to a VGG19 blackbox model. It is common to evaluate untargeted attacks with a tighter noise constraint (Kurakin et al., 2018; Dong et al., 2018) as the task is simpler, so in these tests we use `∞ = 4/255 and = 8/255 (rather than = 16/255 used for targeted tests). For baselines, we use the Madry et al. (2018) random start PGD attack (upgd) and the Dong et al. (2018) competition winning momentum iterative attack (umim). Similar to the layer-wise targeted evaluations, each ”clean” source sample belongs to the same set of 10 previously modeled classes. Figure 8 shows the error rates versus layer depth for the attacks.
As expected, the error rate increases with epsilon and the layer depth at which feature-based attacks are generated from has a large impact on attack success rate. In general, uFDA+fd is the top performer, followed by fd-only, then uFDA. However, uFDA often under-performs the umim baseline, further indicating that for adversarial attacks in feature space it is beneficial to include a term that prioritizes feature disruption (e.g. uFDA+fd & fd-only).
On average across models, at = 4/255, the optimal layer uFDA+fd has an untargeted error rate of 37%, which is 9% higher than the best baseline. At = 8/255, the optimal layer uFDA+fd has an untargeted error rate of 79%, which is 27% higher than the best baseline. Also, both whitebox models perform similarly in terms of attack success rate, however the performance of fd-only varies
between the two (especially at = 8/255). Surprisingly, fd-only which simply disrupts the original feature map is the optimal attack for the DN121 whitebox (by a small margin). Finally, note that the optimal transfer layers from the targeted attacks (i.e. DN121l=7 and RN50l=8) are also high performing layers for the untargeted attacks.

heading: D. ADVERSARIAL EXAMPLE GENERATION PROCESS

text:
Recall, because the auxiliary models are NNs, the optimization objectives described for both the targeted and untargeted attacks can be solved with an iterative gradient descent procedure. For any version of the FDA attacks, we first build a ”composite” model which includes the truncated whitebox model fl and the appropriate auxiliary model gl,c, as shown in Figure 1(bottom). An attack loss function LFDA is then defined which includes a BCELoss term and any additional term which is trivially incorporated (e.g. the feature disruption term). We then iteratively perturb the source image for K iterations using the sign of a collected momentum term. Similar to Dong et al. (2018) and Inkawhich et al. (2019), momentum is calculated as
mk+1 = mk + ∇IkLFDA(Ik; θ) ||∇IkLFDA(Ik; θ)||1 ,
where m0 = 0 and Ik is the perturbed source image at iteration k. The perturbation method for this `∞ constrained attack is then
Ik+1 = Clip(Ik − α ∗ sign(mk+1), 0, 1).
In this work, all attacks perturb for K = 10 iterations and α = /K.

heading: E. ADDITIONAL SALIENCY

title: TRANSFERABLE PERTURBATIONS OF DEEP FEATURE DISTRIBUTIONS