summaries: This paper focuses on model based reinforcement learning (RL). Specifically, the authors consider the setting of combining model based and model free RL algorithms by using the learned dynamics model to generate new data for training the model free algorithm. In order to capture the uncertainty of the environment and the model, the author applied Baysian neural network to learn the dynamics of the environment. The authors approximated the true Bayesian inference process by keeping an anchored ensemble of neural networks, where the prior and posterior of network weights are approximately Gaussian. The ensemble of dynamics model is then used to generate data to train a PPO[1] based agent. In order to prevent the agent from exploiting the learned dynamics model, the authors propose a heuristic way of balancing the amount of real data and model generated data by comparing the rewards.

summaries: This paper presents a new model-free + model-based algorithm, MBPGE, that trains a policy using a policy gradient algorithm on top of the learned models. Contrary to previous approaches, they use a true Bayesian distribution by means of the randomized anchorized MAP. Furthermore, they combine rollouts from the real environment and from the learned dynamics directly for the policy training, instead of relying just on the ones of learned dynamics, which induces a larger distributional shift.

abstractText: We propose a new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory rollouts performed on a learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use a special type of uncertainty quantification by a stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model is addressed by next state predictions based on an uncertainty aware ensemble of dynamics models. The influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and a model-based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.

authors:
Christopher G. Atkeson
Juan Carlos Santamaria

title: A comparison of direct and model-based reinforcement learning

venue: In IN INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION,

title: Mcmc-based image reconstruction with uncertainty quantification

venue: SIAM Journal on Scientific Computing,

authors:
Roberto Calandra
Jan Peters
Carl Edward Rasmussen
Marc Peter Deisenroth

title: Manifold gaussian processes for regression

authors:
Ignasi Clavera
Jonas Rothfuss
John Schulman
Yasuhiro Fujita
Tamim Asfour
Pieter Abbeel

title: Model-based reinforcement learning via meta-policy optimization

authors:
Marc Deisenroth
Carl Rasmussen

title: Pilco: A model-based and data-efficient approach to policy search

venue: In ICML, pp. 465–472,

authors:
Bradley Efron
Robert Tibshirani

title: An introduction to the bootstrap, volume 57 of Monographs on statistics and applied probability

venue: [nachdr.] edition,

authors:
Philipp Ennen
Pia Bresenitz
René Vossen
Frank Hees

title: Learning robust manipulation skills with guided policy search via generative motor reflexes

authors:
Vladimir Feinberg
Alvin Wan
Ion Stoica
Michael I. Jordan
Joseph E. Gonzalez
Sergey Levine

title: Model-based value estimation for efficient model-free reinforcement learning

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017

venue: URL http://arxiv.org/abs/1703

authors:
Justin Fu
Sergey Levine
Pieter Abbeel

title: One-shot learning of manipulation skills with online dynamics adaptation and neural network

venue: priors. CoRR,

title: Uncertainty in deep learning

authors:
Yarin Gal
Zoubin Ghahramani

title: Dropout as a bayesian approximation: Representing model uncertainty in deep learning

authors:
Xavier Glorot
Yoshua Bengio

title: Understanding the difficulty of training deep feedforward neural networks

venue: In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,

title: Practical variational inference for neural networks

venue: In Proceedings of the 24th International Conference on Neural Information Processing Systems,

authors:
Shixiang Gu
Timothy P. Lillicrap
Zoubin Ghahramani
Richard E. Turner
Sergey Levine

title: Qprop: Sample-efficient policy gradient with an off-policy critic

title: Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen

venue: Journal für die reine und angewandte Mathematik (Crelles Journal),

authors:
Geoffrey E. Hinton
Radford M. Neal

title: Bayesian learning for neural networks

title: A natural policy gradient

authors:
Gabriel Kalweit
Joschka Boedecker

title: Uncertainty-driven imagination for continuous deep reinforcement learning

venue: Proceedings of the 1st Annual Conference on Robot Learning,

authors:
Vijay R. Konda
John N. Tsitsiklis

title: Actor-critic algorithms

venue: Advances in Neural Information Processing Systems

authors:
S. Kullback
R.A. Leibler

title: On information and sufficiency

venue: The Annals of Mathematical Statistics,

authors:
Thanard Kurutach
Ignasi Clavera
Yan Duan
Aviv Tamar
Pieter Abbeel

title: Model-ensemble trust-region policy optimization

authors:
Sergey Levine
Pieter Abbeel

title: Learning neural network policies with guided policy search under unknown dynamics

authors:
Sergey Levine
Vladlen Koltun

title: Guided policy search

venue: Proceedings of the 30th International Conference on Machine Learning,

title: Variational policy search via trajectory optimization

title: Learning complex neural network policies with trajectory optimization

authors:
Timothy Lillicrap
Jonathan Hunt
Alexander Pritzel
Nicolas Heess
Tom Erez
Yuval Tassa
David Silver
Daan Wierstra

title: Continuous control with deep reinforcement learning

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis

title: Human-level control through deep reinforcement learning

venue: doi: 10.1038/nature14236. URL https://doi.org/10

authors:
Volodymyr Mnih
Adrià Puigdomènech Badia
Mehdi Mirza
Alex Graves
Timothy P. Lillicrap
Tim Harley
David Silver
Koray Kavukcuoglu

title: Asynchronous methods for deep reinforcement learning

authors:
Tim Pearce
Mohamed Zaki
Alexandra Brintrup
Nicolas Anastassacos
Andy Neely

title: Uncertainty in neural networks: Bayesian ensembling, 2018

authors:
J. Peters
S. Schaal

title: Policy gradient methods for robotics

venue: IEEE/RSJ International Conference on Intelligent Robots and Systems,

title: Early Stopping

venue: But When?,

authors:
Carl Edward Rasmussen
Christopher K.I. Williams

title: Gaussian processes for machine learning. Adaptive computation and machine learning

title: Exploiting model uncertainty estimates for safe dynamic control learning

venue: In Proceedings of the 9th International Conference on Neural Information Processing Systems,

authors:
John Schulman
Sergey Levine
Philipp Moritz
Michael I. Jordan
Pieter Abbeel

title: Trust region policy optimization. CoRR, abs/1502.05477, 2015a

venue: URL http://arxiv.org/abs/1502

authors:
John Schulman
Philipp Moritz
Sergey Levine
Michael Jordan
Pieter Abbeel

title: Highdimensional continuous control using generalized advantage estimation

authors:
John Schulman
Filip Wolski
Prafulla Dhariwal
Alec Radford
Oleg Klimov

title: Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017

authors:
David Silver
Guy Lever
Nicolas Heess
Thomas Degris
Daan Wierstra
Martin Riedmiller

title: Deterministic policy gradient algorithms

venue: In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32,

authors:
Richard Sutton
David Mcallester
Satinder Singh
Yishay Mansour

title: Policy gradient methods for reinforcement learning with function approximation

venue: Adv. Neural Inf. Process. Syst,

authors:
Richard S. Sutton
Andrew G. Barto

title: Reinforcement learning: An introduction. A Bradford book

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

authors:
Hado van Hasselt
Arthur Guez
David Silver

title: Deep reinforcement learning with double qlearning

authors:
Tingwu Wang
Xuchan Bao
Ignasi Clavera
Jerrick Hoang
Yeming Wen
Eric Langlois
Shunshi Zhang
Guodong Zhang
Pieter Abbeel
Jimmy Ba

title: Benchmarking model-based reinforcement learning

heading: 1 INTRODUCTION

text:
Reinforcement Learning (RL) can be broadly classified in two categories: model-free RL, which uses experience from interacting with the environment to learn a value function, and model-based RL, which uses experience to approximate the environment by a model (Deisenroth & Rasmussen (2011)). Recently, impressive results were achieved applying model-free RL aproaches to challenging tasks. A RL agent with a neural network (NN) based Q-value representation was introduced in Mnih et al. (2015) which was able to reach an equivalent or higher performance on many Atari games compared to humans by learning a state function based on raw pixels. Further modifications addressed the problem of Q-value overestimation (van Hasselt et al. (2015)), dropping significant experience (Schaul et al. (2015)) and extended the approach to asynchronous methods (Mnih et al. (2016)) to improve the overall performance. Beside learning a value function, it is possible to use the gradient of the RL objective to update the policy (Peters & Schaal (2006), Sutton & Barto (2010), Silver et al. (2014), Lillicrap et al. (2015)). Actor-critic methods aim at combining the strong points of policy gradients and an approximated value function, in which the critic learns an approximated value function that is used to update the policy of the actor in the direction of its gradient (Konda & Tsitsiklis (2000), Schulman et al. (2015a), Mnih et al. (2016), Gu et al. (2016)).
In contrast, model-based RL can be significantly more data efficient. It is notable that a dynamics model is not necessarily globally valid. In guided policy search (GPS), time-varying linear dynamics models are learned from multiple rollouts with diverse start and goal conditions (Levine & Koltun (2013a), Levine & Koltun (2013b), Levine & Koltun (2014)). These trajectory samples are then used to fit a time-varying Gaussian dynamics model leading to a Gaussian distribution over trajectories, even for unknown dynamics (Levine & Abbeel (2014)). The neural network policy is trained in a supervised fashion such that local policies are improved minimizing a cost function bounded by the trust region of the KL-divergence, and the weights of the NN policy are adjusted to mimick the trajectory distribution. Further improvements extended GPS to a broader robust
state space by introducing a new policy representation which is called general motor reflex (GMR) (Ennen et al. (2018)). In a first step, the state space is encoded to a latent state representation by a variational autoencoder. Then, a translation model predicts motor reflex parameters in order to adjust a Gaussian controller resulting in robust trajectories even outside the distribution of training samples.
However, modelling errors can reduce the efficiency and benefit of model-based RL algorithms since model imperfections are exploited (Deisenroth & Rasmussen (2011), Schneider (1996), Atkeson & Santamaria (1997)) known as model-bias. With model-bias being a result of overfitting, one approach to solve this problem is to incorporate explicit uncertainty estimates by using a probabilistic dynamics model. Probabilistic inference for learning control (PILCO) (Deisenroth & Rasmussen (2011)) uses a Gaussian process (GP) to represent the dynamics of the environment. Such a model provides predictions with true Bayesian uncertainty estimates but suffer from the curse of dimensionality (Calandra et al. (2014)). Recently, Bayesian neural networks (BNN) received much attention. Approximate Bayesian techniques, e.g. variational inference (Graves (2011)) or Monte-Carlo dropout (Gal & Ghahramani (2015)), come at the drawback that correlations between parameters are not maintained, while scaling abilities of the Bayesian ’gold standard’ inference method Markov-Chain-Monte-Carlo (MCMC) are limited (Hinton & Neal (1995). Adding a regularization term to the loss function of a NN, e.g. Tikhonov regularization, permits maximum a posterior (MAP) estimates of the parameters since the solution of the regularized loss function is equivalent to the maximum of the posterior density function (Bardsley (2012)). However, the regularized solution leads to a point estimator of the true posterior density. This insufficiency can be addressed when sampling from the true posterior distribution using a MCMC method providing additional information about the shape of the posterior but is computationally more demanding.
Our approach is inspired by recent advances, Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) (Kurutach et al. (2018)), Model Based Meta Policy Optimization (MB-MPO) (Clavera et al. (2018)), Model-assisted Boostrapped DDPG (MA-BDDPG) (Kalweit & Boedecker (2017)), and Model-Based Value Expansion (MVE) (Feinberg et al. (2018)). In ME-TRPO, a set of samples from the real environment is used to train an ensemble of NNs. In a second iteration, fictitious trajectory rollouts are generated to update the policy using Trust Region Policy Optimization (TRPO) (Schulman et al. (2015a)), i.e., the policy update is only based on samples from the approximated dynamics model of the environment. In MB-MPO, the meta policy learning framework ’model agnostic meta learning’ (Finn et al. (2017)) is used to train a policy on an ensemble of learned dynamics models. The notion is that the meta-learning frame of the dynamics models is used to adapt quickly to any new dynamics model within one update step. The overall similarities of all dynamics models are incorporated in the meta-model that is then adjusted to create the fitted dynamics model. MA-BDDPG employs the same idea to augment the dataset to achieve a better data efficiency. In contrast to our approach, MA-BDDPG makes use of an actor-critic method in which the policy is represented deterministically. Updates in the actor and critic network follow the advances of Deep Q-Networks by refering to time-delayed target networks. A further difference is in the procedure for estimating uncertainty, i.e. Kalweit & Boedecker (2017) applies bootstrapping (Efron & Tibshirani (1998)) to get a distribution over Q-functions while we use an anchored ensemble of NNs. MVE improves value estimates assuming that an approximated dynamics model can be used in training a critic up to a depth H in which we are certain about the model accuracy. Contrary to our approach, MVE defines its dynamics model as a deterministic neural network.
Our new algorithm MBPGE is a model-free RL algorithm that is augmented by a true Bayesian ensemble of dynamics models to generate fictitious trajectory rollouts to decrease the sample complexity. In contrast to ME-TRPO, we take advantage of a very recent inference advance based on randomised anchored MAP sampling that allows for a true Bayesian uncertainty estimate of the true posterior (Pearce et al. (2018)). As a result, our stochastic dynamics model reduces the effect of policy overfitting as model inaccuracies will be addressed by highly random next state predictions. We have seen that ill-distributed datasets, i.e., data points of some regions in the state space are overrepresented while others are underrepresented, might lead to enormous fictitious total rewards that diverge heavily from those of the real environment samples due to overfitting dynamics models. We tackle this issue by introducing a ratio of fictitious and real rewards so that our algorithm
adjusts naturally to become more model-free if the dynamics model is inaccurate and acts in a more model-based way if the approximation of the environment is accurate. To summarize, our main contributions are:
1. A novel model-free RL algorithm that uses a true Bayesian representation for the stochastic dynamics model and
2. a routine that adjusts the composition of the dataset for a policy gradient estimate by weighting the trajectory samples based on the ratio of the total rewards of the real environment and the approximated model.
We derive our approach by reviewing the background in constrained policy gradient methods and randomized maximum a posterior sampling. Then, we present our augmented model-free reinforcement learning algorithm in detail. Finally, we evaluate our approach on challenging simulated tasks in a continuous control domain.

heading: 2 PRELIMINARIES

text:
We formulate the underlying problem of our tasks as a discrete-time finite-horizon Markov decision process (MDP), defined by the tuple (S,A, P, r, ρ0, γ, T ). Here, we assume that our agent acts in a stochastic environment with continuous state space S ⊆ Rn and continuous action space A ⊆ Rm. Additionally, we assume that an unknown dynamics model P (st+1|st,a) is defined for our environment and we are given a reward function r(st,at). Let ρ0 denote the initial state distribution, γ is the discount factor, T is the finite horizon and πθ(a|s) denotes our neural network policy, parameterized by θ, distributed over actions a, and conditioned to states s. The overall aim is to learn an optimal policy π such that the expected total reward J(π) = Eτ∼π [∑T t=0 rs(st, at)
] is maximized, where τ = s0, a0, ..., aT−1, sT denotes a trajectory with s0 ∼ ρ0, at ∼ πθ and st+1 = P (st, at).

heading: 2.1 POLICY GRADIENT METHODS

text:
The maximization problem of the RL objective can be solved by performing gradient ascent computing the derivative with respect to the policy parameters θ. The major problem of policy gradient methods is to find a good estimator for the gradient since the variance of the gradient estimator increases with the time horizon. Fortunately, a variance reduction scheme for policy gradients, generalized advantage estimator (GAE) (Schulman et al. (2015b)), reduces the variance significantly while only introducing a small bias. It is possible to trade-off between bias and variance by using a λ−return (Sutton & Barto (2010)). Extending the idea of an n-step return, one can weight each n-step update by λn−1 and normalize the sum by (1− λ) resulting in
Rλt = (1− λ) ∞∑ n=1 λn−1Rnt . (1)
The finite horizon version assumes that all rewards after time step T equal 0, leading to
Rλt = (1− λ) T−t−1∑ n=1 λn−1Rnt + λ T−t−1RT−tt . (2)
Setting λ = 0 results in the biased single step return R(1)t with low variance, while λ = 1 corresponds to the unbiased Monte-Carlo return R(∞)t with high variance. GAE can be derived when estimating the advantage Aπ(st, at) = Qπ(st, at)− V π(st) with the λ−return. The gradient of the objective now turns to
g := ∇θE [ T∑ t=0 rt ] = E [ T∑ t=0 Ât∇θlogπθ(at|st) ] . (3)
Usually, we intend to update our parameters only by small changes. However, instead of performing update steps in parameter space using the Euclidean distance, it is much more convenient to
measure closeness between the current policy and the updated policy in terms of distances between distributions. The intuition behind is that distances between distributions, e.g. Kullback-Leibler divergence (Kullback & Leibler (1951)) or Hellinger distance (Hellinger (1909)), are invariant to scaling or shifting. This type of approach is known as natural policy gradients (Kakade (2001), Sutton et al. (2000), Peters & Schaal (2006)). Trust region policy optimization (TRPO) utilizes this idea and maximizes its ’surrogate’ objective function by constraining the allowed policy update step (Schulman et al. (2015a))
max θ Êt [ πθ(at|st) πθold(at|st) Ât ] s.t. Êt [KL [πθold(∗|st), πθ(∗|st)]] 5 δ . (4)
TRPO builds on the concept of guaranteed monotonic improvement. The notion is that we can compute an upper bound for the error of diverging policies. Therefore, we can guarantee a policy improvement as long as we optimize the local approximation within a trusted region. The proof is given in Schulman et al. (2015a). PPO combines the appealing guarantee of improvement with a much simpler implementation (Schulman et al. (2017)). Instead of maximizing the ’surrogate’ function in eq. 4, it uses the following clipped objective
J(θ) = Êt [ min(rt(θ)Ât, clip(rt(θ), 1− , 1 + )Ât ] . (5)
Clipping the probability ensures that the probability ratio rt(θ) = πθ(at|st) πθold (at|st)
is in the interval [1 − , 1 + ], while taking the minimum of the clipped and unclipped objective leads to a lower bound of the objective with the same convenient effect as constraining the ’surrogate’ objective in TRPO.

heading: 2.2 MODEL-BASED REINFORCEMENT LEARNING WITH AN ANCHORED ENSEMBLE OF DEEP NEURAL NETWORKS

text:
Contrary to model-free RL, model-based RL utilizes the interactions with the environment to learn an approximated model. Model-based approaches benefit from a significantly lower sample complexity. However, learning an accurate model of the environment can be very challenging for certain domains. As a consequence, model-based algorithms often show a worse asymptotic behaviour. To date, there are several diverse approaches to represent the dynamics model and a proper choice is often crucial. Dynamics model in RL must reliably fit to datasets with a low and high number of data points: during the first few iterations only a limited number of data points is available, causing expressive general function approximators, e.g. deep neural networks, to overfit. With progressing iterations simple function approximators tend to underfit complex system dynamics.
Bayesian regression aims to find the parameters θ of a function y∗ = fθ(x∗), which are likely to have generated the output Y given input X . This can be achieved by updating a prior distribution over the parameters of the neural network θ0 ∼ p(θ) by using Bayes Theorem. If it is possible to compute the posterior distribution
p(θ|X,Y ) = p(Y |X, θ)p(θ) p(Y |X) , (6)
the posterior distribution of new data points y∗ can be inferred by marginalizing over θ p(y ∗ |x∗, X, Y ) = ∫ (Y |X, θ)p(θ)dθ . (7)
Previous work revealed that exploiting uncertainty estimation in RL can increase the sample efficiency dramatically (Deisenroth & Rasmussen (2011)). Gaussian Processes (Rasmussen & Williams (2008)) perform well in low data regimes but do not scale to complex tasks with high dimensionality (Calandra et al. (2014)). On the other hand, probabilistic NNs can approximate arbitrarily complex dynamics but come at the cost that a true Bayesian uncertainty estimate cannot be provided (Gal & Ghahramani (2015)). The computation of the integral in eq. 7 can in general only be approximated numerically under a trade-off between accuracy and performance.
A Baysian neural network (BNN) can be approximated by using a diverse ensemble F = {f1, ..., fK} of NNs. Ensembling provides an uncertainty estimate in such way that the variance
of the ensemble’s prediction is considered to be its uncertainty. Technically, this approach is not Bayesian (Gal (2016)). This drawback was addressed in Pearce et al. (2018) by regularizing the parameters about values drawn from a prior distribution which is called randomized anchored maximum a posterior sampling. This approach uses Bayes’ rule to compute a multivariate posterior by
N (µpost,Σpost) ∝ N (µprior,Σprior)N (µlike,Σlike) (8)
with N (µprior,Σprior) as multivariate prior and N (µlike,Σlike) as multivariate likelihood. When following the MAP approach µMAP = µpost, we can use a parameterized mean of the prior µprior,θ0 so that the true posterior distribution is matched with
µMAP (θ0) = (Σ −1 like + Σ −1 prior) −1(Σ−1likeµlike + Σ −1 priorµprior,θ0) . (9)
According to Pearce et al. (2018) suitable parameters for θ0 can be found by setting E [µMAP (θ0)] = µpost and V [µMAP (θ0)] = Σpost resulting in θ0 ∼ N (µ0,Σ0) with
µ0 = µprior (10)
Σ0 = Σprior + Σ 2 priorΣ −1 like ≈ Σprior . (11)
When using NNs in an anchored ensemble, the loss function needs to be modified. The typical loss function in a neural network is defined as
L = 1
N ||y − ŷ||22 +
1
N ||Γ1/2θ||22 (12)
where ŷ denotes the predictions of the neural net, y is the vector of the corresponding labels, θ is the flattened vector of NN parameters, and Γ is the diagonal square matrix of a L2 regularization. Minimizing the loss in eq. 12 results in parameters that are equal to MAP estimates with a zero mean normal prior. Eq. 12 can be modified in such way that the minimized parameters can be seen as MAP estimates with non-zero centered priors
Lanchored = 1
N ||y − ŷ||22 +
1
N ||Γ1/2(θ − θ0)||22 . (13)
The proof is provided in Pearce et al. (2018). As a result, randomised anchored MAP sampling with NN leads to a good approximation of the true posterior. The algorithm for training an anchored ensemble with deep neural networks is given in Algorithm 1. In the initialization, an ensemble of K neural networks with different parameters θj,0 is created. The parameters are drawn from a distribution over the network weights θj,0 ∼ N (µ0,Σ0) with µ0 according to eq. 10 and Σ0 as presented in eq. 11 (see Alg. 1, 3-8). During training, every individual neural network is trained with the training data X and the corresponding labels Y. The loss is computed according to eq. 13 (see Alg. 1, 9-11). Mean and variance of the true posterior distribution are estimated for the ensemble by
ŷ(x∗) = 1 K K∑ j=1 fj(x∗) (14)
σ̂2y(x∗) = Iσ̂2 + 1
K K∑ j=1 (fj(x∗)− ŷ)T (fj(x∗)− ŷ) , (15)
where fj denotes an individual NN within the set ofK NNs of the ensemble, σ̂2 is an estimate of the noise of the training data, ŷ(x∗) is the predicted mean of the ensemble and σ̂2y denotes the predicted variance of the ensemble at test point x∗. We make use of the uncertainty estimate in eq. 15 by sampling the next state prediction from the distribution predicted by the approximated BNN, i.e. st+1 ∼ N (ŷ(st, at), σ̂2y(st, at)). Consequently, the rollouts sampled from the learned dynamics model will diverge much in regions of high uncertainty which prevents the policy to overfit to optimistic model inaccuracies. The efficiency of this approach was empirically proven by Kurutach et al. (2018).
Algorithm 1 Randomized MAP sampling with anchored ensembles 1: input: Training data X & Y, vector test data point x∗, prior mean µprior and prior covariance
Σprior, ensemble size K, data noise variance estimate σ̂2 2: output: estimate of mean ŷ and variance σ̂2y
# Ensemble Initialization: 3: Γ = σ̂2 Σ
−1 prior
4: for j = 1 to K do 5: µ0 = µprior (eq. 10), Σ0 = Σprior (eq. 11) 6: sample θj,0 from N (µ0,Σ0) 7: create neural network fj with Γ,θj,0 8: end for
# Ensemble Training: 9: for j = 1 to K do
10: train fj with X, Y, loss in eq. 13 11: end for
# Ensemble Prediction: 12: ŷ(x∗) = 1K ∑K j=1 fj(x∗) (eq. 14)
13: σ̂2y(x∗) = Iσ̂2 + 1K ∑K j=1(fj(x∗)− ŷ)T (fj(x∗)− ŷ) (eq. 15)

heading: 3 AUGMENTED MODEL-FREE REINFORCEMENT LEARNING

text: In this section, we describe our gradient based model-free RL algorithm that uses BNNs with randomized MAP sampling as explained in the previous section. Our motivation is to address complex and high-dimensional real robotic tasks in continuous action and state spaces. This aim requires model-free RL algorithms to become more data efficient while maintaining the same asymptotic behaviour. Our approach, model-based policy gradient enrichment, achieves this goal by amending model-free RL with an ensemble of deep NN based dynamics models. Using randomized anchored MAP sampling our approach does not only compute a point estimate of the true posterior distribution but also accounts for the shape of the posterior distribution (Pearce et al. (2018)). This does not only provide a true Bayesian uncertainty estimate, but also prevents overfitting in the approximated stochastic dynamics model.

heading: 3.1 THE MODEL-FREE REINFORCEMENT LEARNING FRAME

text:
The first step of our proposed algorithm follows the commonly accepted procedure of model-free RL algorithms in which we iteratively perform trajectory rollouts to collect data and estimate the gradient (see Alg. 2, 6-7). Policy improvement is performed following the clipped ’surrogate’ objective shown in eq. 5 (see Alg. 2, 12). Additionally, we use an advantage estimator (Schulman et al. (2015b)) to reduce variance when estimating the policy gradient. However, we do not discard the collected samples of the real environment when performing a policy update. Instead, we add the samples to our dataset D in order to train our stochastic dynamics model. In MBPGE, the batch of trajectories UPPO needed for each unique policy update is sampled partially from the real environment (see Alg. 2, 7) and partially from the approximated dynamics model (see Alg. 2, 9). First, the trajectories τ will be collected from the real environment and added to UPPO and D. In a second step, the anchored ensemble of NNs is trained based on all trajectory samples of the real environment (see Alg. 2, 8). The complete algorithm is given in Algorithm 1. This ensemble is used to collect fictitious trajectories τ̃ which are added to UPPO. Afterwards, a standard policy update according to proximal policy optimization (PPO) (Schulman et al. (2017)) based on the batch UPPO will be performed. Depending on the composition of sampled trajectories in the sample batch UPPO, we can effect the influence of the stochastic dynamics model on the policy update. If the learned dynamics model is inaccurate, i.e. it is corrupted by a high model bias, we want our policy update to rely more on trajectories from the real environment. Contrary, we can use an accurate dynamics model to generate the major part of the update samples from the approximated environment to reduce the sample complexity of PPO. In order to control the amount of trajectories sampled from the real and approximated environment, we utilize a simple indicator for model inaccuracies. If the agent achieves on both the real- and approximated environment about the same total reward (i.e.
R(τ) = ∑T t=1 r(st, at)), it indicates that the dynamics model is accurate in the states which are likely to be explored by the policy. Hence, it is sufficient to sample the major part from the learned dynamics model. We use this notion to adjust the composition of trajectory samples drawn from the real and approximated environment by the following heuristic: The number ñk of trajectories from the learned dynamics model is set to
ñk = max
( min (∣∣∣∣ R̄(τ)R̄(τ̃) ∣∣∣∣ , ∣∣∣∣ R̄(τ̃)R̄(τ) ∣∣∣∣)αNr, 1) (16) R̄(τ) = 1
nk−1 nk−1∑ i=1 R(τi) R̄(τ̃) = 1 ñk−1 ñk−1∑ i=1 R(τ̃i)
with k being the index of the current PPO iteration. The hyperparameter α ∈ [0, 1] controls the maximum of trajectories which can be collected from the learned dynamics model. Consequently, the number of trajectories sampled from the real environment is set to
nk = max(Nr − ñk, nmin) (17)
where Nr encodes a fixed number of trajectories utilized for a PPO update and nmin is the minimal amount of real environment trajectories per PPO update. Algorithm 2 summarizes our approach.

heading: 4 RESULTS

text:
In this section, we present the evaluation results of our proposed augmented model-free RL algorithms. The experiments aim to compare the performance of our new method shown in Algorithm 2 to state-of-the-art model-free and model-based algorithms. We used the following algorithms:
• Trust Region Policy Optimization (Schulman et al. (2015a)): a model-free RL algorithm with constraint policy gradient update
• Proximal Policy Optimization (Schulman et al. (2017)): a model-free RL algorithm with clipped ’surrogate’ objective
• Model-Based Meta-Policy-Optimization (Clavera et al. (2018)): a model-based RL algorithm that uses an ensemble of dynamics model to quickly adapt to new dynamcis model with one policy gradient step
The evaluation process is performed on four simulated OpenAI Gym Benchmarking tasks (Brockman et al. (2016)), i.e., Hopper, Half-cheetah, Swimmer, and Walker, which are based on the the MuJoCo physics simulator (Todorov et al. (2012)). A detailed description of the environments is given in Appendix A.1 and the hyperparameter settings are listed in Appendix A.2. In
Algorithm 2 Model-based Policy Gradient Enrichment 1: input: Batch size Nr; nmin; α; PPO hyperparameters; dynamics model hyperparameters 2: initialize D as an empty dataset 3: n0 = Nr 4: ñ0 = 1 5: for k = 0 to Niter do 6: initialize UPPO as an empty dataset 7: collect nk trajectories with the current policy π on the real environment and add them to
UPPO and D 8: train anchored ensemble of NNs on D (Algorithm 1) 9: collect ñk trajectories with the current policy π on the approximated environment and add
them to UPPO 10: compute ñk+1 with equation 16 11: compute nk+1 with equation 17 12: run a PPO update step with the data in UPPO 13: end for
our experiments we would like to evaluate how our approach compares to state-of-the-art RL algorithms in terms of sample complexity and final performance. The results are performed for a fixed trajectory length of 200 time steps. In total, all algorithms perform 4e5 steps on the environment. Figure 1 depicts the comparison of our MBPGE algorithm to prior work on pure policy gradient based RL and MB-MPO that exceeds the performance of ME-TRPO as shown in Wang et al. (2019). Furthermore, Figure 1 illustrates the total number of samples (the samples from the real environment plus fictitious samples) our algorithm has processed for optimization.
The results clearly show that MBPGE can outperform the available alternatives. Incorporating fictitious samples from a learned stochastic dynamics can dramatically improve the raw performance of MF-RL. The necessary amount of training samples from the real environment is greatly reduced while the asymptotic performance of MF-RL algorithms is maintained. The positive effect of the stochastic dynamics model becomes apparent by the increased learning rate. As a matter of fact, the averaged return rises significantly faster for MBPGE than for the pure model-free algorithm. We believe that the increased performance is caused by two reasons. First, having no prior on the actual task, the agent needs to explore how to distinguish between high and low-rewarded actions. Since the dataset of the observed experience in an early training stage is simply not accurate enough, the Baysian uncertainty estimate of the randomized anchored MAP in the anchored ensemble will have high variance. Consequently, the expected reward under high variance of the fictitious trajectory samples is very low, even though the mean would be exactly the same as in a deterministic dynamics model. This permits a policy update in the direction of the highest gradient we are certain about while avoiding catastrophic failures due to erroneously optimistic predictions. Second, with progressing data samples the dataset of collected experiences becomes more expressive resulting in a potentially more accurate dynamics model. Figure 1(e) illustrates that the more certain the agent is about its dynamics model the more samples it generates in the fictitious environment. The less challenging the task, the more confident is the agent about its dynamics model and the higher the factor by which the total number of samples used for policy gradient estimates is upscaled (Hopper: 10.62, Swimmer: 4.62, Half-Cheetah: 2.44, Walker: 1.94). Even for the most challenging environment our approach can double the amount of data samples based on the learned dynamics model. As a result, the raw performance of MBPGE is significantly higher.
Designing a suitable heuristic for estimating the best amount of fictitious data samples is very challenging. We tested diverse heuristics with different trajectory lengths and found that the heuristic in eq. 16 performs best (cf. 2). Additionally, we find that the number of fictitious data samples used for a policy update step is, up to a certain degree, of minor importance for learning speed but effects the asymptotic behaviour. As a consequence, we introduced a hyperparameter α in eq. 16 to manually reduce the size of fictitious data samples. Improvement to this inconvenient solution is part of our future research.

heading: 5 CONCLUSIONS

text: In this paper, we introduce model-based policy gradient enrichment, an algorithm that incorporates a stochastic dynamics model to augment the dataset for estimating the current policy gradient. Our method takes advantage of randomized anchored MAP that results in an ensemble of neural networks providing a true Baysian uncertainty estimate. We presented that our approach leads to significantly faster learning while maintaining the final asymptotic performance of plain model-free RL. We further show that the stochastic dynamics model is a suitable approach to reduce model bias leading to an increased learning rate in an early training stage.

heading: ACKNOWLEDGMENTS

text: We thank Maren Bennewitz for helpful discussions and feeback, and Christian Lagemann for comments that greatly improved the manuscript.

heading: Appendices

heading: A EXPERIMENTAL DESCRIPTION

heading: A.1 ENVIRONMENTS

text: Our method requires the reward of a time step to be a known function of the form r(st, st−1, at) or r(st, at). Otherwise, it would be necessary to introduce another regression model for learning the reward function. Hence, we had to manipulate the default environments slightly as described in the following.

heading: Hopper:

text: The basis of our Hopper was the Hopper-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the x position in space (pseudo code: x = sim.data.qpos[0]). We utilized the default reward function r(st, st−1, at) = xt−xt−1 ∆t + 1−0.001‖at‖ 2 2 and early termination criterion.

heading: Walker:

text: The basis of our Walker was the Walker2d-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the x position in space (pseudo code: x = sim.data.qpos[0]). We utilized the default reward function r(st, st−1, at) = xt−xt−1 ∆t + 1−0.001‖at‖ 2 2 and early termination criterion.

heading: HalfCheetah:

text: The basis of our HalfCheetah was the HalfCheetah-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the x position in space (pseudo code: x = sim.data.qpos[0]). We utilized the default reward function r(st, st−1, at) = xt−xt−1 ∆t − 0.1‖at‖ 2 2 and no kind of early termination criterion.

heading: Swimmer:

text: The basis of our Swimmer was the Swimmer-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities and our only modification was that we extended the observation vector by the x position in space (pseudo code: x = sim.data.qpos[0]). We utilized the default reward function r(st, st−1, at) = xt−xt−1 ∆t − 0.0001‖at‖ 2 2 and no kind of early termination criterion.

heading: Ant:

text: The basis of our Ant was the Ant-v2 from OpenAI Gym. Its observation vector consists of all angular joint positions and velocities as well as some contact related states. Our modification was to remove the contact states from the observation vector and to add the x position in space (pseudo code: x = sim.data.qpos[0]) to it. We utilized the reward function r(st, st−1, at) = xt−xt−1 ∆t + 1 − 0.5‖at‖ 2 2. The only difference of it to the default reward function is that it has no contact penalty. We additionally utilized the default early termination criterion of Ant-v2.

heading: A.2 HYPERPARAMETERS

text:
If not specified differently, we used the hyperparameters as listed in Table 1 (MBPGE), Table 2 (PPO), Table 2 (TRPO) and Table 3 (MB-MPO). For our plain PPO and TRPO experiments we have not defined a fixed number of trajectories to collect for a policy update. Instead, we collected trajectories until the batch contained a certain number of time steps Tmin. The Anchored Ensemble requires an assumption about the prior distribution. We have set the mean µprior of the Gaussian prior distribution to zero and its variance Σprior according to the Xavier initialization (Glorot & Bengio (2010)) (ηin =̂ fan-in, ηout =̂ fan-out)
Σprior,i,j = 2
ηin,i,j + ηout,i,j
within this work. The index i denotes the NN layer and j the unit inside that layer.

heading: Dynamics Model:

text:
The NNs used for representing the dynamics model in MBPGE and in MB-MPO consisted of 4 hidden ReLU layers with 1000, 900, 800 and 700 units (2244000 parameters). (Fu et al., 2015, p. 5) described the positive effect of the second order Markov assumption (i.e. the next state depends on the current- and previous state and action) with regard to the ability of a NN in learning strong nonlinearities, e.g. contacts. Most challenging robotic manipulation or locomotion problems which are typically solved by model free RL include contacts, so in the experiments we used NNs of the form f(st, at, st−1, at−1). A problem with this NN/BNN configuration is the very first prediction of a roll out in which st−1 and at−1 are not defined. We handled this problem by using a second NN/BNN f0 which was trained only on the very first roll out transitions {s1, s0, a0} (i.e. a dynamic model only for predicting s1 = f0(s0, a0)). f0 consisted of 3 hidden ReLU layers with 50 units each (7268 parameters).
Furthermore, we trained our dynamics models to predict ∆st = st+1 − st instead of directly predicting st+1. Additionally, we used the following regularization techniques:
• Early stopping wit GL criterion as specified in (Prechelt, 2012, p. 58) • Normalization of the training dataset so that it has a zero mean and a unit variance • Shuffling of the training data • Weight decay with λ = 0.0001 (only for MB-MPO)

heading: Policy:

text: For each tested algorithm we utilized a stochastic policy of the form at ∼ N (µ(st), σ), in which µ(st) was the output of a NN (2 tanh layers with 128 units each) and σ was a vector containing the standard deviation of each action. The network’s parameters were initialized randomly according to the Xavier initialization (Glorot & Bengio (2010)) and the initial standard deviation vector was set to σi = 1 ∀ i ∈ {1, . . . , Da} with Da being the dimension of the action vector.


summaries: This paper proposed a new defense method for capsule networks. For both white-box and black-box settings, the proposed CapNets has shown superior performance than two variants of CNNs. The visualizations of adversarial examples generated by the CapNets are more aligned with the human perception which is very insightful. On the corrupted MNIST dataset, the results show the proposed defense method can also be used well as an out-of-distribution detector. Overall the paper is clearly written and easy to follow.

summaries: This paper studies the problem of detecting and generating adversarial images using class-conditional capsule networks. Specifically, this paper first introduced a novel method that detects adversarial examples by class-conditional image reconstruction. Motivated by this defense method, this paper further proposed a novel reconstructive attack that minimizes both classification and reconstruction loss. Experimental evaluations are conducted on MNIST, FashionMNIST, SVHN, and CIFAR-10 dataset. Results demonstrate the effectiveness of the proposed defense and the novel reconstructive attack method.

abstractText: Adversarial examples raise questions about whether neural network models are sensitive to the same visual features as humans. In this paper, we first detect adversarial examples or otherwise corrupted images based on a class-conditional reconstruction of the input. To specifically attack our detection mechanism, we propose the Reconstructive Attack which seeks both to cause a misclassification and a low reconstruction error. This reconstructive attack produces undetected adversarial examples but with much smaller success rate. Among all these attacks, we find that CapsNets always perform better than convolutional networks. Then, we diagnose the adversarial examples for CapsNets and find that the success of the reconstructive attack is highly related to the visual similarity between the source and target class. Additionally, the resulting perturbations can cause the input image to appear visually more like the target class and hence become non-adversarial. This suggests that CapsNets use features that are more aligned with human perception and have the potential to address the central issue raised by adversarial examples.

authors:
Anish Athalye
Nicholas Carlini
David A. Wagner

title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples

venue: In International Conference on Machine Learning,

authors:
Battista Biggio
Igino Corona
Davide Maiorca
Blaine Nelson
Nedim Šrndić
Pavel Laskov
Giorgio Giacinto
Fabio Roli

title: Evasion attacks against machine learning at test time

venue: In Joint European conference on machine learning and knowledge discovery in databases,

authors:
Nicholas Carlini
David Wagner

title: Adversarial examples are not easily detected: Bypassing ten detection methods

venue: In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security,

title: Towards evaluating the robustness of neural networks

venue: IEEE Symposium on Security and Privacy (SP),

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: ImageNet: A large-scale hierarchical image database

venue: In IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Reuben Feinman
Ryan R. Curtin
Saurabh Shintre
Andrew B. Gardner

title: Detecting adversarial samples from artifacts

venue: arXiv preprint arXiv:1703.00410,

authors:
Justin Gilmer
Ryan P. Adams
Ian Goodfellow
David Andersen
George E. Dahl

title: Motivating the rules of the game for adversarial example research

venue: arXiv preprint arXiv:1807.06732,

authors:
Justin Gilmer
Luke Metz
Fartash Faghri
Samuel S. Schoenholz
Maithra Raghu
Martin Wattenberg
Ian Goodfellow

title: Adversarial spheres

venue: In International Conference on Learning Representations,

authors:
Zhitao Gong
Wenlu Wang
Wei-Shinn Ku

title: Adversarial and clean data are not twins

venue: arXiv preprint arXiv:1704.04960,

authors:
Ian Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

authors:
Ian Goodfellow
Yao Qin
David Berthelot

title: Evaluation methodology for attacks against confidence thresholding models

authors:
Kathrin Grosse
Praveen Manoharan
Nicolas Papernot
Michael Backes
Patrick McDaniel

title: On the (statistical) detection of adversarial examples

venue: arXiv preprint arXiv:1702.06280,

authors:
Dan Hendrycks
Kevin Gimpel

title: Early methods for detecting adversarial images

authors:
Geoffrey E. Hinton
Sara Sabour
Nicholas Frosst

title: Matrix capsules with em routing

authors:
Hossein Hosseini
Sreeram Kannan
Radha Poovendran

title: Are odds really odd? bypassing statistical detection of adversarial examples

venue: arXiv preprint arXiv:1907.12138,

authors:
Saumya Jetley
Nicholas A. Lord
Philip H.S. Torr

title: With friends like these, who needs adversaries

venue: In Advances in Neural Information Processing Systems,

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

title: Learning multiple layers of features from tiny images

venue: Technical report, University of Toronto,

authors:
Alexey Kurakin
Ian Goodfellow
Samy Bengio

title: Adversarial examples in the physical world

authors:
Alexey Kurakin
Ian Goodfellow
Samy Bengio
Yinpeng Dong
Fangzhou Liao
Ming Liang
Tianyu Pang
Jun Zhu
Xiaolin Hu
Cihang Xie

title: Adversarial attacks and defences competition

venue: arXiv preprint arXiv:1804.00097,

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Xin Li
Fuxin Li

title: Adversarial examples detection in deep networks with convolutional filter statistics

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Jan Hendrik Metzen
Tim Genewein
Volker Fischer
Bastian Bischoff

title: On detecting adversarial perturbations

authors:
Felix Michels
Tobias Uelwer
Eric Upschulte
Stefan Harmeling

title: On the vulnerability of capsule networks to adversarial attacks

authors:
Norman Mu
Justin Gilmer

title: Mnist-c: A robustness benchmark for computer vision

venue: In ICML 2019 Workshop on Uncertainty and Robustness in Deep Learning,

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y. Ng

title: Reading digits in natural images with unsupervised feature learning

venue: In NIPS workshop on deep learning and unsupervised feature learning,

authors:
Jathushan Rajasegaran
Vinoj Jayasundara
Sandaru Jayasekara
Hirunima Jayasekara
Suranga Seneviratne
Ranga Rodrigo

title: Deepcaps: Going deeper with capsule networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Kevin Roth
Yannic Kilcher
Thomas Hofmann

title: The odds are odd: A statistical test for detecting adversarial examples

authors:
Sara Sabour
Nicholas Frosst
Geoffrey E. Hinton

title: Dynamic routing between capsules

authors:
Lukas Schott
Jonas Rauber
Wieland Brendel
Matthias Bethge

title: Robust perception through analysis by synthesis

venue: arXiv preprint arXiv:1805.09190,

authors:
Yang Song
Taesup Kim
Sebastian Nowozin
Stefano Ermon
Nate Kushman

title: Pixeldefend: Leveraging generative models to understand and defend against adversarial examples

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

authors:
Lucas Theis
Aäron van den Oord
Matthias Bethge

title: A note on the evaluation of generative models

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

venue: arXiv preprint arXiv:1708.07747,

authors:
Jianxiong Xiao
James Hays
Krista A Ehinger
Aude Oliva
Antonio Torralba

title: Sun database: Large-scale scene recognition from abbey to zoo

venue: In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,

authors:
Richard Zhang
Phillip Isola
Alexei A Efros
Eli Shechtman
Oliver Wang

title: The unreasonable effectiveness of deep features as a perceptual metric

heading: 1 INTRODUCTION

text:
Adversarial examples (Szegedy et al., 2013) are inputs that are designed by an adversary to cause a machine learning system to make a misclassification. A series of studies on adversarial attacks have shown that it is easy to cause misclassifications using visually imperceptible changes to an image under `p-norm based similarity metrics (Goodfellow et al., 2014; Kurakin et al., 2016; Madry et al., 2017; Carlini & Wagner, 2017b; Goodfellow et al., 2018). Since the discovery of adversarial examples, there has been a constant “arms race” between better attacks and better defenses. Many new defenses have been proposed (Song et al., 2017; Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017), only to be broken shortly thereafter (Carlini & Wagner, 2017a; Athalye et al., 2018). Hinton et al. (2018) showed that capsule models are more robust to simple adversarial attacks than CNNs but Michels et al. (2019) showed that this is not the case for all attacks.
The cycle of attacks and defenses motivates us to rethink both how we can improve the general robustness of neural networks as well as the high-level motivation for this pursuit. One potential path forward is to detect adversarial inputs, instead of attempting to accurately classify them (Schott et al., 2018; Roth et al., 2019). Recent work (Jetley et al., 2018; Gilmer et al., 2018b) argue that adversarial examples can exist within the data distribution, which implies that detecting adversarial examples based on an estimate of the data distribution alone might be insufficient. Instead, in this paper we develop methods for detecting adversarial examples by making use of class-conditional reconstruction networks. These sub-networks, first proposed by Sabour et al. (2017) as part of a Capsule Network (CapsNet), allow a model to produce a reconstruction of its input based on the identity and instantiation parameters of the winning capsule. Interestingly, we find that reconstructing
∗Equal Contributions.
an input from the capsule corresponding to the correct class results in a much lower reconstruction error than reconstructing the input from capsules corresponding to incorrect classes, as shown in Figure 1(a). Motivated by this, we propose using the reconstruction sub-network in a CapsNet as an attack-independent detection mechanism. Specifically, we reconstruct a given input from the pose parameters of the winning capsule and then detect adversarial examples by comparing the difference between the reconstruction distributions for natural and adversarial (or otherwise corrupted) images.
We extend this detection mechanism to standard convolutional neural networks (CNNs) and show its effectiveness against black box and white box attacks on three image datasets; MNIST, FashionMNIST and SVHN. We show that capsule models achieve the strongest attack detection rates and accuracy on these attacks. We then test our method against a stronger attack, the Reconstructive Attack, specifically designed to attack our detection mechanism by generating adversarial examples with a small reconstruction error. With this attack we are able to create undetected adversarial examples, but we show that this attack is less successful in fooling the classifier than a non-reconstructive attack.
Among all these attacks, we find CapsNets perform the best in detecting adversarial examples. To explain the success of CapsNets over CNNs, we further diagnose the adversarial examples for CapsNets and find that 1) the success of the targeted reconstructive attack is highly dependent on the visual similarity between the source image and the target class. 2) many of the resultant attacks resemble members of the target class and so cease to be “adversarial” – i.e., they may also be misclassified by humans. These findings suggest that CapsNets with class conditional reconstructions have the potential to address the real issue with adversarial examples – networks should make predictions based on the same properties of the image that people use rather than using features that can be manipulated by an imperceptible adversarial attack.
In summary, our main contributions are:
• We propose a class-conditional capsule reconstruction based detection method to detect standard white-box/black-box adversarial examples on three datasets. This detection mechanism is attack-agnostic and is successfully extended to standard convolutional neural networks. • We test our detection mechanism on the corrupted MNIST dataset and show that it can work
as a general out-of-distribution detector. • A stronger reconstructive attack is specifically designed to attack our detection mechanism
but becomes less successful in fooling the classifier. • We perform extensive qualitative studies to explain the superior performance of CapsNets
in detecting adversarial examples compared to CNNs. The results suggest that the features captured by CapsNets are more aligned with human perception.

heading: 2 RELATED WORK

text:
Adversarial examples were first introduced in (Biggio et al., 2013; Szegedy et al., 2013), where a given image was modified by following the gradient of a classifier’s output with respect to the
image’s pixels. Goodfellow et al. (2014) then developed the more efficient Fast Gradient Sign method (FGSM), which can change the label of the input image X with a similarly imperceptible perturbation that is constructed by taking an step in the direction of the gradient. Later, the Basic Iterative Method (BIM) (Kurakin et al., 2016) and Projected Gradient Descent (Madry et al., 2017) can generate stronger attacks improved on FGSM by taking multiple steps in the direction of the gradient. In addition, Carlini & Wagner (2017b) proposed another iterative optimization-based method to construct strong adversarial examples with small perturbations.
An early approach to reducing vulnerability to adversarial examples was proposed by (Goodfellow et al., 2014), where a network was trained on both clean images and adversarially perturbed ones. Since then, there has been a constant “arms race” between better attacks and better defenses; Kurakin et al. (2018) provide an overview of this field. However, many defenses against adversarial examples have been demonstrated to be an effect of “obfuscated gradients” and can be further circumvented under the white-box setting (Athalye et al., 2018).
Another line of work attempts to circumvent adversarial examples by detecting them with a separatelytrained classifier (Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017) or using statistical properties (Hendrycks & Gimpel, 2016; Li & Li, 2017; Feinman et al., 2017; Grosse et al., 2017). However, many of these approaches were subsequently shown to be flawed (Carlini & Wagner, 2017a; Athalye et al., 2018). The most recent work in detecting adversarial examples (Roth et al., 2019) that has a 99% true positive rate on CIFAR-10 dataset (Krizhevsky, 2009) has also been fully bypassed by later work (Hosseini et al., 2019) which decreased the true positive rate to less than 2%.
Similar to our work, Schott et al. (2018) also investigated the effectiveness of a class-conditional generative model as a defense mechanism for MNIST digits. However, we differ in some important ways. Their model is in some ways the opposite of ours - they first attempt to generate the input, and then make a classification on the resulting generated images, whereas our method attempts to first classify the input, making use of an otherwise unchanged capsule classification model, and then generates the input from a high level representation. As such, our method does not increase the computational overhead of classifying the input, compared to the approach of Schott et al. (2018). In addition, the work of Schott et al. (2018) is only applied to MNIST, so our results on the more complex datasets represent an improvement.

heading: 3 PRELIMINARIES

text:
Adversarial Examples Given a clean test image x, its corresponding label y, and a classifier f(·) which predicts a class label given an input, we refer to x′ = x+ δ as an adversarial example if it is able to fool the classifier into making a wrong prediction f(x′) 6= f(x) = y. The small adversarial perturbation δ (where “small” is measured under some norm) causes the adversarial example x′ to appear visually similar to the clean image x but to be classified differently. In the unrestricted case where we only require that f(x′) 6= y, we refer to x′ as an “untargeted adversarial example”. A more powerful attack is to generate a “targeted adversarial example”: instead of simply fooling the classifier to make a wrong prediction, we force the classifier to predict some targeted label f(x′) = t 6= y. In this paper, the target label t is selected uniformly at random as any label which is not the ground-truth correct label. As is standard practice in the literature, in this paper we test our detection mechanism on three `∞ norm based attacks (fast gradient sign method (FGSM) (Goodfellow et al., 2014), the basic iterative method (BIM) (Kurakin et al., 2016), projected gradient descent (PGD) (Madry et al., 2017)) and one `2 norm based attack (Carlini-Wagner (CW) (Carlini & Wagner, 2017b)).
Capsule Networks Capsule Networks (CapsNets) are an alternative architecture for neural networks (Sabour et al., 2017; Hinton et al., 2018). In this work we make use of the CapsNet architecture detailed by (Sabour et al., 2017). Unlike a standard neural network which is made up of layers of scalar-valued units, CapsNets are made up of layers of capsules, that output a vector or matrix. Intuitively, just as one can think of the activation of a unit in a normal neural network as the presence of a feature in the input, the activation of a capsule can be thought of as both the presence of a feature and the pose parameters that represent attributes of that feature. A top-level capsule in a classification network therefore outputs both a classification and pose parameters that represent the instance of that class in the input. This high level representation allows us to train a reconstruction network.
Threat Model In this paper, we test our detection mechanism against both white-box and black-box attacks. For white-box attacks, the adversary has full access to the model as well as its parameters. In particular, the adversary is allowed to compute the gradient through the model to generate adversarial examples. To perform black-box attacks, the adversary is allowed to know the network architecture but not its parameters. Therefore, we retrain a substitute model that has the same architecture as the target model and generate adversarial examples by attacking the substitute model. Then we transfer these attacks to the target model. For `∞ based attacks, we always control the `∞ norm of the adversarial perturbation to be within a relatively small bound ∞, specific to each dataset.

heading: 4 DETECTING ADVERSARIAL IMAGES BY RECONSTRUCTION

text: To detect adversarial images, we make use of the reconstruction network proposed in (Sabour et al., 2017), which takes pose parameters v as input and outputs the reconstructed image r(v). The reconstruction network is simply a fully connected neural network with two ReLU hidden layers with 512 and 1024 units respectively, with a sigmoid output with the same dimensionality as the dataset. The reconstruction network is trained to minimize the `2 distance between the input image and the reconstructed image. This same network architecture is used for all the models and datasets we explore. The only difference is what is given to the reconstruction network as input.

heading: 4.1 MODELS

text:
CapsNet The reconstruction network of the CapsNet is class-conditional: It takes in the pose parameters of all the class capsules and masks all values to 0 except for the pose parameters of the predicted class. We use this reconstruction network for detecting adversarial attacks by measuring the Euclidean distance between the input and a class conditional reconstruction. Specifically, for any given input x, the CapsNet outputs a prediction f(x) as well as the pose parameters v for all classes. The reconstruction network takes in the pose parameters and then selects the pose parameter corresponding to the predicted class, denoted as vf(x), to generate a reconstruction r(vf(x)). Then we compute the `2 reconstruction distance d(x) = ‖r(vf(x)), x‖2 between the reconstructed image and the input image, and compare it with a pre-defined detection threshold θ (described below in Section 4.2). If the reconstruction distance d(x) is higher than the detection threshold θ, we flag the input as an adversarial example. Figure 1 (b) shows an example of histograms of reconstruction distances for natural images and typical adversarial examples.
CNN+CR Although our strategy is inspired by the reconstruction networks used in CapsNets, the strategy can be extended to standard convolutional neural networks (CNNs). We create a similar architecture, CNN with conditional reconstruction (CNN+CR), by dividing the penultimate hidden layer of a CNN into groups corresponding to each class. The sum of each neuron group serves as the logit for that particular class and the group itself serves the same purpose as the pose parameters in the CapsNet. We use the same masking mechanism as Sabour et al. (2017) to select the pose parameter corresponding to the predicted label vf(x) and generate the reconstruction based on the selected pose parameters. In this way we extend the class-conditional reconstruction network to standard CNNs.
CNN+R We can also create a more naı̈ve implementation of our strategy by simply computing the reconstruction from the activations in the entire penultimate layer without any masking mechanism. We call this model the ”CNN+R” model. In this way we are able to study the effect of conditioning on the predicted class.

heading: 4.2 DETECTION THRESHOLD

text: We find the threshold θ for detecting adversarial inputs by measuring the reconstruction error between a validation input image and its reconstruction. If the distance between the input and the reconstruction is above the chosen threshold θ, we classify the data as adversarial. Choosing the detection threshold θ involves a trade-off between false positive and false negative detection rates. The optimal threshold depends on the probability of the system being attacked. Such a trade-off is discussed by Gilmer et al. (2018a). In our experiments we don’t tune this parameter and simply set it as the 95th percentile of validation distances. This means our false positive rate on real validation data is 5%.

heading: 4.3 EVALUATION METRICS

text: We use Success Rate to measure the success of attacks. For targeted attacks, the success rate St is defined as the proportion of inputs which are classified as the target class, St = 1N ∑N i (f(x ′ i) = ti), while the success rate for untargeted attacks is defined as the proportion of inputs which are misclassified, Su = 1N ∑N i (f(x ′ i) 6= yi). Previous work (Carlini & Wagner, 2017a; Hosseini et al., 2019) used the True Positive Rate to measure the proportion of adversarial examples that are detected, which alone is insufficient to measure the ability of different detection mechanism because the unsuccessful adversarial examples do not have to be detected. Therefore, in this paper, we propose to use the Undetected Rate: the proportion of attacks that are successful and undetected to evaluate the detection mechanism. For targeted attacks, the undetected rate is defined as Rt = 1N ∑N i (f(x ′ i) = ti) ∩ (d(x′i) ≤ θ), where d(·) computes the reconstruction distance of the input and θ denotes the detection threshold introduced in Section 4.2. Similarly, the undetected rate for untargeted attacks Ru can be defined as Ru = 1N ∑N i (f(x ′ i) 6= yi) ∩ (d(x′i) ≤ θ). The smaller the undetected rate Rt or Ru is, the stronger the model is in detecting adversarial examples. The undetected rate can also be used to evaluate the attacks (higher is better). We also plot the Undetected Rate vs. False Positive Rate curve to compare the detection performance between different models, where False Positive Rate is defined as the proportion of clean examples that are misclassified as the adversarial example by the detection method.

heading: 4.4 TEST MODELS AND DATASETS

text: In all experiments, all three models (CapsNet, CNN+R, and CNN+CR) have the same number of parameters and were trained with Adam (Kingma & Ba, 2014) for the same number of epochs. In general, all models achieved similar test accuracy. We did not do an exhaustive hyperparameter search on these models, instead we chose hyperparameters that allowed each model to perform roughly equivalently on the test sets. We run experiments on three datasets: MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), and SVHN (Netzer et al., 2011). The test error rate for each model on these three datasets, as well as details of the model architectures, can be seen in Section A and Section B in the Appendix.

heading: 5 EXPERIMENTS

text: We first demonstrate how reconstruction networks can detect standard white and black-box attacks in addition to naturally corrupted images. Then, we introduce the “reconstructive attack”, which is specifically designed to circumvent our defense and show that it is a more powerful attack in this setting. Based on this finding, we qualitatively study the kind of misclassifications caused by the reconstructive attack and argue that they suggest that CapsNets learn features that are better aligned with human perception.

heading: 5.1 STANDARD ATTACKS

text:
White Box We present the success and undetected rates for several targeted and untargeted attacks on MNIST (Table 1), FashionMNIST, and SVHN (Table 5 presented in the Appendix). Our method
is able to accurately detect many attacks with very low undetected rates. Capsule models almost always have the lowest undetected rates out of our three models. It is worth noting that this method performs best with the simplest dataset, MNIST, and that the highest undetected rates are found with the Carlini-Wagner attack on the SVHN dataset. This illustrates both the strength of this attack and a shortcoming of our defense, namely that our detection mechanism relies on `2 image distance as a proxy for visual similarity, and in the case of higher dimensional color datasets such as SVHN, this proxy is less meaningful.
Black Box We also tested our detection mechanism results on black box attacks. Given the low undetected rates in the white-box settings, it is not surprising that our detection method is able to detect black box attacks as well. In fact, on the MNIST dataset the capsule model is able to detect all targeted and untargeted PGD attacks. Both the CNN-R and the CNN-CR models are able to detect the black box attacks as well, but with a relatively higher undetected rate. A table of these results can be seen in Table 7 in the Appendix.

heading: 5.2 CORRUPTION ATTACKS

text: Recent work has argued that improving the robustness of neural networks to `p norm bounded adversarial attacks should not come at the expense of increasing error rates under distributional shifts that do not affect human classification rates and are likely to be encountered in the “realworld” (Gilmer et al., 2018a). For example, if an image is corrupted due to adverse weather, lighting, or occlusion, we might hope that our model can continue to provide reliable predictions or detect the distributional shift. We can test our detection method on its ability to detect these distributional shifts by making use of the Corrupted MNIST dataset (Mu & Gilmer, 2019). This data set contains many visual transformations of MNIST that do not seem to affect human performance, but nevertheless are strongly misclassified by state-of-the-art MNIST models. Our three models can almost always detect these distributional shifts (in all corruptions CapsNets have either a small undetected rate or an undetected rate of 0). The error rate (the proportion of misclassified input) and undetected rate of three test models on the Corrupted MNIST dataset is shown in Table 2. Please refer to Figure 7 and Figure 8 in the Appendix for visualization of Corrupted MNIST.

heading: 5.3 RECONSTRUCTIVE ATTACKS

text:
Thus far we have only evaluated previously-defined attacks. Following the suggestion in (Carlini & Wagner, 2017a) that detection methods need to show effectiveness towards defense-aware attacks, we introduce an attack specifically designed to take into account our defense mechanism. In order to construct adversarial examples that cannot be detected by the network, we propose a two-stage optimization method to generate a “reconstructive attack”.
Untargeted Reconstructive Attacks To construct untargeted reconstructive attacks, we first update the perturbation based on the gradient of the cross-entropy loss function following a standard FGSM attack (Goodfellow et al., 2014), that is:
δ ← clip (δ + c · β · sign(∇δ`net(f(x+ δ), y))), (1)
where `net(f(·), y) is the cross-entropy loss function, is the `∞ bound for our attacks, c is a hyperparameter controlling the step size in each iteration and β is a hyperparameter which balances the importance of the cross-entropy loss and the reconstruction loss (explained further below). In the second stage, we focus on constraining the reconstructed image from the newly predicted label to have a small reconstruction distance by updating δ according to
δ ← clip (δ − c · (1− β) · sign(∇δ(‖r(vf(x+δ))− (x+ δ)‖2))), (2)
where r(vf(x+δ)) is the class-conditional reconstruction based on the predicted label f(x+ δ) in a CapsNet or CNN+CR network. The δ used here is the optimized δ from the first stage. ‖r(vf(x+δ))− (x + δ)‖2 is the `2 reconstruction distance between the reconstructed image and the input image. Since the CNN+R network does not use the class conditional reconstruction, we simply use the reconstructed image without the masking mechanism. According to Eqn 1 and Eqn 2, we can see that β balances the importance between the success rate of attacks and the reconstruction distance. This hyperparameter was tuned for each model and each dataset in order to create the strongest attacks. The success rate and undetected rate change as this parameter, which is shown in Figure 9 in Appendix.
Targeted Reconstructive Attacks We perform a similar two-stage optimization to construct targeted reconstructive attacks, by defining a target label and attempting to maximize the classification probability of this label, and minimize the reconstruction error from corresponding capsule. Because the targeted label is given, another way to construct targeted reconstructive attacks is to combine these two stages into one stage via minimizing the loss function ` = β · `net(f(x+δ), y)+(1−β) · ‖r(vf(x+δ))− (x+δ)‖2. We implemented both of these targeted reconstructive attacks and found that the two-stage version is a stronger attack. Therefore, all the Reconstructive Attack experiments performed in this paper are based on two-stage optimization.
We build our reconstructive attack based on the standard PGD attack, denoted as R-PGD, and test the performance of our detection models against this reconstructive attack in a white-box setting (white-box Reconstructive FGSM and BIM are reported in Table 6 in the Appendix). Comparing Table 1 and Table 3, we can see that the Reconstructive Attack is significantly less successful at changing the models prediction (lower success rates than the standard attack). However, this attack is more successful at fooling our detection method. For all attacks and datasets the capsule model has the lowest attack success rate and the lowest undetected rate. We report results for black-box R-PGD attacks in Table 7 in the Appendix, which suggest similar conclusions.
In addition, we report the undetected rate of the white-box targeted defense-aware R-PGD attack versus the False Positive Rate on the MNIST, Fashion-MNIST and SVHN datasets in Figure 2. We can clearly see that the undetected rate of the defense-aware attack against CapsNet is significantly smaller than the CNN-based networks, which suggests that CapsNets are more robust against adversarial attacks. Furthermore, CNN with class-conditional reconstruction (CNN+CR) has smaller undetected rate at the same False Positive Rate compared to the CNN without class-conditional reconstruction (CNN+R), which suggests the class-conditional information is helpful in our models to improve the robustness against adversarial attacks.

heading: 5.4 CIFAR-10 DATASET

text:
In order to show that our method based on CapsNet is capable to scale up to more complex datasets, we test our detection method with a deeper reconstruction network on CIFAR-10 (Krizhevsky, 2009). The classification accuracy on the clean test dataset is 92.2%. In addition, we display the undetected rate of the white-box/black-box defense-aware R-PGD attack against CapsNets versus the False Positive Rate in Figure 3 (Left), where we can see a significant drop of the undetected rate of black-box R-PGD compared to the white-box setting. This indicates the CapsNets greatly reduce the attack transferability and the threat of black-box attacks.
Class-conditional Information To investigate the effectiveness of the class-conditional information in the reconstruction network, we compare our CapsNet based on (Sabour et al., 2017) with the other two variants of CapsNets: “CapsNet All” and “DeepCaps” (Rajasegaran et al., 2019). In “CapsNet All”, we remove the masking mechanism in the CapsNet and use all the capsules to do the reconstruction. In “DeepCaps”, we extract the winning-capsule information as a single vector and used it as the input for the reconstruction network instead of using a masking mechanism to mask out the losing capsules information. In this way, the class information in DeepCaps is more explicitly fed into the reconstruction network. As shown in Figure 3 (right), our CapsNet has the best detection performance (the lowest undetected rate at the same False Positive Rate) compared to the other two
Capsule models. “DeepCaps” performs slightly worse than our “CapsNet” and “CapsNet All” has the worst detection performance. Therefore, we conclude that the class-conditional information used in the reconstruction network increases the model’s robustness to adversarial attack. This also holds true to CNN-based networks because CNN+CR has a better detection performance than CNN+R, shown in Figure 2.

heading: 6 VISUAL COHERENCE OF THE RECONSTRUCTIVE ATTACK

text:
The great success of CapsNet over CNN-based models motivates us to further diagnose the generated adversarial examples for CapsNets. If our true aim in adversarial robustness research is to create models that make predictions based on reasonable and human-observable features, then we would prefer models that are more likely to misclassify a “shirt” as a “t-shirt” (in the case of FashionMNIST) than to misclassify a “bag” as a “sweater”. For a model to behave ideally, the success of an adversarial perturbation would be related to the visual similarity between the source and the target class. By visualizing a matrix of adversarial success rates between each pair of classes (shown in Figure 4), we can see that for the capsule model there is a great variance between the source and target class pairs and that the success rate of attacks is highly related to the visual similarity of the classes. However, this is not the case for either of the other two CNN-based models.
Thus far we have treated all attacks as equal. However, a key component of an adversarial example is that it is visually similar to the source image, and that it does not resemble the adversarial target class. The adversarial research community makes use of a small epsilon bound as a mechanism for ensuring that the resultant adversarial attacks are visually unchanged from the source image. For standard attacks against CNN-based models this heuristic is sufficient, because taking gradient steps in the image space in order to have a network misclassify an image normally results in something
visually similar to the source image. But this is not the case for adversarial attacks against CapsNets. As shown in Figure 5, when we use R-PGD to attack the CapsNet, many of the resultant attacks resemble members of the target class. In this way, they stop being “adversarial”. As such, an attack detection method which does not detect them as adversarial is arguably behaving correctly. This puts the previously undetected rates presented earlier in a new light, and illustrates a difficulty in the evaluation of adversarial attacks and defenses. In addition, it should be noted that this phenomenon rarely occurs in a standard convolutional neural network, which suggests that the features captured by CapsNet are more aligned with human perception.

heading: 7 DISCUSSION

text: Our detection mechanism relies on a similarity metric (i.e. a measure of reconstruction error) between the reconstruction and the input. This metric is required both during training in order to train the reconstruction network and during test time in order to flag adversarial examples. In the four datasets we have evaluated, the distance between examples roughly correlates with semantic similarity. However, this may not be the case for images in more complex datasets such as the SUN dataset (Xiao et al., 2010) and ImageNet (Deng et al., 2009), in which two images may be similar in terms of semantic content but nevertheless have significant `2 distance. A better similarity metric (Theis et al., 2015; Zhang et al., 2018) can be further explored to extend our methods to more complex problems. Furthermore our reconstruction network is trained on a hidden representation of one class but is trained to reconstruct the entire input. In datasets without distractors or backgrounds, this is not a problem. But in the case of ImageNet, in which the object responsible for the classification is not the only object in the image, attempting to reconstruct the entire input from a class encoding seems misguided.

heading: 8 CONCLUSION

text:
We have presented a class-conditional reconstruction-based detection method that does not rely on a specific predefined adversarial attack. We have shown that by reconstructing the input from the internal class-conditional representation, our system is able to accurately detect black-box and white-box FGSM, BIM, PGD, and CW attacks. We then proposed a new attack to beat our defense - the Reconstructive Attack - in which the adversary optimizes not only the classification loss but also minimizes the reconstruction loss. We showed that this attack was able to fool our detection mechanism but with a much smaller success rate than a standard attack.
Compared to CNN-based models, we showed that the CapsNet was able to detect adversarial examples with greater accuracy on all the datasets we explored. To further explain the success of CapsNet, we qualitatively showed that the success of the reconstructive attack was highly related to the visual similarity between the target class and the source class for the CapsNet. In addition, we showed that images generated by this reconstructive attack to attack the CapsNet are not typically adversarial, i.e. many of the resultant attacks resemble members of the target class even with a small `∞ norm bound. These are not the case for the CNN-based models. The extensive qualitative studies indicate that the capsule model relies on visual features similar to those used by humans. We believe this is a step towards solving the true problem posed by adversarial examples.

heading: A NETWORK ARCHITECTURES

text: Figure 6 shows the architecute of the capsule network, the CNN reconstruction model and the CNN conditional reconstruction model used for experiments on MNIST, FashionMNIST and SVHN dataset. MNIST and Fashion MNIST have exactly the same architectures while we use larger models for SVHN. Note that the only difference between the CNN reconstruction (CNN+R) and the CNN conditional reconstruction (CNN+CR) is the masking procedure on the input to the reconstruction network based on the predicted class. All three models have the same number of parameters.

heading: B TEST MODELS

text:
The error rate of each test model used in the paper are presented in Table 4. We ensure that they have similar performance.
C IMPLEMENTATION DETAILS
For all the `∞ based adversarial examples, the `∞ norm of the perturbations is bound by , which is set to 0.3, 0.1, 0.1 for MNIST, Fashion MNIST and SVHN dataset respectively following previous work (Madry et al., 2017; Song et al., 2017). In FGSM based attacks, the step size c is 0.05. In BIM-based (Kurakin et al., 2016) and PGD-based (Madry et al., 2017) attacks, the step size c is 0.01 for all the datasets and the number of iterations are 1000, 500 and 200 for MNIST, Fashion MNIST and SVHN dataset respectively. We choose a sufficiently large number of iterations to ensure the attacks has converged.
We use the publicly released code from the authors of (Carlini & Wagner, 2017b) to perform the CW attack for our models. The number of iterations are set to 1000 for all three datasets.

heading: D WHITE BOX STANDARD ATTACKS

text:
The results of four white box standard attacks on the three datasets are shown in Table 5.
E VISUALIZATION OF CORRUPTED MNIST DATASET
Visualization of examples from Corrupted MNIST dataset (Mu & Gilmer, 2019) and the corresponding reconstructed images for each model are shown in Figure 7 and Figure 8.

heading: F RECONSTRUCTIVE ATTACKS

text:
The results of Reconstructive FGSM, BIM and PGD on the three datasets are reported in Table 6.
F.1 HYPERPARAMETER β
Figure 9 shows the plot of success rate and undetected rate versus the hyperparameter β which balances the importance between attacking the classifier and fooling the detection mechanism in the targeted reconstructive PGD attacks on the MNIST dataset.

heading: G BLACK BOX ATTACKS

text: H VISUALIZATION OF ADVERSARIAL EXAMPLES AND RECONSTRUCTIONS

title: CLASS-CONDITIONAL CAPSULE RECONSTRUCTIONS