summaries: The paper proposes cAdv and sAdv, two new unrestricted adversarial attack methods that manipulates either color or texture of an image. To these end, the paper employes another parametrized colorization techniques (and texture transfer method) and proposes optimization objectives for finding adversarial examples with respect to each semantic technique. Experimental results show that the proposed methods are more robust on existing defense methods and more transferrable accross models. The paper also performs a user study to show that the generated examples are fairly imperceptible like the C&W attack.

summaries: This paper introduces two new adversarial attacks: one is generating adversarial examples by colouring the original images and the other is by changing textures of the original images. Specifically, the former one minimises the cross-entropy between the output of the classifier and the target label with the network weights of a pre-trained colourisation network. While the latter minimises the cross-entropy as well as the loss that defines the texture differences.

abstractText: Machine learning models, especially deep neural networks (DNNs), have been shown to be vulnerable against adversarial examples which are carefully crafted samples with a small magnitude of the perturbation. Such adversarial perturbations are usually restricted by bounding their Lp norm such that they are imperceptible, and thus many current defenses can exploit this property to reduce their adversarial impact. In this paper, we instead introduce “unrestricted” perturbations that manipulate semantically meaningful image-based visual descriptors – color and texture – in order to generate effective and photorealistic adversarial examples. We show that these semantically aware perturbations are effective against JPEG compression, feature squeezing and adversarially trained model. We also show that the proposed methods can effectively be applied to both image classification and image captioning tasks on complex datasets such as ImageNet and MSCOCO. In addition, we conduct comprehensive user studies to show that our generated semantic adversarial examples are photorealistic to humans despite large magnitude perturbations when compared to other attacks.

authors:
Jyoti Aneja
Aditya Deshpande
Alexander G Schwing

title: Convolutional image captioning

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Andrew Brock
Jeff Donahue
Karen Simonyan

title: Large scale gan training for high fidelity natural image synthesis

venue: arXiv preprint arXiv:1809.11096,

authors:
Tom B Brown
Nicholas Carlini
Chiyuan Zhang
Catherine Olsson
Paul Christiano
Ian Goodfellow

title: Unrestricted adversarial examples

venue: arXiv preprint arXiv:1809.08352,

authors:
Nicholas Carlini
David Wagner

title: Towards evaluating the robustness of neural networks

venue: IEEE Symposium on Security and Privacy (SP),

authors:
Hongge Chen
Huan Zhang
Pin-Yu Chen
Jinfeng Yi
Cho-Jui Hsieh

title: Attacking visual language grounding with adversarial examples: A case study on neural image captioning

authors:
Nilaksh Das
Madhuri Shanbhogue
Shang-Tse Chen
Fred Hohman
Li Chen
Michael E Kounavis
Duen Horng Chau

title: Keeping the bad guys out: Protecting and vaccinating deep learning with jpeg compression

venue: arXiv preprint arXiv:1705.02900,

authors:
J. Deng
W. Dong
R. Socher
L.-J. Li
K. Li
L. Fei-Fei

title: ImageNet: A Large-Scale Hierarchical Image Database

venue: In CVPR09,

authors:
Aditya Deshpande
Jiajun Lu
Mao-Chuang Yeh
Min Jin Chong
David A Forsyth

title: Learning diverse image colorization

venue: In CVPR,

authors:
Isaac Dunn
Tom Melham
Daniel Kroening

title: Generating realistic unrestricted adversarial inputs using dual-objective gan training

venue: arXiv preprint arXiv:1905.02463,

authors:
Alexei A Efros
William T Freeman

title: Image quilting for texture synthesis and transfer

venue: In Proceedings of the 28th annual conference on Computer graphics and interactive techniques,

authors:
Leon Gatys
Alexander S Ecker
Matthias Bethge

title: Texture synthesis using convolutional neural networks

venue: In Advances in Neural Information Processing Systems,

authors:
Leon A Gatys
Alexander S Ecker
Matthias Bethge

title: Image style transfer using convolutional neural networks

authors:
Robert Geirhos
Patricia Rubisch
Claudio Michaelis
Matthias Bethge
Felix A Wichmann
Wieland Brendel

title: Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

venue: arXiv preprint arXiv:1811.12231,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In Advances in neural information processing systems,

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: arXiv preprint arXiv:1412.6572,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Geoffrey Hinton
Li Deng
Dong Yu
George E Dahl
Abdel-rahman Mohamed
Navdeep Jaitly
Andrew Senior
Vincent Vanhoucke
Patrick Nguyen
Tara N Sainath

title: Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups

venue: IEEE Signal Processing Magazine,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Hossein Hosseini
Radha Poovendran

title: Semantic adversarial examples

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops,

authors:
Xun Huang
Serge J Belongie

title: Arbitrary style transfer in real-time with adaptive instance normalization

venue: In ICCV,

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei

title: A Efros. Image-to-image translation with conditional adversarial networks. 2017

authors:
Justin Johnson
Alexandre Alahi
Li Fei-Fei

title: Perceptual losses for real-time style transfer and superresolution

venue: In European Conference on Computer Vision,

authors:
Ameya Joshi
Amitangshu Mukherjee
Soumik Sarkar
Chinmay Hegde

title: Semantic adversarial attacks: Parametric transformations that fool deep classifiers

authors:
Daniel Kang
Yi Sun
Dan Hendrycks
Tom Brown
Jacob Steinhardt

title: Testing robustness against unforeseen adversaries

venue: arXiv preprint arXiv:1908.08016,

authors:
Andrej Karpathy
Li Fei-Fei

title: Deep visual-semantic alignments for generating image descriptions

authors:
Tero Karras
Timo Aila
Samuli Laine
Jaakko Lehtinen

title: Progressive growing of gans for improved quality, stability, and variation

venue: arXiv preprint arXiv:1710.10196,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: ImageNet classification with deep convolutional neural networks

authors:
Alexey Kurakin
Ian J. Goodfellow
Samy Bengio

title: Adversarial examples in the physical world

authors:
Sergey Levine
Chelsea Finn
Trevor Darrell
Pieter Abbeel

title: End-to-end training of deep visuomotor policies

authors:
Yijun Li
Chen Fang
Jimei Yang
Zhaowen Wang
Xin Lu
Ming-Hsuan Yang

title: Universal style transfer via feature transforms

authors:
Tsung-Yi Lin
Michael Maire
Serge Belongie
James Hays
Pietro Perona
Deva Ramanan
Piotr Dollár
C Lawrence Zitnick

title: Microsoft coco: Common objects in context

venue: In European conference on computer vision,

authors:
Dong C Liu
Jorge Nocedal

title: On the limited memory bfgs method for large scale optimization

venue: Mathematical programming,

authors:
Nicolas Papernot
Patrick McDaniel
Ian Goodfellow

title: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples

venue: arXiv preprint arXiv:1605.07277,

authors:
Haonan Qiu
Chaowei Xiao
Lei Yang
Xinchen Yan
Honglak Lee
Bo Li

title: Semanticadv: Generating adversarial examples via attribute-conditional image editing

authors:
Karen Simonyan
Andrew Zisserman

title: Very deep convolutional networks for large-scale image recognition

venue: arXiv preprint arXiv:1409.1556,

title: Constructing unrestricted adversarial examples with generative models

authors:
Liwei Wang
Alexander Schwing
Svetlana Lazebnik

title: Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space

authors:
Chaowei Xiao
Bo Li
Jun-Yan Zhu
Warren He
Mingyan Liu
Dawn Song

title: Generating adversarial examples with adversarial networks

venue: arXiv preprint arXiv:1801.02610,

authors:
Chaowei Xiao
Jun-Yan Zhu
Bo Li
Warren He
Mingyan Liu
Dawn Song

title: Spatially transformed adversarial examples

venue: arXiv preprint arXiv:1801.02612,

authors:
Weilin Xu
David Evans
Yanjun Qi

title: Feature squeezing: Detecting adversarial examples in deep neural networks

venue: arXiv preprint arXiv:1704.01155,

authors:
Yan Xu
Baoyuan Wu
Fumin Shen
Yanbo Fan
Yong Zhang
Heng Tao Shen
Wei Liu

title: Exact adversarial attack to image captioning via structured output learning with latent variables

authors:
Mao-Chuang Yeh
Shuai Tang
Anand Bhattad
Chuhang Zou
David Forsyth

title: Improving style transfer with calibrated metrics

venue: arXiv preprint arXiv:1910.09447,

authors:
Richard Zhang
Phillip Isola
Alexei A Efros

title: Colorful image colorization

authors:
Richard Zhang
Jun-Yan Zhu
Phillip Isola
Xinyang Geng
Angela S Lin
Tianhe Yu
Alexei A Efros

title: Real-time user-guided image colorization with learned deep priors

venue: arXiv preprint arXiv:1705.02999,

heading: 1 INTRODUCTION

text:
Machine learning (ML), especially deep neural networks (DNNs) have achieved great success in various tasks, including image recognition (Krizhevsky et al., 2012; He et al., 2016), speech processing (Hinton et al., 2012) and robotics training (Levine et al., 2016). However, recent literature has shown that these widely deployed ML models are vulnerable to adversarial examples – carefully crafted perturbations aiming to mislead learning models (Carlini & Wagner, 2017; Kurakin et al., 2016; Xiao et al., 2018b). The fast growth of DNNs based solutions demands in-depth studies on adversarial examples to help better understand potential vulnerabilities of ML models and therefore improve their robustness.
To date, a variety of different approaches has been proposed to generate adversarial examples (Goodfellow et al., 2014b; Carlini & Wagner, 2017; Kurakin et al., 2016; Xiao et al., 2018a); and many of these attacks search for perturbation within a bounded Lp norm in order to preserve their photorealism. However, it is known that the Lp norm distance as a perceptual similarity metric is not ideal (Johnson et al., 2016; Isola et al., 2017). In addition, recent work show that defenses trained on Lp bounded perturbation are not robust at all against new types of unseen attacks (Kang et al., 2019). Therefore, exploring diverse adversarial examples, especially those with “unrestricted" magnitude of perturbation has acquired a lot of attention in both academia and industries (Brown et al., 2018).
Recent work based on generative adversarial networks (GANs) (Goodfellow et al., 2014a) have introduced unrestricted attacks (Song et al, 2018). However, these attacks are limited to datasets like MNIST, CIFAR and CelebA, and are usually unable to scale up to bigger and more complex datasets such as ImageNet. Xiao et al. (2018b) directly manipulated spatial pixel flow of an image to produce adversarial examples without Lp bounded constraints on the perturbation. However, the attack does not explicitly control visual semantic representation. More recently, Hosseini & Poovendran (2018) manipulated hue and saturation of an image to create adversarial perturbations. However, these examples are easily distinguishable by human and are also not scalable to complex datasets.
∗ indicates equal contributions.
In this work, we propose unrestricted attack strategies that explicitly manipulate semantic visual representations to generate natural-looking adversarial examples that are “far” from the original image in tems of the Lp norm distance. In particular, we manipulate color (cAdv) and texture (tAdv) to create realistic adversarial examples (see Fig 1). cAdv adaptively chooses locations in an image to change their colors, producing adversarial perturbation that is usually fairly substantial, while tAdv utilizes the texture from other images and adjusts the instance’s texture field using style transfer.
These semantic transformation-based adversarial perturbations shed light upon the understanding of what information is important for DNNs to make predictions. For instance, in one of our case studies, when the road is recolored from gray to blue, the image gets misclassified to tench (a fish) although a car remains evidently visible (Fig. 2b). This indicates that deep learning models can easily be fooled by certain large scale patterns. In addition to image classifiers, the proposed attack methods can be generalized to different machine learning tasks such as image captioning ( Karpathy & Fei-Fei (2015)). Our attacks can either change the entire caption to the target (Chen et al., 2017; Xu et al., 2019) or take on more challenging tasks like changing one or two specific target words from the caption to a target. For example, in Fig. 1, “stop sign” of the original image caption is changed to “cat sitting” and “umbrella is” for cAdv and tAdv respectively.
To ensure our “unrestricted" semantically manipulated images are natural, we conducted extensive user studies with Amazon Mechanical Turk. We also tested our proposed attacks on several state of the art defenses. Rather than just showing the attacks break these defenses (better defenses will come up), we aim to show that cAdv and tAdv are able to produce new types of adversarial examples. Experiments also show that our proposed attacks are more transferable given their large and structured perturbations (Papernot et al., 2016). Our semantic adversarial attacks provide further insights about the vulnerabilities of ML models and therefore encourage new solutions to improve their robustness.
In summary, our contributions are: 1) We propose two novel approaches to generate “unrestricted" adversarial examples via semantic transformation; 2) We conduct extensive experiments to attack both image classification and image captioning models on large scale datasets (ImageNet and MSCOCO); 3) We show that our attacks are equipped with unique properties such as smooth cAdv perturbations and structured tAdv perturbations. 4) We perform comprehensive user studies to show that when compared to other attacks, our generated adversarial examples appear more natural to humans despite their large perturbations; 5) We test different adversarial examples against several state of the art defenses and show that the proposed attacks are more transferable and harder to defend.
2 COLORIZATION ATTACK (cADV)
Background. Image Colorization is the task of giving natural colorization to a grayscale image. This is an ill-posed problem as there are multiple viable natural colorizations given a single grayscale
image. Deshpande et al. (2017) showed that diverse image colorization can be achieved by using an architecture that combines VAE (Kingma & Welling (2013)) and Mixture Density Network; while Zhang et al. (2017) demonstrated an improved and diverse image colorization by using input hints from users guided colorization process.
Our goal is to adversarially color an image by leveraging a pretrained colorization model. We hypothesize that it is possible to find a natural colorization that is adversarial for a target model (e.g., classifier or captioner) by searching in the color space. Since a colorization network learns to color natural colors that conform to boundaries and respect short-range color consistency, we can use it to introduce smooth and consistent adversarial noise with a large magnitude that looks natural to humans. This attack differs from common adversarial attacks which tend to introduce short-scale high-frequency artifacts that are minimized to be invisible for human observers.
We leverage Zhang et al. (2016; 2017) colorization model for our attack. In their work, they produce natural colorizations on ImageNet with input hints from the user. The inputs to their network consist of the L channel of the image in CIELAB color space XL ∈ RH×W×1, the sparse colored input hints Xab ∈ RH×W×2, and the binary mask M ∈ BH×W×1, indicating the location of the hints. cAdv Objectives. There are a few ways to leverage the colorization model to achieve adversarial objectives. We experimented with two main methods and achieved varied results.
Network weights. The straightforward approach of producing adversarial colors is to modify Zhang et al. (2017) colorization network, C directly. To do so, we simply update C by minimizing the adversarial loss objective Jadv, which in our case, is the cross entropy loss. t represents target class and F represents the victim network.
θ∗ = argmin θ Jadv(F(C(XL, Xab,M ; θ)), t) (1)
Hints and mask. We can also vary input hints Xab and mask M to produce adversarial colorizations. Hints provides the network with ground truth color patches that guides the colorization, while the mask provides its spatial location. By jointly varying both hints and mask, we are able to manipulate the output colorization. We can update the hints and mask as follows:
M∗, X∗ab = argmin M,Xab Jadv(F(C(XL, Xab,M ; θ)), t) (2)
cAdvAttack Methods. Attacking network weights allows the network to search the color space with no constraints for adversarial colors. This attack is the easiest to optimize, but the output colors are not realistic as shown in Fig. 2. Our various strategies outlined below are ineffective as the model learns to generate the adversarial colors without taking into account color realism. However, colorizations produced often correlate with colors often observed in the target class. This suggests that classifiers associate certain colors with certain classes which we will discuss more in our case study.
Attacking input hints and mask jointly gives us natural results as the pretrained network will not be affected by our optimization. Attacking hints and mask separately also works but takes a long optimization time and give slightly worse results. For our experiments, we use Adam Optimizer (Kingma & Ba (2014)) with a learning rate of 10−4 in cAdv. We iteratively update hints and mask until our adversarial image reaches the target class and the confidence change of consecutive iterations does not exceed a threshold of 0.05.
Control over colorization. Current attack methods lack control over where the attack occurs, opting to attack all pixels indiscriminately. This lack of control is not important for most attacks where the is small but is concerning in cAdv where making unstructured large changes can be jarring. To produce realistic colorization, we need to avoid making large color changes at locations where colors are unambiguous (e.g. roads in general are gray) and focus on those where colors are ambiguous (e.g. an umbrella can have different colors). To do so, we need to segment an image and determine which segments should be attacked or preserved.
To segment the image into meaningful areas, we cluster the image’s ground truth AB space using K-Means. We first use a Gaussian filter of σ = 3 to smooth the AB channels and then cluster them into 8 clusters. Then, we have to determine which cluster’s colors should be preserved. Fortunately, Zhang et al. (2017) network output a per-pixel color distribution for a given image which we used to calculate the entropy of each pixel. The entropy represents how confident the network is at assigning a color at that location. The average entropy of each cluster represents how ambiguous their color is. We want to avoid making large changes to clusters with low-entropy while allowing our attack to change clusters with high entropy. One way to enforce this behavior is through hints, which are sampled from the ground truth at locations belonging to clusters of low-entropy. We sample hints from the k clusters with the lowest entropy which we refer as cAdvk (e.g. cAdv2 samples hints from the 2 lowest entropy clusters).
Number of input hints. Network hints constrain our output to have similar colors as the ground truth, avoiding the possibility of unnatural colorization at the cost of color diversity. This trade-off is controlled by the number of hints given to the network as initialization (Fig. 4). Generally, providing more hints gives us similar colors that are observed in original image. However, having too many hints is also problematic. Too many hints makes the optimization between drawing adversarial colors and matching local color hints difficult. Since the search space for adversarial colors is constrained because of more hints, we may instead generate unrealistic examples.
Number of Clusters. The trade-off between the color diversity and the color realism is also controlled by the number of clusters we sample hints from as shown in Fig. 3. Sampling from multiple clusters gives us realistic colors closer to the ground truth image at the expense of color diversity.
Empirically, from our experiments we find that in terms of color diversity, realism, and robustness of attacks, using k = 4 and 50 hints gives us better adversarial examples. For the rest of this paper, we fix 50 hints for all cAdvk methods.
3 TEXTURE ATTACK (tADV)
Background. Texture transfer extracts texture from one image and adds it to another. Transferring texture from one source image to another target image has been widely studied in computer vision ( Efros & Freeman (2001); Gatys et al. (2015)). The Convolutional Neural Network (CNN) based texture transfer from Gatys et al. (2015) led to a series of new ideas in the domain of artistic style transfer ( Gatys et al. (2016); Huang & Belongie (2017); Li et al. (2017); Yeh et al. (2019)). More recently, Geirhos et al. (2018) showed that DNNs trained on ImageNet are biased towards texture for making predictions.
Our goal is to generate adversarial examples by infusing texture from another image without explicit constraints on Lp norm of the perturbation. For generating our tAdv examples, we used a pretrained VGG19 network (Simonyan & Zisserman, 2014) to extract textural features. We directly optimize our victim image (Iv) by adding texture from a target image (It). A natural strategy to transfer texture is by minimizing within-layer feature correlation statistics (gram matrices) between two images Gatys et al. (2015; 2016). Based on Yeh et al. (2019), we find that optimizing cross-layer gram matrices instead of within-layer gram matrices helps produce more natural looking adversarial examples. The difference between the within-layer and the cross-layer gram matrices is that for a within-layer, the feature’s statistics are computed between the same layer. For a cross-layer, the statistics are computed between two adjacent layers.
tAdv Objectives. tAdv directly attacks the image to create adversarial examples without modifying network parameters. Moreover, there is no additional content loss that is used in style transfer methods (Gatys et al. (2016); Yeh et al. (2019)). Our overall objective function for the texture attack contains a texture transfer loss (LAt ) and an cross-entropy loss (Jadv).
LAtAdv = αL A t (Iv, It) + βJadv(F(Iv), t) (3)
Unlike style transfer methods, we do not want the adversarial examples to be artistically pleasing. Our goal is to infuse a reasonable texture from a target class image to the victim image and fool a classifier or captioning network. To ensure a reasonable texture is added without overly perturbing the victim image too much, we introduce an additional constraint on the variation in the gram matrices of the victim image. This constraint helps us to control the image transformation procedure and prevents it from producing artistic images. Let m and n denote two layers of a pretrained VGG-19 with a decreasing spatial resolution and C for number of filter maps in layer n, our texture transfer loss is then given by
LAt (Iv, It) = ∑
(m,n)∈L
1
C2 ∑ ij ∥∥Gm,nij (Iv)−Gm,nij (It)∥∥2 std ( Gm,nij (Iv) ) (4)
Let f be feature maps, Ufn be an upsampled fn that matches the spatial resolution of layer m. The cross layer gram matrices G between the victim image (Iv) and a target image (It) is given as
Gm,nij (I) = ∑ p [ fmi,p(I) ] [ Ufnj,p(I) ]T (5) Texture Transfer. To create tAdv adversarial examples, we need to find images to extract the texture from, which we call “texture source” (Ts). A naive strategy is to randomly select an image from the data bank as Ts. Though this strategy is successful, their perturbations are clearly perceptible. Alternatively, we can randomly select Ts from the adversarial target class. This strategy produces less perceptible perturbations compared to the random Ts method as we are extracting a texture from the known target class. A better strategy to select Ts is to find a target class image that lies closest to the victim image in the feature space using nearest neighbors. This strategy is sensible as we assure our victim image has similar feature statistics as our target image. Consequently, minimizing gram matrices is easier and our attack generates more natural looking images (see Fig. 5).
For texture transfer, we extract cross-layer statistics in Eq. 4 from the R11, R21, R31, R41, and R51 of a pretrained VGG19. We optimize our objective (Eq. 3) using an L-BFGS (Liu & Nocedal (1989)) optimizer. tAdv attacks are sensitive and if not controlled well, images get transformed into artistic images. Since we do not have any constraints over the perturbation norm, it is necessary to decide when to stop the texture transfer procedure. For a successful attack (images look realistic), we limit our L-BFGS to fixed number of small steps and perform two set of experiments: one with only one iteration or round of L-BFGS for 14 steps and another with three iterations of 14 steps. For the three iterations setup, after every iteration, we look at the confidence of our target class and stop if the confidence is greater than 0.9.
Texture and Cross-Entropy Weights. Empirically, we found setting α to be in the range [150, 1000] and β in the range [ 10−4, 10−3 ] to be successful and also produce less perceptible tAdv examples. The additional cross-entropy based adversarial objective Jadv helps our optimization. We ensure large flow of gradients is from the texture loss and they are sufficiently larger than the adversarial crossentropy objective. The adversarial objective also helps in transforming victim image to adversarial without stylizing the image. All our tabulated results are shown for one iteration, α = 250 and β = 10−3, unless otherwise stated. We use the annotation tAdviterα for the rest of the paper to denote the texture method that we are using.
Control over Texture. The amount of texture that gets added to our victim image is controlled by the texture weight coefficient (α). Increasing texture weights improves attack success rate at the cost of noticeable perturbation. When compared to within-layer statistics, the cross-layer statistics that we use are not only better at extracting texture, it is also easier to control the texture weight.

heading: 4 EXPERIMENTAL RESULTS

text:
In this section, we evaluate the two proposed attack methods both quantitatively, via attack success rate under different settings, and qualitatively, based on interesting case studies. We conduct our experiments on ImageNet Deng et al. (2009) by randomly selecting images from 10 sufficiently different classes predicted correctly for the classification attack.
We use a pretrained ResNet 50 classifier (He et al. (2016)) for all our methods. DenseNet 121 and VGG 19 (Huang et al.; Simonyan & Zisserman (2014)) are used for our transferability analysis.
4.1 cADV ATTACK
cAdv achieves high targeted attack success rate by adding realistic color perturbation. Our numbers in Table 1 and Table 2 also reveal that cAdv examples with larger color changes (consequently more color diversity) are more robust against transferability and adversarial defenses. However, these big changes are found to be slightly less realistic from our user study (Table 2, Table 4).
Smooth cAdv perturbations. Fig. 8 in our Appendix shows interesting properties of the adversarial colors. We observe that cAdv perturbations are locally smooth and are relatively low-frequency. This is different from most adversarial attacks that generate high-frequency noise-like perturbations. This phenomenon can be explained by the observation that colors are usually smooth within object boundaries. The pretrained colorization model will thus produce smooth, low-frequency adversarial colors that conform to object boundaries.
Importance of color in classification. From Fig. 2, we can compare how different target class affects our colorization results if we relax our constraints on colors (cAdv on Network Weights, 0 hints). In many cases, the images contain strong colors that are related to the target class. In the case of golf-cart, we get a green tint over the entire image. This can push the target classifier to misclassify the image as green grass is usually overabundant in benign golf-cart images. Fig. 2b shows our attack on an image of a car to tench (a type of fish). We observe that the gray road turned blue and that the colors are tinted. We can hypothesize that the blue colors and the tint fooled the classifier into thinking the image is a tench in the sea.
The colorization model is originally trained to produce natural colorization that conforms to object boundaries. By adjusting its parameters, we are able to produce such large and abnormal color change that is impossible with our attack on hints and mask. These colors, however, show us some evidence that colors play a stronger role in classification than we thought. We reserve the exploration of this observation for future works.
While this effect (strong color correlation to target class) is less pronounced for our attack on hints and mask, for all cAdv methods, we observe isoluminant color blobs. Isoluminant colors are characterized
by a change in color without a corresponding change in luminance. As most color changes occur along edges in natural images, it is likely that classifiers trained on ImageNet have never seen isoluminant colors. This suggests that cAdv might be exploiting isoluminant colors to fool classifiers.
4.2 tADV ATTACK
tAdv successfully fools the classifiers with a very small weighted adversarial cross-entropy objective (β) when combined with texture loss, while remaining realistic to humans. As shown in Table 1, our attacks are highly successful on white-box attacks tested on three different models with the nearest neighbor texture transfer approach. We also show our attacks are more transferable to other models. In our Appendix, we show ablation results for tAdv attacks along with other strategies that we used for generating tAdv adversarial examples.
Structured tAdv Perturbations. Since we extract features across different layers of VGG, the tAdv perturbations follow a textural pattern. They are more structured and organized when compared to others. Our tAdv perturbations are big when compared with existing attack methods in Lp norm. They are of high-frequency and yet imperceptible (see Fig. 1 and Fig. 8).
Importance of Texture in Classification. Textures are crucial descriptors for image classification and Imagenet trained models can be exploited by altering the texture. Their importance is also shown in the recent work from Geirhos et al. (2018). Our results also shows that even with a small or invisible change in the texture field can break the current state of the art classifiers.

heading: 4.3 DEFENSE AND TRANSFERABILITY ANALYSIS

text:
We test all our attacks and other existing methods with images attacked from Resnet50. We evaluate them on three defenses – JPEG defense (Das et al., 2017), feature squeezing (Xu et al., 2017) and adversarial training. By leveraging JPEG compression and decompression, adversarial noise may be removed. We tested our methods against JPEG compression of 75. Feature squeezing is a family of simple but surprisingly effective strategies, including reducing color bit depth and spatial smoothing. Adversarial training has been shown as an effective but costly method to defend against adversarial attacks. Mixing adversarial samples into training data of a classifier improves its robustness without affecting the overall accuracy. We were able to obtain an adversarially pretrained Resnet152 model on ImageNet dataset and hence we tested our Resnet50 attacked images with this model.
Robustness. In general, our attacks are more robust to the considered defenses and transferable for targeted attacks. For cAdv, there is a trade-off between more realistic colors (using more hints and sampling from more clusters) and attack robustness. From Table 1 and 2, we show that as we progressively use more clusters, our transferability and defense numbers drop. A similar trend is observed with the change in the number of hints. cAdv is robust to JPEG defense and adversarial training because of their large and spatially smooth perturbations. For tAdv, increasing texture weight (α) does not necessarily perform well with the defense even though it increases attack success rate, but increasing texture flow with more iterations improves attack’s robustness against defenses.

heading: 5 HUMAN PERCEPTUAL STUDIES

text: To quantify how realistic tAdv and cAdv examples are, we conducted a user study on Amazon Mechanical Turk (AMT). We follow the same procedure as described in (Zhang et al., 2016; Xiao et al., 2018b). For each attack, we choose the same 200 adversarial images and their corresponding benign ones. During each trial, one random adversarial-benign pair appears for three seconds and workers are given five minutes to identify the realistic one. Each attack has 600 unique pairs of images and each pair is evaluated by at least 10 unique workers. We restrict biases in this process by allowing each unique user up to 5 rounds of trials and also ignore users who complete the study in less than 30 seconds. In total, 598 unique workers completed at least one round of our user study. For each image, we can then calculate the user preference score as the number of times it is chosen divided by the number of times it is displayed. 0.5 represents that users are unable to distinguish if the image is fake. For cAdv and tAdv, user preferences averages at 0.476 and 0.433 respectively, indicating that workers have a hard time distinguishing them. The user preferences for all attacks are summarized in Table 2 and their comparison with Lp norm is in Table 4 and Table 5.

heading: 6 ATTACKING CAPTIONING MODEL

text:
Our methods are general and can be easily adapted for other learning tasks. As proof of concept, we test our attacks against image captioning task. Image captioning is the task of generating a sequence of word description for an image. The popular architecture for captioning is a Long-ShortTerm-Memory (LSTM) (Hochreiter & Schmidhuber, 1997) based models (Karpathy & Fei-Fei, 2015; Wang et al., 2017). Recently, (Aneja et al., 2018) proposed a convolutional based captioning model for a fast and accurate caption generation. This convolutional based approach does not suffer from the commonly known problems of vanishing gradients and overly confident predictions of LSTM network. Therefore, we choose to attack the current state of the art convolutional captioning model. We randomly selected images from MSCOCO (Lin et al., 2014) for image captioning attack.
Attacking captioning models is harder than attacking classifiers when the goal is to change exactly one word in the benign image’s caption unlike pixel based attacks (Chen et al., 2017; Xu et al., 2019). We show that our attacks are successful and have no visible artifacts even for this challenging task. In Fig. 6, we change the second word of the caption to dog while keeping the rest of the caption the same. This is a challenging targeted attack because, in many untargeted attacks, the resulted captions do not make sense. More examples are in our Appendix.
Adversarial Cross-Entropy Objective for Captioning. Let t be the target caption, w denote the word position of the caption, F for the captioning model, Iv for the victim image and Jadv for the cross-entropy loss
LAcapt = ∑ w Jadv((F(Iv))w, tw) (6)
For cAdv, we give all color hints and optimize to get an adversarial colored image to produce target caption. For tAdv, we add Eqn 6 to Eqn 4 to optimize the image. We select TS as the nearest neighbor of the victim image from the ones in the adversarial target class using ImageNet dataset. We stop our attack once we reach the target caption and the caption does not change in consecutive iterations. Note we do not change the network weights, we only optimize hints and mask (for cAdv) or the victim image (for tAdv) to achieve our target caption.

heading: 7 RELATED WORK

text:
Here we briefly summarize existing unrestricted and semantic adversarial attacks. Xiao et al. (2018b) proposed geometric or spatial distortion of pixels in image to create adversarial examples. They distort the input image by optimizing pixel flow instead of pixel values to generate adversarial examples. While this attack leads to “natural” looking adversarial examples with large L∞ norm, it does not take image semantics into account. Song et al (2018) and Dunn et al. (2019) considered GANs for adversarial attacks. This attack is unrestricted in Lp norm but they are restricted to simple datasets as it involves training GANs, which have been known to be unstable and computationally intensive for complex datasets like ImageNet (Karras et al., 2017; Brock et al., 2018).
Hosseini & Poovendran (2018), changes the hue & saturation of an image randomly to create adversarial examples. It is similar to cAdv as they both involve changing colors, however, their search space is limited to two dimensions and their images are unrealistic, Appendix (Fig. 10). Also, while this method has a non-trivial untargeted attack success rate, it performs extremely poorly for targeted attacks (1.20% success rate in our own experiments on ImageNet). Our work is also related to Joshi et al. (2019) and Qiu et al. (2019), who manipulate images conditioned on face dataset attributes like glasses, beard for their attacks. These work focuses on changing single image visual attribute and are conditionally dependent. Our work focuses on changing visual semantic descriptors to misclassify images and are not conditioned to any semantic attributes.

heading: 8 CONCLUSION

text: Our proposed two novel unrestricted semantic attacks shed light on the role of texture and color fields in influencing DNN’s predictions. They not only consistently fool human subjects but in general are harder to defend against. We hope by presenting our methods, we encourage future studies on unbounded adversarial attacks, better metrics for measuring perturbations, and more sophisticated defenses.

heading: ACKNOWLEDGEMENTS

text: We thank Chaowei Xiao for sharing their code to compare our methods with Xiao et al. (2018b) and helping us setup the user study. We also thank Tianyuan Zhang for providing the AdvRes152 pretrained model. This work was supported by NSF Grant No. 1718221 and ONR MURI Award N00014-16-1-2007.

heading: A APPENDIX

text:
A.1 OTHER DETAILS ON HUMAN STUDY
We also chose BIM (Kurakin et al., 2016) and CW (Carlini & Wagner, 2017) for comparing our perturbations. Since these attacks are known to have low Lp norm, we designed an aggressive version of BIM by relaxing its L∞ bound to match the norm of our attacks. We settled with two aggressive versions of BIM with average L∞ = {0.21, 0.347}, which we refer to as BIM0.21, BIM0.34. The average user preferences for BIM drops drastically from 0.497 to 0.332 when we relax the norm to BIM0.34; the decrease in user preferences for tAdv (0.433 to 0.406) and cAdv (0.476 to 0.437) is not significant. In Fig. 7, we plot a density plot of L∞ vs user preference scores.
A.2 ADDITIONAL RESULTS
Model Resnet50 Dense121 VGG 19 Accuracy 76.15 74.65 74.24
A tta ck Su cc es
s Random Ts 99.67 99.72 96.16 Random Target Ts 99.72 99.89 99.94 Nearest Target Ts 97.99 99.72 99.50 cAdv4 25 hints 99.78 99.83 99.93 cAdv4 50 hints 99.78 99.83 100.00 cAdv4 100 hints 99.44 99.50 99.93
Whitebox target attack success rate. Our attacks are highly successful on different models across all strategies. tAdv results are for α = 250, β = 10−3
and iter= 1.
β α 250 500 750 1000
0 25.00 99.61 98.55 95.92 10−4 99.88 99.61 98.55 95.92 10−3 97.99 99.27 99.66 99.50 10−2 96.26 95.42 96.32 96.59
tAdv ablation study. Whitebox target success rate with nearest target Ts (texture source). In columns, we have increasing texture weight (α) and in rows, we have increasing adversarial cross-entropy weight (β). All attacks
are done on Resnet50.
Table 6: Ablation Studies.

heading: GT k=1 k=2 k=4 k=8k=6

text: Figure 9: Additional qualitative examples for controlling cAdv. We show a comparison of sampling 50 color hints from k clusters with low-entropy. All images are attacked to golf-cart. Even numbered rows visualize our cluster segments, with darker colors representing higher mean entropy and red dots representing the location we sample hints from. Sampling hints across more clusters gives less color variety.


summaries: In this paper, the authors present a method, Learning to Control (LTC), that enables a reinforcement learning agent to learn to read and write external memory. They follow the intuition that human has two degrees of plasticity for memory, which leads to the dense-sparse memory design in this paper. The proposed method can be applied to a few-shot setting.

summaries: The paper proposes a memory network architecture with a sparse memory. Each memory entry contains a key (vector) and a value (class label). The memory is addressed using a policy pi_theta, which selects a single memory entry to be updated at each time step. The policy pi_theta is trained using policy gradient with the reward being the increase in the policy's certainty (measured as entopy). The model is evaluated on an online NER task that mimics the meta-learning setting of http://proceedings.mlr.press/v48/santoro16.html. In that work, the examples are provided in episodes. The labels are renamed at the beginning of each episode (to prevent fitting to the labels). The model has to predict the label of each example in sequence, and the correct label is given after each prediction.

summaries: The authors build on work regarding few-shot learning with memory augmented networks, specifically [Kaiser, et al., ICLR17] where the goal is to learn a memory address mapping such that generalization is achieved by finding the nearest neighbor memory address when predicting the label. For correct predictions, the memory key is updated to include the associated (predicted) address while new memory locations are written for mistakes. Whereas [Kaiser, et al., ICLR17] follows a LRU-like procedure for replacing memory, the current work proposes performing policy-gradient RL where the action space is the memory locations and the reward is reduction of entropy over the memory address assignment distribution over the memory locations. This approach is empirically studied for an RNN approach to NER, specifically considering few-shot learning for NER in the Stanford Task-Oriented Dialogue (STOD) dataset — showing non-negligible improvements over Memory Augmented Networks [Santoro, et al., ICML16] and Matching Networks [Vinyals, et al., NeurIPS16].

abstractText: Humans excel in continuously learning with small data without forgetting how to solve old problems. However, neural networks require large datasets to compute latent representations across different tasks while minimizing a loss function. For example, a natural language understanding (NLU) system will often deal with emerging entities during its deployment as interactions with users in realistic scenarios will generate new and infrequent names, events, and locations. Here, we address this scenario by introducing a RL trainable controller that disentangles the representation learning of a neural encoder from its memory management role. Our proposed solution is straightforward and simple: we train a controller to execute an optimal sequence of read and write operations on an external memory with the goal of leveraging diverse activations from the past and provide accurate predictions. Our approach is named Learning to Control (LTC) and allows few-shot learning with two degrees of memory plasticity. We experimentally show that our system obtains accurate results for few-shot learning of entity recognition in the Stanford Task-Oriented Dialogue dataset.

authors:
D. Bahdanau
K. Cho
Y. Bengio

title: Neural machine translation by jointly learning to align and translate

authors:
M. Eric
L. Krishnan
F. Charette
C.D. Manning

title: Key-value retrieval networks for task-oriented dialogue

venue: Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, 37–49. Saarbrücken, Germany: Association for Computational Linguistics.

authors:
I. Goodfellow
J. Pouget-Abadie
M. Mirza
B. Xu
D. Warde-Farley
S. Ozair
A. Courville
Y. Bengio

title: Generative adversarial nets

venue: Advances in Neural Information Processing Systems 27. 2672–2680.

authors:
A. Graves
G. Wayne
I. Danihelka

title: Neural turing machines

authors:
M. Johnson
M. Schuster
Q.V. Le
M. Krikun
Y. Wu
Z. Chen
N. Thorat
F. Viégas
M. Wattenberg
G. Corrado
M. Hughes
J. Dean

title: Google’s multilingual neural machine translation system: Enabling zero-shot translation

venue: Technical report, Google.

authors:
L. Kaiser
O. Nachum
A. Roy
S. Bengio

title: Learning to remember rare

authors:
J. Kirkpatrick
R. Pascanu
N.C. Rabinowitz
J. Veness
G. Desjardins
A.A. Rusu
K. Milan
J. Quan
T. Ramalho
A. Grabska-Barwinska
D. Hassabis
C. Clopath
D. Kumaran
R. Hadsell

title: Overcoming catastrophic forgetting in neural networks

venue: CoRR.

authors:
G. Koch
R. Zemel
R. Salakhutdinov

title: Siamese neural networks for one-shot image recognition

venue: PhD Thesis.

authors:
T. Kraska
A. Beutel
E.H. Chi
J. Dean
N. Polyzotis

title: The case for learned index structures

authors:
A. Krizhevsky
I. Sutskever
G.E. Hinton

title: Imagenet classification with deep convolutional neural networks

venue: Commun. ACM 60(6).

authors:
T. Luong
H. Pham
C.D. Manning

title: Effective approaches to attention-based neural machine translation

venue: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 1412–1421. Lisbon, Portugal: Association for Computational Linguistics.

authors:
Q. Qian
J. Hu
R. Jin
J. Pei
S. Zhu

title: Distance metric learning using dropout: A structured regularization approach

venue: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, 323–332.

authors:
A. Santoro
S. Bartunov
M. Botvinick
D. Wierstra
T. Lillicrap

title: Meta-learning with memory-augmented neural networks

venue: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, 1842–1850.

authors:
O. Vinyals
C. Blundell
T. Lillicrap
k. kavukcuoglu
D. Wierstra

title: Matching networks for one shot learning

venue: In Advances in Neural Information Processing Systems

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

venue: Mach. Learn. 8(3-4).

authors:
J.T. Wixted
S.D. Goldinger
L.R. Squire
J.R. Kuhn
M.H. Papesh
K.A. Smith
D.M. Treiman
P.N. Steinmetz

title: Coding of episodic memory in the human hippocampus

venue: Proceedings of the National Academy of Sciences.

authors:
S. Zhu
X. Ji
W. Xu
Y. Gong

title: Multi-labelled classification using maximum entropy method

venue: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

heading: 1 MOTIVATION

text:
Today, supervised models have problems incorporating new tasks over time while protecting previously acquired knowledge. This is because these algorithms require that all data is given prior to training. This becomes a problem in the presence of more general scenarios in which new classes emerge during training or data exhibit long-tailed distributions. Hence, classes with large support dominate the learning of gradient-based representations causing the catastrophic forgetting of under-represented classes, Kirkpatrick et al. (2016).
Unlike deep neural networks, humans and other mammals exel in learn incrementally with small data. Biological evidence suggests that the process of acquiring new skills occurs in different brain areas with at least two degrees of plasticity, Wixted et al. (2018): 1) A slow memory that is dense and requires extensive practive. 2) A fast memory that is sparse and stores volatile information. Inspired by these observations, we introduce a trainable controller, Learning to Control (LTC), that learns to interact with both a slow memory (consisting of neural encoders) and a fast memory (consisting of an associative array storing key-value pairs). We experimentally show the advantage of using LTC for a few-shot learning setup. Figure 1 depicts the network architecture of our model.
We present the following contributions:
• We propose a novel architecture that uses a trainale controller to manipulate latent representations in an external memory (Section 3).
• We introduce the use of a reward signal that is proportional to the average reduction of entropy when attending the memory entries. This enables us to propose a reinforcement learning approach that learns a policy based on interactions with the external memory.
• We show the generality of our solution for the few-shot learning of entities in the Stanford Task-Oriented Dialogue dataset.

heading: 2 DENSE AND SPARSE MEMORIES

text:
The proposed architecture considers the use of two types of memories that behave differently during backpropagation. First, we define a memory as follows. Definition 2.1. (Memory) Given a neural model f(x, y, θ) that maps an observation x to the learning task y and is parameterized by θ. Then, the memory of f(x, y, θ) is a collection of co-activations responding similarly to the reocurring patterns associated to the learning task y.
A Neural Network uses most of its memory to remember patterns in the dataset. However, our approach distinguishes between dense and sparse memories. Definition 2.2. (Dense Memory) A dense memory is a type of memory, in which all its learnable parameters θ have the possibility of being updated during training.
We want to preserve latent representations of the input between training steps but a dense memory will likely overwrite them based on frequent global updates. We then need a sparse memory with addressable memory entries and sparse updates. Definition 2.3. (Sparse Memory) Given a model f(x, y, θ), let a(x, y) be a memory operation which returns k updating parameters to incorporate the pair (x, y). Then, a sparse memory is a type of memory that satisfies the sparse updating constraint:
0 < |a(x, y)| |θ|
where |θ| is the total number of trainable parameters. We use the sparse memory introduced in Kaiser et al. (2017) which consists of two arrays of size mem_size for storing keys (K) and values (V ). Each key has a dimensionality of key_dim and each value is a scalar representing a class label. In our experiments, we fixed the number of updating parameters, i.e. keys, to be very small (k = 10) in comparison to mem_size in order to satisfy the sparse updating constraint, k mem_size. The final form of the sparse memory is as follows.
M := (Kmem_size×key_dim, Vmem_size)
The state of a sparse memory changes based on the following memory operations. Definition 2.4. (Memory Operations) Let s be a input embedding that we want to incorporate into M , y be a class label, and i be the memory index of the most similar key to s based on the Cosine similarity.
1. If V [i] 6= y, the write(i, s, y) operation registers a new class with M [i] = 〈s, y〉.
2. If V [i] = y, the update(i, s, y) operation modifies the stored key-based representation K[i] with M [i] = 〈‖K[i] + s‖ , y〉.

heading: 3 LEARNING TO CONTROL

text: The goal of Learning To Control (LTC) is to learn a strategy to manage the entries of a sparse memory. We characterize this decision process with a memory policy πθ which represents the probability of running an memory operation given the current state of the memory. Rathern than following a heuristics (e.g., overwritting in the oldest memory entry as in Kaiser et al. (2017)), learning such policy has the advantage of optimizing the allocation of infrequent classes considering the limited number of memory entries in M . Indeed, πθ transitions M to a new configuration in which its keys are more adapted to store and maintain informative latent vectors. To clarify the connection between the components, let us consider the forward and backward computations involved in LTC.

heading: 3.1 FORWARD COMPUTATION

text:
As illustrated in Figure 1, information flows through the hidden layers of the neural encoder, feeds the controller, and reaches the external memory M . The encoder is a Long-Short Term Memory (LSTM)
neural network that transforms the raw input x into the last hidden state of the following recurrence hi = LSTM(x, hi−1), where s = hi is the state of the environment prior to any memory operation and corresponds to the last hidden state of the LSTM encoder.
Which memory entries should the controller update to incorporate the learning task (s, y)? Our approach is to induce the probablity distribution πθ(i|s) over the memory entries given the state s and sample from there the most likely memory index iπθ
πθ(i|s) = Softmax(s ·Ki) iπθ ∼ argmaxi(πθ(i|s)).
Then, the controller executes the memory operation a(iπθ , s, y) at position iπθ considering the Definition 2.4. During inference, ŷ = V [iπθ ] returns the predicted class of the raw input x .

heading: 3.2 BACKWARD COMPUTATION

text:
The forward step performs an operation a(iπθ , s, y) in the sparse memory and modifies its keys to integrate latent representations. The backward step estimates the objective function J(θ) and computes its gradients with respect to the trainable parameters of the encoder and controller modules.
In the case of the controller, we want to compute the gradient vector ∇θJ(θ) that updates the stochastic policy πθ in a direction that maximizes J(θ) during few-shot learning. For our purposes, each memory operation a returns a reward r that measures the efficacy of this operation to reduce uncertainty about which of the keys in K will lead the transition between memory states. This means going from a state in which no operation took place in memory (Kt−1) to an updated state (Kt). More formaly, our reward function rt(s, a) corresponds to the reduction of entropy H when softly attending over the memory entries before and after executing the operation a.
hKt = s ·Kt attKt = Softmax(hKt)
We use the dot-product attention score Luong, Pham, and Manning (2015) to compute the logits hKt over the memory keys and use Softmax to generate the probability distribution attKt . Zhu et al. (2005) demonstrates that models that show high entropy in their output distributions often fail to discriminate classes because of exhibiting uniform-like distributions and a high degree of uncertainty. Thus, we use the following entropy-based reward signal to encourage the use of few memory entries to modify the sparse memory M .
rt = − ( H(attKt)−H(attKt−1) ) Based on memory operations, we can approximate the objective function from a batch of sampled episodes of length T as the expected value of the cumulative sum of rewards under the policy πθ,
J(θ) = Eπθ [ T∑ t=1 rt(s, a) ] .
The REINFORCE algorithm Williams (1992) allow us to directly take the gradients of the objective function J(θ) as described in Equation 1 and used them to perform backpropagation.
∇θJ(θ) ≈ 1
N N∑ n=1 ( T∑ t=1 ∇θ log πθ(itn|stn) )( T∑ t′=t rt′(s t′ n , a t′ n ) ) (1)

heading: 4 EXPERIMENTS

text: We evaluate our proposed method on the Stanford Task-Oriented Dialogue Dataset (STDO) Eric et al. (2017) which consists of 3, 031 dialogues in the domain of an in-car assistant that provides automatic responses to the requests of a driver considering weather, point-of-interest, and scheduled domains.

heading: 4.1 BASELINES

text: We compare Learning to Control (LTC) with the following baselines: 1) Matching Networks (MN), Vinyals et al. (2016): This algorithm provides one-shot learning capabilities by jointly mapping a small labeled support set and an unlabeled example to its most likely label, and 2) Memory Augmented Neural Networks (MANN), Santoro et al. (2016): This algorithm is designed to provide one-shot learning based on a Neural Turing Machine and a curriculum training regime.

heading: 4.2 FEW-SHOT LEARNING FOR ENTITY RECOGNITION

text:
We study the problem of sequence labeling to find the best label sequence (named entities) for a given input sentence considering an LSTM encoder augmented with an external memory to store hidden representations for each recurrent unit. We use the Stanford Named Entity Recognizer (NER)1 to augment the STDO dataset with 7 classes (Location, Person, Organization, Money, Percent, Date, Time) and an non-entity class. This results in 15, 928 observations, 8 classes, and an average sequence length of 44 words.
We change the format of the STOD dataset to simulate a scenario in which classes obtain incremental support during training. This is done by forming episodes of labeled examples that show a uniform distribution over k classes (e.g., 5-way), so we can track the number of times a particular class was presented to the model during training (e.g., 1-shot learning). This format was proposed by Santoro et al. (2016) to study few-shot learning. The goal of this experiment is to learn with small data, so we study the learning behavior of LTC with an external memory of 1, 000 entries and in relation to two models designed to provide few-shot learning capabilities: Memory Augmented Neural Networks, Santoro et al. (2016) and Matching Networks, Vinyals et al. (2016).
Table 1 shows that LTC provides an average improvement of +5.2 in the STOD dataset with respect to the MN neural model, which is consistently the second most accurate option for both 5-way and 20-way classification and when the number of classes ranges from small (5-way) to large (20-way). Similarly, LTC also shows an average improvement of +10.35% with respect to the MANN model. For the scenario of learning with infrequent classes, LTC outperforms MN when only 1 training observation is presented (1-shot) and 5 and 20 classes are available during training by +4.2% and +3.1%, respectively. The performance advantage in the results can be explained by the extra memory capacity of LTC and represent an opportunity to explore later what is the minimum memory size to continuously learn with small data, an even more challenging problem to study in the future.
1https://nlp.stanford.edu/software/CRF-NER.html

heading: 5 RELATED WORK

text:
The idea of training a controller that interacts with a memory to incrementally allocate distributional representations builds upon a wide range of research in machine learning and memory augmented algorithms.
Kraska et al. (2018) introduced the concept of learned indices structures as a learning architecture that predicts the position or existence of records. Their main goal is to prevent too many collisions from being mapped to the same position inside the hash index. This is because collisions often cause memory overhead when traversing a linked list or require the allocation of additional memory for storing more records. They approach this problem as supervised learning of the cumulative distribution function of hash keys, which leads to minimizing the number of collisions. In contrast, we propose the use of an associative array as an indexed structure that supports generalization, so the collision of multiple observations to a similar hash key if adequate for predicting the class of similar input data. While the idea of learning hash functions as neural networks is not new, existing work mainly focused on learning a better hash-function to map observations into low-dimensional embeddings for similarity search assuming a fixed data distribution Qian et al. (2014). To our knowledge, it has not been explored yet if it is possible to learn a hash index according to a data distribution in which few training observations are presented per class during training.
Deep Neural Networks are models that solve classification problems with non-linearly separable classes. These are shown to be good for problems such as object detection Krizhevsky, Sutskever, and Hinton (2017), language translation Johnson et al. (2016) and image generation Goodfellow et al. (2014). Recently, recurrent neural networks (RNN) have been studied to manage the states between time steps using an attention mechanism that addresses similar content Bahdanau, Cho, and Bengio (2014). Some similar approaches have also studied the problem of few-shot learning Santoro et al. (2016)Koch, Zemel, and Salakhutdinov (2015)Vinyals et al. (2016), a representative example is Matching Networks Vinyals et al. (2016), which uses an attention mechanism to augment neural networks for set-to-set learning.
A relevant research topic to this work is the idea of memory augmented networks that extend its capabilities by coupling an external memory. For example, Kaiser et al. (2017) also proposes a key-value memory, but with no controller mechanism for adequate training of few-shot learning tasks. Also, Neural Turing Machines (NTM) Graves, Wayne, and Danihelka (2014) are differentiable architectures allowing efficient training with gradient descend and showing important properties for associative recall for learning different sequential patterns. Although they have important properties for one-shot learning given their sequential memory management, the supervised nature of its training still shows issues related to catastrophic forgetting.

heading: 6 CONCLUSIONS

text: Learning new tasks without forgetting the previously ones references the plasticity of our own brain to retain previously acquired knowledge. In this work, we show a learning system that stores information about named entities with two levels of representation: dense and sparse. While a dense memory captures features that are shared across tasks, sparse memory incorporates embedding vectors and prevents them from being overwritten by global updates during backpropagation. To do this, we rely on the sparse structure of associative arrays to locally update a small set of memory entries and on the use of a trainable controller that manages the transition of the sparse memory to a state of low entropy that also addresses the learning task. This enables a model to incrementally learn in the presence of a few observations per class. We experimentally show that our system obtains accurate results for few-shot learning of entities in the Stanford Task-Oriented Dialogue dataset in comparison with state-of-the-art few shot learning algorithms (Memory Augmented Neural Networks and Matching Networks).