An intriguing property of deep neural networks is the existence of adversarial examples, which can transfer among different architectures. These transferable adversarial examples may severely hinder deep neural network-based applications. Previous works mostly study the transferability using small scale datasets. In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels. We study both non-targeted and targeted adversarial examples, and show that while transferable non-targeted adversarial examples are easy to find, targeted adversarial examples generated using existing approaches almost never transfer with their target labels. Therefore, we propose novel ensemble-based approaches to generating transferable adversarial examples. Using such approaches, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels for the first time. We also present some geometric studies to help understanding the transferable adversarial examples. Finally, we show that the adversarial examples generated using ensemble-based approaches can successfully attack Clarifai.com, which is a black-box image classification system.
An intriguing property of deep neural networks is the existence of adversarial examples, which can transfer among different architectures. These transferable adversarial examples may severely hinder deep neural network-based applications. Previous works mostly study the transferability using small scale datasets. In this work, we are the first to conduct an extensive study of the transferability over large models and a large scale dataset, and we are also the first to study the transferability of targeted adversarial examples with their target labels. We study both non-targeted and targeted adversarial examples, and show that while transferable non-targeted adversarial examples are easy to find, targeted adversarial examples generated using existing approaches almost never transfer with their target labels. Therefore, we propose novel ensemble-based approaches to generating transferable adversarial examples. Using such approaches, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels for the first time. We also present some geometric studies to help understanding the transferable adversarial examples. Finally, we show that the adversarial examples generated using ensemble-based approaches can successfully attack Clarifai.com, which is a black-box image classification system.
1 INTRODUCTION
Recent research has demonstrated that for a deep architecture, it is easy to generate adversarial examples, which are close to the original ones but are misclassified by the deep architecture (Szegedy et al. (2013); Goodfellow et al. (2014)). The existence of such adversarial examples may have severe consequences, which hinders vision-understanding-based applications, such as autonomous driving. Most of these studies require explicit knowledge of the underlying models. It remains an open question how to efficiently find adversarial examples for a black-box model.
Several works have demonstrated that some adversarial examples generated for one model may also be misclassified by another model. Such a property is referred to as transferability, which can be leveraged to perform black-box attacks. This property has been exploited by constructing a substitute of the black-box model, and generating adversarial instances against the substitute to attack the black-box system (Papernot et al. (2016a;b)). However, so far, transferability is mostly examined over small datasets, such as MNIST (LeCun et al. (1998)) and CIFAR-10 (Krizhevsky & Hinton (2009)). It has yet to be better understood transferability over large scale datasets, such as ImageNet (Russakovsky et al. (2015)).
In this work, we are the first to conduct an extensive study of the transferability of different adversarial instance generation strategies applied to different state-of-the-art models trained over a large scale dataset. In particular, we study two types of adversarial examples: (1) non-targeted adversarial examples, which can be misclassified by a network, regardless of what the misclassified labels may be; and (2) targeted adversarial examples, which can be classified by a network as a target label. We examine several existing approaches searching for adversarial examples based on a single model. While non-targeted adversarial examples are more likely to transfer, we observe few targeted adversarial examples that are able to transfer with their target labels.
∗Work is done while visiting UC Berkeley.
We further propose a novel strategy to generate transferable adversarial images using an ensemble of multiple models. In our evaluation, we observe that this new strategy can generate non-targeted adversarial instances with better transferability than other methods examined in this work. Also, for the first time, we observe a large proportion of targeted adversarial examples that are able to transfer with their target labels.
We study geometric properties of the models in our evaluation. In particular, we show that the gradient directions of different models are orthogonal to each other. We also show that decision boundaries of different models align well with each other, which partially illustrates why adversarial examples can transfer.
Last, we study whether generated adversarial images can attack Clarifai.com, a commercial company providing state-of-the-art image classification services. We have no knowledge about the training dataset and the types of models used by Clarifai.com; meanwhile, the label set of Clarifai.com is quite different from ImageNet’s. We show that even in this case, both non-targeted and targeted adversarial images transfer to Clarifai.com. This is the first work documenting the success of generating both non-targeted and targeted adversarial examples for a black-box state-of-the-art online image classification system, whose model and training dataset are unknown to the attacker.
Contributions and organization. We summarize our main contributions as follows:
• For ImageNet models, we show that while existing approaches are effective to generate non-targeted transferable adversarial examples (Section 3), only few targeted adversarial examples generated by existing methods can transfer (Section 4).
• We propose novel ensemble-based approaches to generate adversarial examples (Section 5). Our approaches enable a large portion of targeted adversarial examples to transfer among multiple models for the first time.
• We are the first to present that targeted adversarial examples generated for models trained on ImageNet can transfer to a black-box system, i.e., Clarifai.com, whose model, training data, and label set is unknown to us (Section 7). In particular, Clarifai.com’s label set is very different from ImageNet’s.
• We conduct the first analysis of geometric properties for large models trained over ImageNet (Section 6), and the results reveal several interesting findings, such as the gradient directions of different models are orthogonal to each other.
In the following, we first discuss related work, and then present the background knowledge and experiment setup in Section 2. Then we present each of our experiments and conclusions in the corresponding section as mentioned above.
Related work. Transferability of adversarial examples was first examined by Szegedy et al. (2013), which studied the transferability (1) between different models trained over the same dataset; and (2) between the same or different model trained over disjoint subsets of a dataset; However, Szegedy et al. (2013) only studied MNIST.
The study of transferability was followed by Goodfellow et al. (2014), which attributed the phenomenon of transferability to the reason that the adversarial perturbation is highly aligned with the weight vector of the model. Again, this hypothesis was tested using MNIST and CIFAR-10 datasets. We show that this is not the case for models trained over ImageNet.
Papernot et al. (2016a;b) examined constructing a substitute model to attack a black-box target model. To train the substitute model, they developed a technique that synthesizes a training set and annotates it by querying the target model for labels. They demonstrate that using this approach, black-box attacks are feasible towards machine learning services hosted by Amazon, Google, and MetaMind. Further, Papernot et al. (2016a) studied the transferability between deep neural networks and other models such as decision tree, kNN, etc.
Our work differs from Papernot et al. (2016a;b) in three aspects. First, in these works, only the model and the training process are a black box, but the training set and the test set are controlled by the attacker; in contrast, we attack Clarifai.com, whose model, training data, training process, and even the test label set are unknown to the attacker. Second, the datasets studied in these works are small
scale, i.e., MNIST and GTSRB (Stallkamp et al. (2012)); in our work, we study the transferability over larger models and a larger dataset, i.e., ImageNet. Third, to attack black-box machine learning systems, we do not query the systems for constructing the substitute model ourselves.
In a concurrent and independent work, Moosavi-Dezfooli et al. (2016) showed the existence of a universal perturbation for each model, which can transfer across different images. They also show that the adversarial images generated using these universal perturbations can transfer across different models on ImageNet. However, they only examine the non-targeted transferability, while our work studies both non-targeted and targeted transferability over ImageNet.
2 ADVERSARIAL DEEP LEARNING AND TRANSFERABILITY
2.1 THE ADVERSARIAL DEEP LEARNING PROBLEM
We assume a classifier fθ(x) outputs a category (or a label) as the prediction. Given an original image x, with ground truth label y, the adversarial deep learning problem is to seek for adversarial examples for the classifier fθ(x). Specifically, we consider two classes of adversarial examples. A non-targeted adversarial example x? is an instance that is close to x, in which case x? should have the same ground truth as x, while fθ(x?) 6= y. For the problem to be non-trivial, we assume fθ(x) = y without loss of generality. A targeted adversarial example x? is close to x and satisfies fθ(x ?) = y?, where y? is a target label specified by the adversary, and y? 6= y.
2.2 APPROACHES FOR GENERATING ADVERSARIAL EXAMPLES
In this work, we consider three classes of approaches for generating adversarial examples: optimization-based approaches, fast gradient approaches, and fast gradient sign approaches. Each class has non-targeted and targeted versions respectively.
2.2.1 APPROACHES FOR GENERATING NON-TARGETED ADVERSARIAL EXAMPLES
Formally, given an image x with ground truth y = fθ(x), searching for a non-targeted adversarial example can be modeled as searching for an instance x? to satisfy the following constraints:
fθ(x ?) 6= y (1) d(x, x?) ≤ B (2) where d(·, ·) is a metric to quantify the distance between an original image and its adversarial counterpart, and B, called distortion, is an upper bound placed on this distance. Without loss of generality, we consider model f is composed of a network Jθ(x), which outputs the probability for each category, so that f outputs the category with the highest probability.
Optimization-based approach. One approach is to approximate the solution to the following optimization problem:
argminx?λd(x, x ?)− `(1y, Jθ(x?)) (3)
where 1y is the one-hot encoding of the ground truth label y, ` is a loss function to measure the distance between the prediction and the ground truth, and λ is a constant to balance constraints (2) and (1), which is empirically determined. Here, loss function ` is used to approximate constraint (1), and its choice can affect the effectiveness of searching for an adversarial example. In this work, we choose `(u, v) = log (1− u · v), which is shown to be effective by Carlini & Wagner (2016).
Fast gradient sign (FGS). Goodfellow et al. (2014) proposed the fast gradient sign (FGS) method so that the gradient needs be computed only once to generate an adversarial example. FGS can be used to generate adversarial images to meet the L∞ norm bound. Formally, non-targeted adversarial examples are constructed as
x? ← clip(x+Bsgn(∇x`(1y, Jθ(x)))) Here, clip(x) is used to clip each dimension of x to the range of pixel values, i.e., [0, 255] in this work. We make a slight variation to choose `(u, v) = log (1− u · v), which is the same as used in the optimization-based approach.
Fast gradient (FG). The fast gradient approach (FG) is similar to FGS, but instead of moving along the gradient sign direction, FG moves along the gradient direction. In particular, we have
x? ← clip(x+B ∇x`(1y, Jθ(x)) ||∇x`(1y, Jθ(x))|| ))
Here, we assume the distance metric in constraint (2), d(x, x?) = ||x − x?|| is a norm of x − x?. The term sgn(∇x`) in FGS is replaced by ∇x`||∇x`|| to meet this distance constraint.
We call both FGS and FG fast gradient-based approaches.
2.2.2 APPROACHES FOR GENERATING TARGETED ADVERSARIAL EXAMPLES
A targeted adversarial image x? is similar to a non-targeted one, but constraint (1) is replaced by fθ(x
?) = y? (4) where y? is the target label given by the adversary. For the optimization-based approach, we approximate the solution by solving the following dual objective:
argminx?λd(x, x ?) + `′(1y? , Jθ(x ?)) (5) In this work, we choose the standard cross entropy loss `′(u, v) = − ∑ i ui log vi.
For FGS and FG, we construct adversarial examples as follows: x? ← clip(x−Bsgn(∇x`′(1y? , Jθ(x)))) (FGS)
x? ← clip(x−B ∇x` ′(1y? , Jθ(x))
||∇x`′(1y? , Jθ(x))|| ) (FG)
where `′ is the same as the one used for the optimization-based approach.
2.3 EVALUATION METHODOLOGY
For the rest of the paper, we focus on examining the transferability among state-of-the-art models trained over ImageNet (Russakovsky et al. (2015)). In this section, we detail the models to be examined, the dataset to be evaluated, and the measurements to be used.
Models. We examine five networks, ResNet-50, ResNet-101, ResNet-152 (He et al. (2015))1, GoogLeNet (Szegedy et al. (2014))2, and VGG-16 (Simonyan & Zisserman (2014))3. We retrieve the pre-trained models for each network online. The performance of these models on the ILSVRC 2012 (Russakovsky et al. (2015)) validation set can be found in our online technical report: Liu et al. (2016). We choose these models to study the transferability between homogeneous architectures (i.e., ResNet models) and heterogeneous architectures.
Dataset. It is less meaningful to examine the transferability of an adversarial image between two models which cannot classify the original image correctly. Therefore, from the ILSVRC 2012 validation set, we randomly choose 100 images, which can be classified correctly by all five models in our examination. These 100 images form our test set. To perform targeted attacks, we manually choose a target label for each image, so that its semantics is far from the ground truth. The images and target labels in our evaluation can be found on website4.
Measuring transferability. Given two models, we measure the non-targeted transferability by computing the percentage of the adversarial examples generated for one model that can be classified correctly for the other. We refer to this percentage as accuracy. A lower accuracy means better non-targeted transferability. We measure the targeted transferability by computing the percentage of the adversarial examples generated for one model that are classified as the target label by the other model. We refer to this percentage as matching rate. A higher matching rate means better targeted transferability. For clarity, the reported results are only based on top-1 accuracy. Top-5 accuracy’s counterparts can be found in our online technical report: Liu et al. (2016).
1 https://github.com/KaimingHe/deep-residual-networks 2 https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet 3 https://gist.github.com/ksimonyan/211839e770f7b538e2d8 4 https://github.com/sunblaze-ucb/transferability-advdnn-pub
Distortion. Besides transferability, another important factor is the distortion between adversarial images and the original ones. We measure the distortion by root mean square deviation, i.e., RMSD, which is computed as d(x?, x) = √∑ i(x ? i − xi)2/N , where x? and x are the vector representations of an adversarial image and the original one respectively, N is the dimensionality of x and x?, and xi denotes the pixel value of the i-th dimension of x, within range [0, 255], and similar for x?i .
3 NON-TARGETED ADVERSARIAL EXAMPLES
In this section, we examine different approaches for generating non-targeted adversarial images.
3.1 OPTIMIZATION-BASED APPROACH
To apply the optimization-based approach for a single model, we initialize x? to be x and use Adam Optimizer (Kingma & Ba (2014)) to optimize Objective (3) . We find that we can tune the RMSD by adjusting the learning rate of Adam and λ. We find that, for each model, we can use a small learning rate to generate adversarial images with small RMSD, i.e. < 2, with any λ. In fact, we find that when initializing x? with x, Adam Optimizer will search for an adversarial example around x, even when we set λ to be 0, i.e., not restricting the distance between x? and x. Therefore, we set λ to be 0 for all experiments using optimization-based approaches throughout the paper. Although these adversarial examples with small distortions can successfully fool the target model, however, they cannot transfer well to other models (details can be found in our online technical report: Liu et al. (2016)).
We increase the learning rate to allow the optimization algorithm to search for adversarial images with larger distortion. In particular, we set the learning rate to be 4. We run Adam Optimizer for 100 iterations to generate the adversarial images. We observe that the loss converges after 100 iterations. An alternative optimization-based approach leading to similar results can be found in our online technical report: Liu et al. (2016).
Non-targeted adversarial examples transfer. We generate non-targeted adversarial examples on one network, but evaluate them on another, and Table 1 Panel A presents the results. From the table, we can observe that
• The diagonal contains all 0 values. This says that all adversarial images generated for one model can mislead the same model.
• A large proportion of non-targeted adversarial images generated for one model using the optimization-based approach can transfer to another.
• Although the three ResNet models share similar architectures which differ only in the hyperparameters, adversarial examples generated against a ResNet model do not necessarily transfer to another ResNet model better than other non-ResNet models. For example, the adversarial examples generated for VGG-16 have lower accuracy on ResNet-50 than those generated for ResNet-152 or ResNet-101.
3.2 FAST GRADIENT-BASED APPROACHES
We then examine the effectiveness of fast gradient-based approaches. A good property of fast gradient-based approaches is that all generated adversarial examples lie in a 1-D subspace. Therefore, we can easily approximate the minimal distortion in this subspace of transferable adversarial examples between two models. In the following, we first control the RMSD to study fast gradientbased approaches’ effectiveness. Second, we study the transferable minimal distortions of fast gradient-based approaches.
3.2.1 EFFECTIVENESS AND TRANSFERABILITY OF THE FAST GRADIENT-BASED APPROACHES
Since the distortion B and the RMSD of the generated adversarial images are highly correlated, we can choose this hyperparameter B to generate adversarial images with a given RMSD. In Table 1
Panel B, we generate adversarial images using FG such that the average RMSD is almost the same as those generated using the optimization-based approach. We observe that the diagonal values in the table are all positive, which means that FG cannot fully mislead the models. A potential reason is that, FG can be viewed as approximating the optimization, but is tailored for speed over accuracy.
On the other hand, the values of non-diagonal cells in the table, which correspond to the accuracies of adversarial images generated for one model but evaluated on another, are comparable with or less than their counterparts in the optimization-based approach. This shows that non-targeted adversarial examples generated by FG exhibit transferability as well.
We also evaluate FGS, but the transferability of the generated images is worse than the ones generated using either FG or optimization-based approaches. The results can be found in our online technical report: Liu et al. (2016). It shows that when RMSD is around 23, the accuracies of the adversarial images generated by FGS is greater than their counterparts for FG. We hypothesize the reason why transferability of FGS is worse to this fact.
3.2.2 ADVERSARIAL IMAGES WITH MINIMAL TRANSFERABLE RMSD
For an image x and two models M1,M2, we can approximate the minimal distortion B along a direction δ, such that xB = x+Bδ generated for M1 is adversarial for both M1 and M2. Here δ is the direction, i.e., sgn(∇x`) for FGS, and∇x`/||∇x`|| for FG. We refer to the minimal transferable RMSD from M1 to M2 using FG (or FGS) as the RMSD of a transferable adversarial example xB with the minimal transferable distortion B from M1 to M2 using FG (or FGS). The minimal transferable RMSD can illustrate the tradeoff between distortion and transferability.
In the following, we approximate the minimal transferable RMSD through a linear search by sampling B every 0.1 step. We choose the linear-search method rather than binary-search method to determine the minimal transferable RMSD because the adversarial images generated from an original image may come from multiple intervals. The experiment can be found in our online technical report: Liu et al. (2016).
Minimal transferable RMSD using FG and FGS. Figure 1 plots the cumulative distribution function (CDF) of the minimal transferable RMSD from VGG-16 to ResNet-152 using non-targeted FG (Figure 1a) and FGS (Figure 1b). From the figures, we observe that both FG and FGS can find 100% transferable adversarial images with RMSD less than 80.91 and 86.56 respectively. Further,
the FG method can generate transferable attacks with smaller RMSD than FGS. A potential reason is that while FGS minimizes the distortion’s L∞ norm, FG minimizes its L2 norm, which is proportional to RMSD.
3.3 COMPARISON WITH RANDOM PERTURBATIONS
We also evaluate the test accuracy when we add a Gaussian noise to the 100 images in our test set. The concrete results can be found in our online technical report: Liu et al. (2016), where we show the conclusion that the “transferability” of this approach is significantly worse than either optimization-based approaches or fast gradient-based approaches.
4 TARGETED ADVERSARIAL EXAMPLES
In this section, we examine the transferability of targeted adversarial images. Table 2 presents the results for using optimization-based approach. We observe that (1) the prediction of targeted adversarial images can match the target labels when evaluated on the same model that is used to generate the adversarial examples; but (2) the targeted adversarial images can be rarely predicted as the target labels by a different model. We call the latter that the target labels do not transfer. Even when we increase the distortion, we still do not observe improvements on making target label transfer. Some results can be found in our online technical report: Liu et al. (2016). Even if we compute the matching rate based on top-5 accuracy, the highest matching rate is only 10%. The results can be found in our online technical report: Liu et al. (2016).
We also examine the targeted adversarial images generated by fast gradient-based approaches, and we observe that the target labels do not transfer as well. The results can be found in our online technical report: Liu et al. (2016). In fact, most targeted adversarial images cannot mislead the model, for which the adversarial images are generated, to predict the target labels, regardless of how large the distortion is used. We attribute it to the fact that the fast gradient-based approaches only
search for attacks in a 1-D subspace. In this subspace, the total possible predictions may contain a small subset of all labels, which usually does not contain the target label. In Section 6, we study decision boundaries regarding this issue.
We also evaluate the matching rate of images added with Gaussian noise, as described in Section 3.3. However, we observe that the matching rate of any of the 5 models is 0%. Therefore, we conclude that by adding Gaussian noise, the attacker cannot generate successful targeted adversarial examples at all, let alone targeted transferability.
5 ENSEMBLE-BASED APPROACHES
We hypothesize that if an adversarial image remains adversarial for multiple models, then it is more likely to transfer to other models as well. We develop techniques to generate adversarial images for multiple models. The basic idea is to generate adversarial images for the ensemble of the models. Formally, given k white-box models with softmax outputs being J1, ..., Jk, an original image x, and its ground truth y, the ensemble-based approach solves the following optimization problem (for targeted attack):
argminx? − log ( ( k∑ i=1 αiJi(x ?)) · 1y? ) + λd(x, x?) (6)
where y? is the target label specified by the adversary, ∑ αiJi(x
?) is the ensemble model, and αi are the ensemble weights, ∑k i=1 αi = 1. Note that (6) is the targeted objective. The non-targeted counterpart can be derived similarly. In doing so, we hope the generated adversarial images remain adversarial for an additional black-box model Jk+1.
We evaluate the effectiveness of the ensemble-based approach. For each of the five models, we treat it as the black-box model to attack, and generate adversarial images for the ensemble of the rest four, which is considered as white-box. We evaluate the generated adversarial images over all five models. Throughout the rest of the paper, we refer to the approaches evaluated in Section 3 and 4 as the approaches using a single model, and to the ensemble-based approaches discussed in this section as the approaches using an ensemble model.
Optimization-based approach. We use Adam to optimize the objective (6) with equal ensemble weights across all models in the ensemble to generate targeted adversarial examples. In particular, we set the learning rate of Adam to be 8 for each model. In each iteration, we compute the Adam update for each model, sum up the four updates, and add the aggregation onto the image. We run 100 iterations of updates, and we observe that the loss converges after 100 iterations. By doing so, for the first time, we observe a large proportion of the targeted adversarial images whose target labels can transfer. The results are presented in Table 3. We observe that not all targeted adversarial images can be misclassified to the target labels by the models used in the ensemble. This suggests that while searching for an adversarial example for the ensemble model, there is no direct supervision to mislead any individual model in the ensemble to predict the target label. Further, from the diagonal numbers of the table, we observe that the transferability to ResNet models is better than to VGG-16 or GoogLeNet, when adversarial examples are generated against all models except the target model.
We also evaluate non-targeted adversarial images generated by the ensemble-based approach. We observe that the generated adversarial images have almost perfect transferability. We use the same procedure as for the targeted version, except the objective to generate the adversarial images. We evaluate the generated adversarial images over all models. The results are presented in Table 4. The generated adversarial images all have RMSDs around 17, which are lower than 22 to 23 of the optimization-based approach using a single model (See Table 1 for comparison). When the adversarial images are evaluated over models which are not used to generate the attack, the accuracy is no greater than 6%. For a reference, the corresponding accuracies for all approaches evaluated in Section 3 using one single model are at least 12%. Our experiments demonstrate that the ensemblebased approaches can generate almost perfectly transferable adversarial images.
Fast gradient-based approach. The results for non-targeted fast gradient-based approaches applied to the ensemble can be found in our online technical report: Liu et al. (2016). We observe that the diagonal values are not zero, which is the same as we observed in the results for FG and
FGS applied to a single model. We hypothesize a potential reason is that the gradient directions of different models in the ensemble are orthogonal to each other, as we will illustrate in Section 6. In this case, the gradient direction of the ensemble is almost orthogonal to the one of each model in the ensemble. Therefore searching along this direction may require large distortion to reach adversarial examples.
For targeted adversarial examples generated using FG and FGS based on an ensemble model, their transferability is no better than the ones generated using a single model. The results can be found in our online technical report: Liu et al. (2016). We hypothesize the same reason to explain this: there are only few possible target labels in total in the 1-D subspace.
6 GEOMETRIC PROPERTIES OF DIFFERENT MODELS
In this section, we show some geometric properties of the models to try to better understand transferable adversarial examples. Prior works also try to understand the geometic properties of adversarial examples theoretically (Fawzi et al. (2016)) or empirically (Goodfellow et al. (2014)). In this work, we examine large models trained over a large dataset with 1000 labels, whose geometric properties are never examined before. This allows us to make new observations to better understand the models and their adversarial examples.
The gradient directions of different models in our evaluation are almost orthogonal to each other. We study whether the adversarial directions of different models align with each other. We calculate cosine value of the angle between gradient directions of different models, and the results can be found in our online technical report: Liu et al. (2016). We observe that all non-diagonal values are close to 0, which indicates that for most images, their gradient directions with respect to different models are orthogonal to each other.
Decision boundaries of the non-targeted approaches using a single model. We study the decision boundary of different models to understand why adversarial examples transfer. We choose two
normalized orthogonal directions δ1, δ2, one being the gradient direction of VGG-16 and the other being randomly chosen. Each point (u, v) in this 2-D plane corresponds to the image x+uδ1+vδ2, where x is the pixel value vector of the original image. For each model, we plot the label of the image corresponding to each point, and get Figure 3 using the image in Figure 2.
We can observe that for all models, the region that each model can predict the image correctly is limited to the central area. Also, along the gradient direction, the classifiers are soon misled. One interesting finding is that along this gradient direction, the first misclassified label for the three ResNet models (corresponding to the light green region) is the label “orange”. A more detailed study can be found in our online technical report: Liu et al. (2016). When we look at the zoomout figures, however, the labels of images that are far away from the original one are different for different models, even among ResNet models.
On the other hand, in Table 5, we show the total number of regions in each plane. In fact, for each plane, there are at most 21 different regions in all planes. Compared with the 1,000 total categories in ImageNet, this is only 2.1% of all categories. That means, for all other 97.9% labels, no targeted adversarial example exists in each plane. Such a phenomenon partially explains why fast gradientbased approaches can hardly find targeted adversarial images.
Further, in Figure 4, we draw the decision boundaries of all models on the same plane as described above. We can observe that
• The boundaries align with each other very well. This partially explains why non-targeted adversarial images can transfer among models.
• The boundary diameters along the gradient direction is less than the ones along the random direction. A potential reason is that moving a variable along its gradient direction can change the loss function (i.e., the probability of the ground truth label) significantly. Therefore along the gradient direction it will take fewer steps to move out of the ground truth region than a random direction.
• An interesting finding is that even though we move left along the x-axis, which is equivalent to maximizing the ground truth’s prediction probability, it also reaches the boundary much sooner than moving along a random direction. We attribute this to the non-linearity of the loss function: when the distortion is larger, the gradient direction also changes dramatically. In this case, moving along the original gradient direction no longer increases the probability to predict the ground truth label (details can be found in our online technical report: Liu et al. (2016)).
• As for VGG-16 model, there is a small hole within the region corresponding to the ground truth. This may partially explain why non-targeted adversarial images with small distortion exist, but do not transfer well. This hole does not exist in other models’ decision planes. In this case, non-targeted adversarial images in this hole do not transfer.
Decision boundaries of the targeted ensemble-based approaches. In addition, we choose the targeted adversarial direction of the ensemble of all models except ResNet-101 and a random orthogonal direction, and we plot decision boundaries on the plane spanned by these two direction vectors in Figure 5. We observe that the regions of images, which are predicted as the target label, align well for the four models in the ensemble. However, for the model not used to generate the adversarial image, i.e., ResNet-101, it also has a non-empty region such that the prediction is successfully misled to the target label, although the area is much smaller. Meanwhile, the region within each closed curve of the models almost has the same center.
7 REAL WORLD EXAMPLE: ADVERSARIAL EXAMPLES FOR CLARIFAI.COM
Clarifai.com is a commercial company providing state-of-the-art image classification services. We have no knowledge about the dataset and types of models used behind Clarifai.com, except that we have black-box access to the services. The labels returned from Clarifai.com are also different from the categories in ILSVRC 2012. We submit all 100 original images to Clarifai.com and the returned labels are correct based on a subjective measure.
We also submit 400 adversarial images in total, where 200 of them are targeted adversarial examples, and the rest 200 are non-targeted ones. As for the 200 targeted adversarial images, 100 of them are generated using the optimization-based approach based on VGG-16 (the same ones evaluated in Table 2), and the rest 100 are generated using the optimization-based approach based on an ensemble of all models except ResNet-152 (the same ones evaluated in Table 3). The 200 nontargeted adversarial examples are generated similarly (the same ones evaluated in Table 1 and 4).
For non-targeted adversarial examples, we observe that for both the ones generated using VGG-16 and those generated using the ensemble, most of them can transfer to Clarifai.com.
More importantly, a large proportion of our targeted adversarial examples are misclassified by Clarifai.com as well. We observe that 57% of the targeted adversarial examples generated using VGG-16, and 76% of the ones generated using the ensemble can mislead Clarifai.com to predict labels irrelevant to the ground truth.
Further, our experiment shows that for targeted adversarial examples, 18% of those generated using the ensemble model can be predicted as labels close to the target label by Clarifai.com. The corresponding number for the targeted adversarial examples generated using VGG-16 is 2%. Considering that in the case of attacking Clarifai.com, the labels given by the target model are different from those given by our models, it is fairly surprising to see that when using the ensemble-based approach, there is still a considerable proportion of our targeted adversarial examples that can mislead this black-box model to make predictions semantically similar to our target labels. All these numbers are computed based on a subjective measure, and we include some examples in Table 6. More examples can be found in our online technical report: Liu et al. (2016).
original image
true label
Clarifai.com results of
original image
target label
targeted adversarial example
Clarifai.com results of targeted adversarial example
viaduct
bridge, sight, arch, river, sky
window screen
window, wall, old,
decoration, design
hip, rose hip,
rosehip
fruit, fall, food, little,
wildlife
stupa, tope
Buddha, gold,
temple, celebration,
artistic
dogsled, dog sled, dog
sleigh
group together, four,
sledge, sled,
enjoyment
hip, rose hip,
rosehip
cherry, branch, fruit, food, season
8 CONCLUSION
In this work, we are the first to conduct an extensive study of the transferability of both non-targeted and targeted adversarial examples generated using different approaches over large models and a large scale dataset. Our results confirm that the transferability for non-targeted adversarial examples are prominent even for large models and a large scale dataset. On the other hand, we find that it is hard to use existing approaches to generate targeted adversarial examples whose target labels can transfer. We develop novel ensemble-based approaches, and demonstrate that they can generate transferable targeted adversarial examples with a high success rate. Meanwhile, these new approaches exhibit better performance on generating non-targeted transferable adversarial examples than previous work. We also show that both non-targeted and targeted adversarial examples generated using our new approaches can successfully attack Clarifai.com, which is a black-box image classification system. Furthermore, we study some geometric properties to better understand the transferable adversarial examples.
ACKNOWLEDGMENTS
This material is in part based upon work supported by the National Science Foundation under Grant No. TWC-1409915. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

This paper present an experimental study of the robustness of state-of-the-art CNNs to different types of "attacks" in the context of image classication. Specifically, an attack aims to fool the classification system with a specially corrupted image, i.e. making it misclassify the image as (1) any wrong class (non-targeted attack) or (2) a target class, chosen in advance by the attacker (targeted attack). For instance, the attacker could corrupt an image of an ostrich in such a way that it would be classified as a megalith. Even though the attacker's agenda is not so clear in this example, it is still interesting to study the weaknesses of current systems in view of (1) improving them in general and (2) actual risks with e.g. autonomous vehicles.
The paper is mostly experimental. In short, it compares different strategies (already published in previous papers) for all popular networks  (VGG, GoogLeNet, ResNet-50/101/152) and the two aforementionned types of attacks. The experiments are well conducted and clearly exposed. A convincing point is that attacks are also conducted on "clarifai.com" which is a black-box classification system. Some analysis and insightful explanations are also provided to help understanding why CNNs are prone to such attacks (Section 6).
To sum up, the main findings are that non-targeted attacks are easy to perform, even on a black-box system. Non-targeted attacks are more difficult to realize with existing schemes, but the authors propose a new approach for that that vastly improves over existing attacks (even though it's still far from perfect: ~20% success rate on clarifai.com versus 2% with previous schemes).
Arguably, The paper still has some weaknesses:
 - The authors are treating the 3 ResNet-based networks as different, yet they are obviously clearly correlated. See Table 7 for instance. This is naturally expected because their architecture is similar (only their depth varies). Hence, it does not sound very fair to state that "One interesting finding is that [...] the first misclassified label (non-targeted) is the same for all models except VGG-16 and GoogLeNet.", i.e., the three ResNet-based networks.
 - A subjective measure is employed to evaluate the effectiveness of the attacks on the black box system. While this is for a good reason (clarifai.com returns image labels that are different from ImageNet), it is not certain that the reported numbers are fair (even though the qualitative results look convincing).
 - The novelty of the proposed approach (optimizing an ensemble of network instead of a single network) is limited. However, this was not really the point of the paper, and it is effective, so it seems ok overall.
 - The paper is quite long. This is expected because it is an extensive evaluation study, but still. I suggest the authors prune some near-duplicate content (e.g. Section 2.3 has a high overlap with Section 1, etc.).
 - The paper would benefit from additional discussions with the recent and related work of Fawzi et al (NIPS'16) in Section 6. Indeed the work of Fawzi et al. is mostly theoretical and well aligned with the experimental findings and observations (in particular in Section 6).
To conclude, I think that this paper is somewhat useful for the community and could help to further improve existing architectures, as well as better assess their flaws and weaknesses.


NEURAL CODE COMPLETION
Code completion, an essential part of modern software development, yet can be challenging for dynamically typed programming languages. In this paper we explore the use of neural network techniques to automatically learn code completion from a large corpus of dynamically typed JavaScript code. We show different neural networks that leverage not only token level information but also structural information, and evaluate their performance on different prediction tasks. We demonstrate that our models can outperform the state-of-the-art approach, which is based on decision tree techniques, on both next non-terminal and next terminal prediction tasks by 3.8 points and 0.5 points respectively. We believe that neural network techniques can play a transformative role in helping software developers manage the growing complexity of software systems, and we see this work as a first step in that direction.
1 INTRODUCTION
As the scale and complexity of modern software libraries and tools continue to grow, code completion has become an essential feature in modern integrated development environments (IDEs). By suggesting the right libraries, APIs, and even variables in real-time, intelligent code completion engines can substantially accelerate software development. Furthermore, as many projects move to dynamically typed and interpreted languages, effective code completion can help to reduce costly errors by eliminating typos and identifying the right arguments from context.
However, existing approaches to intelligent code completion either rely on strong typing (e.g., Visual Studio for C++), which limits their applicability to widely used dynamically typed languages (e.g., JavaScript and Python), or are based on simple heuristics and term frequency statistics which are often brittle and are relatively error-prone. In particular, Raychev et al. (2016a) proposes the state-ofthe-art probabilistic model for code, which generalizes both simple n-gram models and probabilistic grammar approaches. This approach, however, examines only a limited number of elements in the source code when completing the code. Therefore, the effectiveness of this approach may not scale well to large programs.
In this paper we explore the use of deep learning techniques to address the challenges of code completion for the widely used and dynamically typed JavaScript programming language. We formulate the code completion problem as a sequential prediction task over the traversal of a parse-tree structure consisting of both non-terminal structural nodes and terminal nodes encoding program text. We then present simple, yet expressive, LSTM-based (Hochreiter & Schmidhuber (1997)) models that leverage additional side information obtained by parsing the program structure.
Compared to widely used heuristic techniques, deep learning for code completion offers the opportunity to learn rich contextual models that can capture language and even library specific code patterns without requiring complex rules or expert intervention.
We evaluate our recurrent neural network architecture on an established benchmark dataset for the JavaScript code completion. Our evaluations reveal several findings: (1) when evaluated on short programs, our RNN-based models can achieve better performance on the next node prediction tasks compared to the prior art (Bielik et al. (2016); Raychev et al. (2016a)), which are based on decisiontree models; (2) our models’ prediction accuracies on longer programs, which is provided in the test set, but were not evaluated upon by previous work, are better than our models’ accuracies on shorter
∗The first and second authors contributed equally and are listed in an alphabetical order.
return (function(modules) { … function __webpack_require__(moduleId){ …
return module.exports; }
__webpack_require__.m = modules;
__webpack_require__.c = installedModules;
__webpack_require__. p: 0.9464 c: 0.0300 m: 0.0061
Figure 2: Correct prediction of the program in Figure 1
programs; and (3) in the scenario that the code completion engine suggests a list of candidates, our RNN-based models allow users to choose from a list of 5 candidates rather than inputting manually for over 96% of all time when this is possible.
These promising results encourage more investigation into developing neural network approaches for the code completion problem. We believe that our work not only highlights the importance of the field of neural network-based code completion, but is also an important step toward neural network-based program synthesis.
2 RELATED WORK
Existing approaches that build probabilistic models for code can typically be categorized as n-gram models (Hindle et al., 2012; Nguyen et al., 2013; Tu et al., 2014), probabilistic grammars (Collins, 2003; Allamanis & Sutton, 2014; Allamanis et al., 2015; Maddison & Tarlow, 2014; Liang et al., 2010), and log-bilinear models (Allamanis et al., 2015). Bielik et al. (2016) generalizes the PCFG approach and n-gram approach, while Raychev et al. (2016a) further introduces decision tree approaches to generalize Bielik et al. (2016).
Raychev et al. (2014) and White et al. (2015) explore how to use recurrent neural networks (RNNs) to facilitate the code completion task. However, these works only consider running RNNs on top of a token sequence to build a probabilistic model. Although the input sequence considered in Raychev et al. (2014) is produced from an abstract object, the structural information contained in the abstract syntax tree is not directly leveraged by the RNN structure in both of these two works. In contrast, we consider extending LSTM, a RNN structure, to leverage the structural information directly for the code prediction task.
Recently there has been an increasing interest in developing neural networks for program synthesis (Ling et al. (2016); Beltagy & Quirk (2016); Dong & Lapata (2016); Chen et al. (2016)). These works all consider synthesizing a program based on inputs in other formats such as images or natural language descriptions.
3 CODE COMPLETION VIA BIG CODE
In this section, we first introduce the problem of code completion and its challenges. Then we explain abstract syntax trees (AST), which we use as the input for our problems. Lastly, we formally define the code completion problem in different settings as several prediction problems based on a partial AST.
3.1 CODE COMPLETION: AN EXAMPLE
Code completion is a feature in some integrated development environments (IDEs) to speed up programmers’ coding process. Figure 1 demonstrates this feature in IntelliJ IDEA1. In this example,
1https://www.jetbrains.com/idea/
a part of a JavaScript program has been input to the IDE. When the dot symbol (i.e., “.”) is added after webpack require , the IDE prompts with a list of candidates that the programmer is most likely to input next. When a candidate matches the intention, the programmer can choose it from the list rather than typing it manually. In this work, we define the code completion problem as predicting the next symbol while a program is being written. We consider this problem as an important first step toward completing an entire program.
Traditional code completion techniques are developed by the programming language community to leverage context information for prediction. For example, when a programmer writes a Java program and inputs a variable name and then a dot symbol, the code completion engine will analyze the class of the variable and prompt the members of the class. In programming language literature, such information is referred to as type information. Statically typed languages, such as C and Java, enforces type checking at static time, so that the code completion engine can take advantage of full type information to make prediction without executing the code.
In recent years, dynamically typed languages, such as Python or JavaScript, have become increasingly popular. In these languages, type checking is usually performed dynamically while executing a program. Thus, type information may be only partially available to the code completion engine while the programmer is writing the code. Despite their popularity, the dynamic typing of these languages makes code completion for them challenging. For example, in Figure 1, the next symbol to be added is p. This symbol does not appear in the previous part of the program, and thus the code completion engine in IntelliJ IDEA IDE cannot prompt with this symbol.
However, this challenge may be remedied by leveraging a large corpus of code, a.k.a., big code. In fact, webpack require .p is a frequently used combination appearing in many programs on Github.com, one of the largest repositories of source code. Therefore, a code completion engine powered by big code is likely to learn this combination and to prompt p. In fact, our methods discussed in later sections can predict this case very well (Figure 2),
3.2 ABSTRACT SYNTAX TREE
Regardless of whether it is dynamically typed or statically typed, any programming language has an unambiguous context free grammar (CFG), which can be used to parse source code into an abstract syntax tree (AST). Further, an AST can be converted back into source code easily. Therefore we consider the input of our code completion problem as an AST, which is a typical assumption made by most code completion engines.
An AST is a rooted tree. In an AST, each non-leaf node corresponds to a non-terminal in the CFG specifying structure information. In JavaScript, non-terminals may be ExpressionStatement, ForStatement, IfStatement, SwitchStatement, etc. Each leaf node corresponds to a terminal in the CFG encoding program text. There are infinite possibilities for terminals. They can be variable names, string or numerical literals, operators, etc.
Figure 3 illustrates a part of the AST of the code snippet in Figure 1. In this tree, a node without a surrounding box (e.g., ExpressionStatement, etc.) denotes a non-terminal node. A node embraced by an orange surrounding box (e.g., installedModules) denotes a terminal node. At the bottom of the figure, there is a non-terminal node Property and a terminal node p. They have not been observed by the editor, so we use green to indicate this fact. Note that each non-terminal has at most one terminal as its child.
In a traditional code completion engine, the AST can be further processed by a type checker so that type information will be attached to each node. In this work, however, we focus on dynamically typed languages, and type information is not always available. Therefore, we do not consider the type information provided by a compiler, and leave it for our future work.
3.3 PROBLEM SETUP
In this work, we consider the input to be a partial AST, and the code completion problem is to predict the next node given the partial AST. In the following, we first define a partial AST, and then present the code completion problems in different scenarios.
ExpressionStatement
AssignmentStatement
MemberStatement
Identifier
Property
Identifier
__webpack_require__
c
installedModules
ExpressionStatement
AssignmentStatement
MemberStatement
Identifier
__webpack_require__
Property p
Figure 3: AST example (part)
ExpressionStatement
AssignmentStatement
MemberStatement
Identifier
Property
Identifier
__webpack_require__
c
installedModules
ExpressionStatement
AssignmentStatement
MemberStatement
Identifier
__webpack_require__
Property p
Partial AST
Right-most node
Next node following the partial AST
Figure 4: Partial AST example
Input: a partial AST. Given a complete AST T , we define a partial AST to be a subtree T ′ of T , such that for each node n in T ′, its left set LT (n) with respect to T is a subset of T ′, i.e., LT (n) ⊆ T ′. Here, the left set LT (n) of a node n with respect to T is defined as the set of all nodes in the in-order sequence during the depth-first search of T that are visited earlier than n .
Under this definition, in each partial AST T ′, there exists the right-most node nR, such that all other nodes in T ′ form its left set LT (nR). The next node in the in-order depth-first search visiting sequence after nR is also the first node not appearing in T ′. We call this node the next node following the partial AST. Figure 4 illustrates these concepts using the example in Figure 3. In the rest of the paper, we also refer to a partial AST as a query.
Next node prediction. Given a partial AST, the next node prediction problem, as suggested by its name, is to predict the next node following the partial AST. Based on the node’s kind, i.e., whether its a non-terminal node or a terminal one, we can categorize the problem into the next nonterminal prediction problem and the next terminal prediction problem. Although the next terminal prediction problem may sound more interesting, the next non-terminal prediction problem is also important, since it predicts the structure of the program. For example, when then next non-terminal is ForStatement, the next token in the source program is the keyword for, which does not have a corresponding terminal in the dataset. In this case, a model able to predict the next non-terminal can be used by the code-completion engine to emit the keyword for. These two tasks are also the same problems considered by previous works employing domain specific languages to achieve heuristic-based code completion (Raychev et al. (2016b); Bielik et al. (2016)).
Predicting the next node versus predicting the next token. A natural alternative formulation of the problem is predicting the next token given the token sequence that has been inputted so far. Such a formulation, however, does not take advantage of the AST information, which is very easy to acquire with a suitable parser. Predicting the next node allows taking advantage of such information to enable more intelligent code completion.
In particular, predicting the next non-terminal allows completing the structure of a code block rather than a single (keyword) token. For example, when the next token is a keyword for, the corresponding next non-terminal is ForStatement, which corresponding to the following code block:
for( ___ ; ___ ; ___ ) { // for-loop body
}
In this case, successfully predicting the next non-terminal node allows completing not only the next key token for, but also tokens such as (, ;, ), {, and }. Such structure completion enabled by predicting the next non-terminal is more compelling in modern IDEs.
Predicting the next terminal node allows completing identifiers, properties, literals, etc., which is similar to the next token prediction. However, predicting the next terminal node can leverage the information of the predicting node’s non-terminal parent, indicating what is being predicted, i.e., an identifier, a property, or a literal, etc. For example, when completing the following expression:
__webpack_require_.
the code completion engine with AST information will predict a property of webpack require , while the engine without AST information only learns two tokens webpack require and a dot “.” and tries to predict the next token without any constraint. In our evaluation, we show that by leveraging the information from the non-terminal parent can significantly improve the performance.
In this work, we focus on the next node prediction task, and leave the comparison with next token prediction as our future work.
Joint prediction. A more important problem than predicting only the next non-terminal or terminal itself is to predict the next non-terminal and terminal together. We refer to this task to predict both next non-terminal and terminal as the joint prediction problem. We hope code completion can be used to generate the entire parsing tree in the end, and joint prediction is one step further toward this goal than next node prediction.
Formally, the joint prediction problem that we consider is that, given a partial AST whose following node is a non-terminal one, we want to predict both the next non-terminal and the next terminal.
There may be non-terminal nodes which do not have a terminal child (e.g., the AssignmentStatement). In this case, we artificially add an EMPTY terminal as its child. Note that this treatment is the same as in Bielik et al. (2016). We count it as a correct prediction if both the next non-terminal and terminal are predicted correctly.
Denying prediction. There may be infinite possibilities for terminals, so it is impossible to predict all terminals correctly. We consider an alternative scenario that, when it thinks that the programmer will input a rare terminal, the code completion engine should have the ability to identify this case, and deny predicting the next node(s).
In our problem, we build a vocabulary for frequent terminals. All terminals not in this vocabulary are considered as an UNK terminal. In this case, when it predicts UNK for the next terminal, the code completion model is considered as denying prediction. Since non-terminals’ vocabulary size is very small, denying prediction is only considered for the next terminal prediction, but not for the next non-terminal prediction.
4 MODELS
In this section, we present the basic models considered in this work. In particular, given a partial AST as input, we first convert the AST into its left-child right-sibling representation, and serialize it as its in-order depth first search sequence. Thus, we consider the input for the next non-terminal prediction as a sequence of length k, i.e., (N1, T1), (N2, T2), ..., (Nk, Tk). Here, for each i, Ni is a non-terminal, and Ti is the terminal child of Ni. For each non-terminal node Ni, we encode not only its kind, but also whether the non-terminal has at least one non-terminal child, and/or one rightsibling. In doing so, from an input sequence, we can reconstruct the original AST. This encoding is also employed by Raychev et al. (2016a). We refer to each element in the sequence (e.g., (Ni, Ti)) as a token. As mentioned above, a non-terminal without a terminal child is considered to have an EMPTY child.
This input sequence (N1, T1), (N2, T2), ..., (Nk, Tk) is the only input for all problems except the next terminal prediction. For the next terminal prediction problem, besides the input sequence, we also have the information about the parent of the current predicting terminal, which is a non-terminal, i.e., Nk+1.
Throughout the rest of the discussion, we assume that both Ni and Ti employ one-hot encoding. The vocabulary sets of non-terminals and terminals are separate.
4.1 NEXT NON-TERMINAL PREDICTION
Given an input sequence, our first model predicts the next non-terminal. The architecture is illustrated in Figure 5. We refer to this model as NT2N, which stands for using the sequence of
Non-terminal and Terminal pairs TO predict the next Non-terminal. We first explain each layer of NT2N, and then introduce two variants of this model.
Embedding non-terminal and terminal. Given an input sequence, the embedding of each token is computed as
Ei = ANi +BTi (1)
where A is a J × VN matrix and B is a J × VT matrix. Here J is the size of the embedding vector, VN and VT are the vocabulary sizes of non-terminals and terminals respectively.
LSTM layer. Then the embedded sequence is fed into a LSTM layer to get the hidden state. In particular, a LSTM cell takes an input token and a hidden state hi−1, ci−1 from the previous LSTM cell as input, computes a hidden state hi, ci, and outputs hi, based on the following formulas:
qfo g  =  σσσ tanh PJ,2J ( xihi−1 )
ci = f ci−1 + q g hi = o tanh(ci)
Here, PJ,2J denotes a J×2J parameter matrix, where J is the size of the hidden state, i.e. dimension of hi, which is equal to the size of embedding vectors. σ and denote the sigmoid function and pointwise multiplication respectively.
Softmax layer. Assume hk is the output hidden state of the last LSTM cell. hk is fed into a softmax classifier to predict the next non-terminal. In particular, we have
N̂k+1 = softmax(WN × hk + bN )
where WN and bN are a matrix of size VN × J and a VN -dimensional vector respectively.
Using only non-terminal inputs. One variant of this model is to omit all terminal information from the input sequence. In this case, the embedding is computed as Ei = ANi. We refer to this model as N2N, which stands for using Non-terminal sequence TO predict the next Non-terminal.
Predicting the next terminal and non-terminal together. Based on NT2N, we can predict not only the next non-terminal but also the next terminal, using
T̂k+1 = softmax(WT × hk + bT )
where WT and bT are a matrix of size VT × J and a VT -dimensional vector respectively. In this case, the loss function has an extra term to give supervision on predicting T̂ . We refer to this model as NT2NT, which stands for using the sequence of Non-terminal and Terminal pairs TO predict the next Non-terminal and Terminal pair.
4.2 NEXT TERMINAL PREDICTION
In the next terminal prediction problem, the partial AST does not only contain (N1, T1), ..., (Nk, Tk), but also Nk+1. In this case, we can employ the architecture in Figure 6 to predict Tk+1. In particular, we first get the LSTM output hk in the same way as in NT2N. The final prediction is based on
T̂k+1 = softmax(WThk +WNTNk+1 + bT )
where WNT is a matrix of size VT × VN , and WT and bT are the same as in NT2NT. We refer to this model as NTN2T, which stands for Non-terminal and Terminal pair sequence and the next Non-terminal TO predict the next Terminal.
Note that the model NT2NT can also be used for the next terminal prediction task, although the non-terminal information Nk+1 is not leveraged. We will compare the two approaches later.
4.3 JOINT PREDICTION
We consider two approaches to predict the next non-terminal and the next terminal together. The first approach is NT2NT, which is designed to predict the two kinds of nodes together.
An alternative approach is to (1) use a next non-terminal approach X to predict the next nonterminal; and (2) feed the predicted non-terminal and the input sequence into NTN2T to predict the next terminal. We refer to such an approach as X+NTN2T.
4.4 DENYING PREDICTION
We say a model denies prediction when it predicts the next terminal to be UNK, a special terminal to substitute rare terminals. However, due to the large amount of rare terminals, the occurrences of UNK may be much greater than any single frequent terminals. In this case, a model that can deny prediction may tend to predict UNKs, and thus may predict for fewer queries than it should.
To mitigate this problem, we modify the loss function to be adaptive. Specifically, training a machine learning model fθ is to optimize the following objective:
argminθ ∑ i l(fθ(qi), yi)
where {(qi, yi)} is the training dataset consisting pairs of a query qi and its ground truth next token yi. l is the loss function to measure the distance between the prediction ŷi = fθ(qi) and the ground
truth yi. We choose l to be the standard cross-entropy loss. We introduce a weight αi for each sample (qi, yi) in the training dataset to change the objective to be as follows:
argminθ ∑ i αil(fθ(qi), yi)
When training a model not allowed to deny prediction, we set αi = 0 for yi = UNK, and αi = 1 otherwise. In doing so, it is equivalent to remove all queries whose ground truth next token is UNK.
When training a model that allows denying prediction, we set all αi to be 1. To denote this case, we put a notation “+D” at the end of the model, (e.g., NT2NT+D, etc.).
5 EVALUATION
5.1 DATASET
We use the JavaScript dataset2 provided by Raychev et al. (2016b) to evaluate different approaches. The statistics of the dataset can be found in Table 1. Raychev et al. (2016a) provides an approach, called PHOG, for the next token prediction. The reported accuracy results are based on a subset of 5.3 × 107 queries from the full test set. Specifically, Raychev et al. (2016a) chose all queries in each program containing fewer than 30,000 tokens.3 When we compare with their results, we use the same testset. Otherwise, without a special specification, our results are based on the full test set consisting of 8.3× 107 queries.
5.2 TRAINING DETAILS
Vocabulary In our dataset, there are 44 different kinds of non-terminals. Combining two more bits of information to indicate whether the non-terminal has a child and/or a right sibling, there are at most 176 different non-terminals. However, not all such combinations are possible: a ForStatement must have a child. In total, the vocabulary size for non-terminals is 97. For terminals, we sort all terminals in the training set by their frequencies. Then we choose the 50,000 most frequent terminals to build the vocabulary. We further add three special terminals: UNK for outof-vocabulary tokens, EOF indicating the end of program, and Empty for the non-terminal which does not have a terminal. Note that about 45% terminals in the dataset are Empty terminals.
Training details. We use a single layer LSTM network with hidden unit size of 1500 as our base model. To train the model, we use Adam (Kingma & Ba (2014)) with base learning rate 0.001. The learning rate is multiplied by 0.9 every 0.2 epochs. We clip the gradients’ norm to 5. The batch size is b = 80. We use truncated backpropagation through time, by unrolling the LSTM model s = 50 times to take an input sequence of length 50 in each batch (and therefore each batch contains b× s = 4000 tokens). We divide each program into segments consisting of s consecutive tokens. The last segment of a program, which may not be full, is padded with 〈EOF〉 tokens. We coalesce multiple epochs together. We organize all training data into b buckets. In each epoch, we randomly shuffle all programs in the training data to construct a queue. Whenever a bucket is empty, a program is popped from the queue and all segments of the program are inserted into the empty bucket sequentially. When the queue becomes empty, i.e., the current epoch finishes, all programs are re-shuffled randomly to reconstruct the queue. Each mini-batch is formed by b segments, i.e., one segment popped from each bucket. When the training data has been shuffled for e = 8 times, i.e., e epochs are inserted into the
2http://www.srl.inf.ethz.ch/js150 3This detail was not explained in the paper. We contacted the authors to confirm it.
bucket, we stop adding whole programs, and start adding only the first segment of each program: when a bucket is empty, a program is chosen randomly, and its first segment is added to the bucket. We terminate the training process when all buckets are empty at the same time. That is, all programs from the first 8 epochs have been trained. This is illustrated in Figure 7.
The hidden states are initialized with h0, c0, which are two trainable vectors. The hidden states of LSTM from the previous segment are fed into the next one as input if both segments belong to the same program. Otherwise, the hidden states are reset to be h0, c0. We observe that resetting the hidden states for every new program improves the performance a lot.
We initialize all parameters in h0, c0 to be 0. All other parameters are initialized with values uniformly randomly sampled from [−0.05, 0.05]. For each model, we train 5 sets of parameters using different random initializations. We evaluate the ensemble of the 5 models by averaging 5 softmax outputs. In our evaluation, we find that the ensemble improves the accuracy by 1 to 3 points in general.
5.3 NEXT NODE PREDICTION
In this section, we present the results of our models on next node prediction, and compare them with the counterparts in Bielik et al. (2016), which is the state-of-the-art on these tasks. Therefore, we use the same testset consisting of 5.3 × 107 queries as in Bielik et al. (2016). In the following, we first report results of next non-terminal prediction and of next terminal prediction, then evaluate our considered models’ performance on programs with different lengths.
Next non-terminal prediction. The results are presented in Table 2. From the table, we can observe that both NT2N and NT2NT can outperform Raychev et al. (2016a). In particular, an ensemble of 5 NT2N models improves Raychev et al. (2016a) by 3.8 percentage points. We also report the average accuracies of the 5 single models and the variance among them. We observe that the variance is very small, i.e., 0.1%− 0.2%. This indicates that the trained models’ accuracies are robust to random initialization.
Among the neural network approaches, NT2NT’s performance is lower than NT2N, even given that the former is provided with more supervision. This shows that given the limited capacity of the model, it may learn to trade off non-terminal prediction performance in favor of the terminal prediction task it additionally needs to perform.
Next terminal prediction. The results are presented in Table 3. We observe that an ensemble of 5 NTN2T models can outperform Raychev et al. (2016a) by 0.5 points. Without the ensemble, its accuracies are around 82.1%, i.e., 0.8 points less than Raychev et al. (2016a). For the 5 single models, we also observe that the variance on their accuracies is also very small, i.e., 0.1%. On the other hand, we observe that NT2NT has much worse performance than NTN2T, i.e., by 4.8 percentage points. This shows that leveraging additional information about the parent non-terminal of the current predicting terminal can improve the performance significantly.
Prediction accuracies on programs with different lengths. We examine our considered models’ performance over different subsets of the test set. In particular, we consider the queries in programs containing no more than 30,000 tokens, which is the same as used in Bielik et al. (2016); Raychev et al. (2016a). We also consider the rest of the queries in programs which have more than 30,000 tokens. The results are presented in Table 4.
We can observe that for both non-terminal and terminal prediction, accuracies on longer programs are higher than on shorter programs. This shows that a LSTM-based model may become more accurate when observing more code inputted by programmers.
We also report top 5 prediction accuracy. We can observe that the top 5 accuracy improves upon top 1 accuracy dramatically. This metric corresponds to the code completion scenario that an IDE may pop up a list of few (i.e., 5) candidates for users to choose from. In particular, NT2N can achieve 99.1% top-5 accuracy on the non-terminal prediction task. On the other hand, NTN2T can also achieve 89.2% accuracy on the terminal prediction task. In the test set, there are 7.4% of tokens in the data whose ground truth is UNK, i.e., non-top 50,000 most frequent tokens. This means that NTN2T can predict over 89.2/(100 − 7.4)% = 96.3% of all tokens whose ground truth is not UNK. Therefore, this means that the users can choose from the popup list without typing the token manually over 96% of all time that the code completion is possible if the completion is restricted to the top 50,000 most frequent tokens in the dataset.
The effectiveness of different UNK thresholds. We evaluate the effectiveness of how to choose the threshold to cut for UNK terminals on the accuracy. We randomly sample 1/10 of the training dataset and the test dataset and vary the thresholds to cut for UNK terminals from 10000 to 80000. We plot the percentage of UNK terminals in both the full test set and its subset in Figure 8. We can observe that the distributions of UNK terminals are almost the same in both sets. Further, when the threshold is 10000, i.e., all terminals out of the top 10000 most frequent ones are turned into UNKs, there are more than 11% UNK queries (i.e., queries with ground truth being UNK) in the test set. When the threshold increases to 50000 or more, this number drops to 7% to 6%. The variance of the UNK queries’ percentages is not large when threshold of UNK is varied from 50000 to 80000.
We train one NTN2T model for each threshold, and evaluate it using the sampled test set. The accuracies of different models are plotted in Figure 9. The trend of different models’ accuracies is similar to the trend of the percentage of non-UNK tokens in the test set. This is expected, since when the threshold increases the model has more chance to make correct predictions for original UNK queries. However, we observe that this is not always the case. For example, the accuracies of models trained with thresholds being 30000 and 40000 are almost the same, i.e., the difference is only 0.02%. We make similar observations among the models trained with thresholds being 60000, 70000, and 80000. Notice that we have observed above that when we train 5 models with different random initialization, the variance of the accuracies of these models is within 0.1%. Therefore, we conclude that when we increase the UNK threshold from 30000 to 40000 and from 60000 to 80000, the accuracies do not change significantly. One potential explanation is that when increasing the UNK threshold, while it has more chance to predict those otherwise UNK terminals, a model may also more likely make mistakes when it needs to choose the next terminal from more candidates.
5.4 JOINT PREDICTION
In this section, we evaluate different approaches to predict the next non-terminal and terminal together for the joint prediction task. In fact, NT2NT is designed for this task. Alternative approaches can predict the next non-terminal first, and then predict the next terminal based on the predicted next non-terminal. We choose NTN2T method as the second step to predict the next terminal, and we examine two different approaches as the first step to predict the next non-terminal: N2N and NT2N. Therefore, we compare three methods in total.
The top 1 accuracy results are presented in Table 5. N2N+NTN2T is less effective than NT2N+NTN2T, as expected, since when predicting the non-terminal in the first step, N2N is less effective than NT2N as we have shown in Table 4. On the other hand, NT2NT’s performance is better than N2N+NTN2T, but is worse than NT2N+NTN2T.
We observe that for all these three combinations, we have
Pr(T̂k+1 = Tk+1 ∧ N̂k+1 = Nk+1) > Pr(T̂k+1 = Tk+1)Pr(N̂k+1 = Nk+1)
These facts indicate that the events of the next non-terminal and terminal being predicted correctly are not independent, but very relevant to each other instead. This is also the case for NT2NT, even though NT2NT predicts the next non-terminal and the next terminal independently conditional upon the LSTM hidden states.
5.5 DENYING PREDICTION
We compare the models which do not deny prediction (i.e., NT2NT and NTN2T) and those which do (i.e., NT2NT+D and NTN2T+D). Results are presented in Table 6. For a reference, in the test set, there are 7.42% UNK queries. We can observe that deny prediction models (i.e., +D models) have higher accuracies than the corresponding original models. This is expected. Since deny prediction models allow predicting UNK terminals, while NT2NT and NTN2T fail on all UNK queries, +D will succeed on most of them. We further evaluate the accuracy on non-UNK terminals. One may expect that since +D models may prefer to predict UNK, a standard model should have a higher accuracy on non-UNK terminals than its deny prediction counterpart. The results show that this is indeed the case, but the margin is very small, i.e., 0.1% for NT2NT and 0.3% for NTN2T. This means that, allowing denying prediction does not necessarily sacrifice a model’s ability on predicting non-UNK terminals.
We are also interested in how frequent a +D model will deny prediction. We can observe that NTN2T+D will deny prediction for only 6.1% of all queries, which is even less than the percentage of UNK queries (i.e., 7.42%). This shows that although we allow the model to deny prediction, it is conservative when executing this privilege. This partially explains why NTN2T+D’s accuracy on non-UNK terminals is not much less than NTN2T’s.
Effectiveness of the value of α. We are interested in how the hyperparameter α in a +D model affects its accuracy. We train 11 different NTN2T+D models on the 1/10 subset of the training set, which is used above to examine the effectiveness of UNK thresholds, by varying α from 0.0 to 1.0. Notice that α = 0.0, this model becomes a standard NTN2T model.
We plot both overall accuracies and accuracies on non-UNK terminals in Figure 10. We observe the same effect as above: 1) the overall accuracy for α = 1 is 6% higher than the one for α = 0; and 2) the accuracy on non-UNK terminals for α = 1 is less than the one for α = 0, but the margin is not large (i.e., less than 1%). When we increase α from 0 to 0.3, we can observe that the overall accuracy steeply increases. When we further increase α, however, the overall accuracy becomes steady. This is also the case for accuracy on non-UNK terminals. The result of this experiment
shows that how to set α is a trade-off between the overall accuracy and the accuracy on non-UNK terminals and how to choose α depends on the application.
5.6 RUNTIME
We evaluate our models’ runtime performance. Our models are implemented in TensorFlow (Abadi et al. (2016)). We evaluate our models on a machine equipped with 16 Intel Xeon CPUs, 16 GB RAM, and a single GPU Tesla K80. All queries from the same program are processed incrementally. That is, given two queries A,B, if A has one more node than B, then the LSTM outputs for B will be reused for processing A, so that only the additional node in A needs to be processed. Note that this is consistent with the practice where programs are written incrementally from beginning to end. For each model, we feed in one query at a time into the model. There are 3939 queries in total coming from randomly chosen programs. We measure the overall response latency for each query. We observe that the query response time is consistent across all queries. On average, each model takes around 16 milliseconds to respond a query on GPU, and around 33 milliseconds on CPU. Note that these numbers are from just a proof of concept implementation and we have not optimized the code. Considering that a human being usually does not type in a token within 30 milliseconds, we conclude that our approach is efficient enough for potential practical usage. We emphasize that these numbers do not directly correspond to the runtime latency when the techniques are deployed to a code completion engine, since the changes of AST serialization may not be sequential while users are programming incrementally. This analysis, however, only provides an evidence to show the feasibility of applying our approach toward a full-fledged code completion engine.
6 CONCLUSION
In this paper we introduce, motivate, and formalize the problem of automatic code completion. We describe LSTM-based approaches that capture parsing structure readily available in the code completion task. We introduce a simple LSTM architecture to model program context. We then explore several variants of our basic architecture for different variants of the code completion problem. We evaluate our techniques on a challenging JavaScript code completion benchmark and compare against the state-of-the-art code completion approach. We demonstrate that deep learning techniques can achieve better prediction accuracy by learning program patterns from big code. In addition, we find that using deep learning techniques, our models perform better for longer programs than for shorter ones, and when the code completion engine can pop up a list of candidates, our approach allows users to choose from the list instead of inputting the token over 96% of all time that this is possible. We also evaluate our approaches’ runtime performance and demonstrate that deep code completion has the potential to run in real-time as users type. We believe that deep learning techniques can play a transformative role in helping software developers manage the growing complexity of software systems, and we see this work as a first step in that direction.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable comments. This material is based upon work partially supported by the National Science Foundation under Grant No. TWC-1409915, and a DARPA grant FA8750-15-2-0104. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation and DARPA.

While the overall direction is promising, there are several serious issues with the paper which affect the novelty and validity of the results:
1. Incorrect claims about related work affecting novelty:
  - This work is not the first to explore a deep learning approach to automatic code completion: “Toward Deep Learning Software Repositories”, MSR’15 also uses deep learning for code completion, and is not cited.
  - “Code Completion with Statistical Language Models”, PLDI’14 is cited incorrectly -- it also does code completion with recurrent neural networks.
  - PHOG is independent of JavaScript -- it does representation learning and has been applied to other languages (e.g., Python, see OOPSLA’16 below).
  - This submission is not the only one that “can automatically extract features”. Some high-precision (cited) baselines do it.
  - “Structured generative models of natural source code” is an incorrect citation. It is from ICML’14 and has more authors. It is also a log-linear model and conditions on more context than claimed in this submission.
2. Uses a non-comparable prediction task for non-terminal symbols: The type of prediction made here is simpler than the one used in PHOG and state-of-the-art (see OOPSLA’16 paper below) and thus the claimed 11 point improvement is not substantiated. In particular, in JavaScript there are 44 types of nodes. However, a PHOG and OOPSLA’16 predictions considers not only these 44 types, but also whether there are right siblings and children of a node. This is necessary for predicting tree fragments instead of a sequence of nodes. It however makes the prediction harder than the one considered here (it leads to 150+ labels, a >3x increase).
3. Not comparing to state-of-the-art: the state-of-the-art however is not the basic PHOG cited here, but “Probabilistic Model for Code with Decision Trees”, (OOPSLA 2016) which appeared before the submission deadline for ICLR’17: