We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.
1 INTRODUCTION
Adversarial examples have been shown to exist for a variety of deep learning architectures.1 They are small perturbations of the original inputs, often barely visible to a human observer, but carefully crafted to misguide the network into producing incorrect outputs. Seminal work by Szegedy et al. (2013) and Goodfellow et al. (2014), as well as much recent work, has shown that adversarial examples are abundant and finding them is easy.
Most previous work focuses on the application of adversarial examples to the task of classification, where the deep network assigns classes to input images. The attack adds small adversarial perturbations to the original input image. These perturbations cause the network to change its classification of the input, from the correct class to some other incorrect class (possibly chosen by the attacker). Critically, the perturbed input must still be recognizable to a human observer as belonging to the original input class.2
Deep generative models, such as Kingma & Welling (2013), learn to generate a variety of outputs, ranging from handwritten digits to faces (Kulkarni et al., 2015), realistic scenes (Oord et al., 2016), videos (Kalchbrenner et al., 2016), 3D objects (Dosovitskiy et al., 2016), and audio (van den Oord et al., 2016). These models learn an approximation of the input data distribution in different ways, and then sample from this distribution to generate previously unseen but plausible outputs.
To the best of our knowledge, no prior work has explored using adversarial inputs to attack generative models. There are two main requirements for such work: describing a plausible scenario in which an attacker might want to attack a generative model; and designing and demonstrating an attack that succeeds against generative models. We address both of these requirements in this work.
One of the most basic applications of generative models is input reconstruction. Given an input image, the model first encodes it into a lower-dimensional latent representation, and then uses that representation to generate a reconstruction of the original input image. Since the latent representation
1 Adversarial examples are even easier to produce against most other machine learning architectures, as shown in Papernot et al. (2016), but we are focused on deep networks.
2 Random noise images and “fooling” images (Nguyen et al., 2014) do not belong to this strict definition of an adversarial input, although they do highlight other limitations of current classifiers.
usually has much fewer dimensions than the original input, it can be used as a form of compression. The latent representation can also be used to remove some types of noise from inputs, even when the network has not been explicitly trained for denoising, due to the lower dimensionality of the latent representation restricting what information the trained network is able to represent. Many generative models also allow manipulation of the generated output by sampling different latent values or modifying individual dimensions of the latent vectors without needing to pass through the encoding step.
These properties of input reconstruction generative networks suggest a variety of different attacks that would be enabled by effective adversaries against generative networks. Any attack that targets the compression bottleneck of the latent representation can exploit natural security vulnerabilities in applications built to use that latent representation. Specifically, if the person doing the encoding step is separated from the person doing the decoding step, the attacker may be able to cause the encoding party to believe they have encoded a particular message for the decoding party, but in reality they have encoded a different message of the attacker’s choosing. We explore this idea in more detail as it applies to the application of compressing images using a VAE or VAE-GAN architecture.
2 RELATED WORK AND BACKGROUND
This work focuses on adversaries for variational autoencoders (VAEs, proposed in Kingma & Welling (2013)) and VAE-GANs (VAEs composed with a generative adversarial network, proposed in Larsen et al. (2015)).
2.1 RELATED WORK ON ADVERSARIES
Many adversarial attacks on classification models have been described in existing literature (Goodfellow et al., 2014; Szegedy et al., 2013). These attacks can be untargeted, where the adversary’s goal is to cause any misclassification, or the least likely misclassification (Goodfellow et al., 2014; Kurakin et al., 2016); or they can be targeted, where the attacker desires a specific misclassification. Moosavi-Dezfooli et al. (2016) gives a recent example of a strong targeted adversarial attack. Some adversarial attacks allow for a threat model where the adversary does not have access to the target model (Szegedy et al., 2013; Papernot et al., 2016), but commonly it is assumed that the attacker does have that access, in an online or offline setting (Goodfellow et al., 2014; Kurakin et al., 2016).3
Given a classifier f(x) : x ∈ X → y ∈ Y and original inputs x ∈ X , the problem of generating untargeted adversarial examples can be expressed as the following optimization: argminx∗ L(x,x
∗) s.t. f(x∗) 6= f(x), where L(·) is a chosen distance measure between examples from the input space (e.g., the L2 norm). Similarly, generating a targeted adversarial attack on a classifier can be expressed as argminx∗ L(x,x
∗) s.t. f(x∗) = yt, where yt ∈ Y is some target label chosen by the attacker.
These optimization problems can often be solved with optimizers like L-BFGS or Adam (Kingma & Ba, 2015), as done in Szegedy et al. (2013) and Carlini & Wagner (2016). They can also be approximated with single-step gradient-based techniques like fast gradient sign (Goodfellow et al., 2014), fast gradient L2 (Huang et al., 2015), or fast least likely class (Kurakin et al., 2016); or they can be approximated with iterative variants of those and other gradient-based techniques (Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016).
An interesting variation of this type of attack can be found in Sabour et al. (2015). In that work, they attack the hidden state of the target network directly by taking an input image x and a target image xt and searching for a perturbed variant of x that generates similar hidden state at layer l of the target network to the hidden state at the same layer generated by xt. This approach can also be applied directly to attacking the latent vector of a generative model.
A variant of this attack has also been applied to VAE models in the concurrent work of Tabacof et al. (2016)4, which uses the KL divergence between the latent representation of the source and target images to generate the adversarial example. However in their paper, the authors mention that they tried attacking the output directly and that this only managed to make the reconstructions more
3 See Papernot et al. (2015) for an overview of different adversarial threat models. 4 This work was made public shortly after we published our early drafts.
blurry. While they do not explain the exact experimental setting, the attack sounds similar to our LVAE attack, which we find very successful. Also, in their paper the authors do not consider the more advanced VAE-GAN models and more complex datasets like CelebA.
2.2 BACKGROUND ON VAES AND VAE-GANS
The general architecture of a variational autoencoder consists of three components, as shown in Figure 8. The encoder fenc(x) is a neural network mapping a high-dimensional input representation x into a lower-dimensional (compressed) latent representation z. All possible values of z form a latent space. Similar values in the latent space should produce similar outputs from the decoder in a well-trained VAE. And finally, the decoder/generator fdec(z), which is a neural network mapping the compressed latent representation back to a high-dimensional output x̂. Composing these networks allows basic input reconstruction x̂ = fdec(fenc(x)). This composed architecture is used during training to backpropagate errors from the loss function.
The variational autoencoder’s loss functionLVAE enables the network to learn a latent representation that approximates the intractable posterior distribution p(z|x):
LVAE = −DKL[q(z|x)||p(z)] + Eq[log p(x|z)]. (1)
q(z|x) is the learned approximation of the posterior distribution p(z|x). p(z) is the prior distribution of the latent representation z. DKL denotes the Kullback–Leibler divergence. Eq[log p(x|z)] is the variational lower bound, which in the case of input reconstruction is the cross-entropy H[x, x̂] between the inputs x and their reconstructions x̂. In order to generate x̂ the VAE needs to sample q(z|x) and then compute fdec(z). For the VAE to be fully differentiable while sampling from q(z|x), the reparametrization trick (Kingma & Welling, 2013) extracts the random sampling step from the network and turns it into an input, ε. VAEs are often parameterized with Gaussian distributions. In this case, fenc(x) outputs the distribution parametersµ andσ2. That distribution is then sampled by computing z = µ+ε √ σ2 where ε ∼ N(0, 1) is the input random sample, which does not depend on any parameters of fenc, and thus does not impact differentiation of the network.
The VAE-GAN architecture of Larsen et al. (2015) has the same fenc and fdec pair as in the VAE. It also adds a discriminator fdisc that is used during training, as in standard generative adversarial networks (Goodfellow et al., 2014). The loss function of fdec uses the disciminator loss instead of cross-entropy for estimating the reconstruction error.
3 PROBLEM DEFINITION
We provide a motivating attack scenario for adversaries against generative models, as well as a formal definition of an adversary in the generative setting.
3.1 MOTIVATING ATTACK SCENARIO
To motivate the attacks presented below, we describe the attack scenario depicted in Figure 1. In this scenario, there are two parties, the sender and the receiver, who wish to share images with each other over a computer network. In order to conserve bandwidth, they share a VAE trained on the input distribution of interest, which will allow them to send only latent vectors z.
The attacker’s goal is to convince the sender to send an image of the attacker’s choosing to the receiver, but the attacker has no direct control over the bytes sent between the two parties. However, the attacker has a copy of the shared VAE. The attacker presents an image x∗ to the sender which resembles an image x that the sender wants to share with the receiver. For example, the sender wants to share pictures of kittens with the receiver, so the attacker presents a web page to the sender with a picture of a kitten, which is x∗. The sender chooses x∗ and sends its corresponding z to the receiver, who reconstructs it. However, because the attacker controlled the chosen image, when the receiver reconstructs it, instead of getting a faithful reproduction x̂ of x (e.g., a kitten), the receiver sees some other image of the attacker’s choosing, x̂adv, which has a different meaning from x (e.g., a request to send money to the attacker’s bank account).
There are other attacks of this general form, where the sender and the receiver may be separated by distance, as in this example, or by time, in the case of storing compressed images to disk for later retrieval. In the time-separated attack, the sender and the receiver may be the same person or multiple different people. In either case, if they are using the insecure channel of the VAE’s latent space, the messages they share may be under the control of an attacker. For example, an attacker may be able to fool an automatic surveillance system if the system uses this type of compression to store the video signal before it is processed by other systems. In this case, the subsequent analysis of the video signal could be on compromised data showing what the attacker wants to show.
While we do not specifically attack their models, viable compression schemes based on deep neural networks have already been proposed in the literature, showing promising results Toderici et al. (2015; 2016).
3.2 DEFINING ADVERSARIAL EXAMPLES AGAINST GENERATIVE MODELS
We make the following assumptions about generating adversarial examples on a target generative model, Gtarg(x) = fdec(fenc(x)). Gtarg is trained on inputs X that can naturally be labeled with semantically meaningful classes Y , although there may be no such labels at training time, or the labels may not have been used during training. Gtarg normally succeeds at generating an output x̂ = Gtarg(x) in class y when presented with an input x from class y. In other words, whatever target output class the attacker is interested in, we assume that Gtarg successfully captures it in the latent representation such that it can generate examples of that class from the decoder. This target output class does not need to be from the most salient classes in the training dataset. For example, on models trained on MNIST, the attacker may not care about generating different target digits (which are the most salient classes). The attacker may prefer to generate the same input digits in a different style (perhaps to aid forgery). We also assume that the attacker has access to Gtarg. Finally, the attacker has access to a set of examples from the same distribution as X that have the target label
yt the attacker wants to generate. This does not mean that the attacker needs access to the labeled training dataset (which may not exist), or to an appropriate labeled dataset with large numbers of examples labeled for each class y ∈ Y (which may be hard or expensive to collect). The attacks described here may be successful with only a small amount of data labeled for a single target class of interest.
One way to generate such adversaries is by solving the optimization problem argminx∗ L(x,x
∗) s.t. ORACLE(Gtarg(x∗)) = yt, where ORACLE reliably discriminates between inputs of class yt and inputs of other classes. In practice, a classifier trained by the attacker may server as ORACLE. Other types of adversaries from Section 2.1 can also be used to approximate this optimization in natural ways, some of which we describe in Section 4.
If the attacker only needs to generate one successful attack, the problem of determining if an attack is successful can be solved by manually reviewing the x∗ and x̂adv pairs and choosing whichever the attacker considers best. However, if the attacker wants to generate many successful attacks, an automated method of evaluating the success of an attack is necessary. We show in Section 4.5 how to measure the effectiveness of an attack automatically using a classifier trained on z = fenc(x).
4 ATTACK METHODOLOGY
The attacker would like to construct an adversarially-perturbed input to influence the latent representation in a way that will cause the reconstruction process to reconstruct an output for a different class. We propose three approaches to attacking generative models: a classifier-based attack, where we train a new classifier on top of the latent space z and use that classifier to find adversarial examples in the latent space; an attack using LVAE to target the output directly; and an attack on the latent space, z. All three methods are technically applicable to any generative architecture that relies on a learned latent representation z. Without loss of generality, we focus on the VAE-GAN architecture.
4.1 CLASSIFIER ATTACK
By adding a classifier fclass to the pre-trained generative model5, we can turn the problem of generating adversaries for generative models back into the previously solved problem of generating adversarial examples for classifiers. This approach allows us to apply all of the existing attacks on classifiers in the literature. However, as discussed below, using this classifier tends to produce lower-quality reconstructions from the adversarial examples than the other two attacks due to the inaccuracies of the classifier.
Step 1. The weights of the target generative model are frozen, and a new classifier fclass(z)→ ŷ is trained on top of fenc using a standard classification loss Lclassifier such as cross-entropy, as shown in Figure 3. This process is independent of how the original model is trained, but it requires a
5 This is similar to the process of semi-supervised learning in Kingma et al. (2014), although the goal is different.
training corpus pulled from approximately the same input distribution as was used to train Gtarg, with ground truth labels for at least two classes: yt and yt̃, the negative class.
Step 2. With the trained classifier, the attacker finds adversarial examples x∗ using the methods described in Section 4.4.
Using fclass to generate adversarial examples does not always result in high-quality reconstructions, as can be seen in the middle column of Figure 5 and in Figure 11. This appears to be due to the fact that fclass adds additional noise to the process. For example, fclass sometimes confidently misclassifies latent vectors z that represent inputs that are far from the training data distribution, resulting in fdec failing to reconstruct a plausible output from the adversarial example.
4.2 LVAE ATTACK
Our second approach generates adversarial perturbations using the VAE loss function. The attacker chooses two inputs, xs (the source) and xt (the target), and uses one of the standard adversarial methods to perturb xs into x∗ such that its reconstruction x̂∗ matches the reconstruction of xt, using the methods described in Section 4.4.
The adversary precomputes the reconstruction x̂t by evaluating fdec(fenc(xt)) once before performing optimization. In order to use LVAE in an attack, the second term (the reconstruction loss) of LVAE (see Equation 1) is changed so that instead of computing the reconstruction loss between x and x̂, the loss is computed between x̂∗ and x̂t. This means that during each optimization iteration, the adversary needs to compute x̂∗, which requires the full fdec(fenc(x∗)) to be evaluated.
4.3 LATENT ATTACK
Our third approach attacks the latent space of the generative model.
Single latent vector target. This attack is similar to the work of Sabour et al. (2015), in which they use a pair of source image xs and target image xt to generate x∗ that induces the target network to produce similar activations at some hidden layer l as are produced by xt, while maintaining similarity between xs and x∗.
For this attack to work on latent generative models, it is sufficient to compute zt = fenc(xt) and then use the following loss function to generate adversarial examples from different source images xs, using the methods described in Section 4.4:
Llatent = L(zt, fenc(x∗)). (2) L(·) is a distance measure between two vectors. We use the L2 norm, under the assumption that the latent space is approximately euclidean.
We also explored a variation on the single latent vector target attack, which we describe in Section A.1 in the Appendix.
4.4 METHODS FOR SOLVING THE ADVERSARIAL OPTIMIZATION PROBLEM
We can use a number of different methods to generate the adversarial examples. We initially evaluated both the fast gradient sign Goodfellow et al. (2014) method and an L2 optimization method. As the latter produces much better results we focus on the L2 optimization method, while we include some FGS results in the Appendix. The attack can be used either in targeted mode (where we want a specific class, yt, to be reconstructed) or untargeted mode (where we just want an incorrect class to be reconstructed). In this paper, we focus on the targeted mode of the attacks.
L2 optimization. The optimization-based approach, explored in Szegedy et al. (2013) and Carlini & Wagner (2016), poses the adversarial generation problem as the following optimization problem:
argminx∗ λL(x,x ∗) + L(x∗, yt). (3)
As above, L(·) is a distance measure, and L is one of Lclassifier, LVAE, or Llatent. The constant λ is used to balance the two loss contributions. For the LVAE attack, the optimizer must do a full
reconstruction at each step of the optimizer. The other two attacks do not need to do reconstructions while the optimizer is running, so they generate adversarial examples much more quickly, as shown in Table 1.
4.5 MEASURING ATTACK EFFECTIVENESS
To generate a large number of adversarial examples automatically against a generative model, the attacker needs a way to judge the quality of the adversarial examples. We leverage fclass to estimate whether a particular attack was successful.6
Reconstruction feedback loop. The architecture is the same as shown in Figure 3. We use the generative model to reconstruct the attempted adversarial inputs x∗ by computing:
x̂∗ = fdec(fenc(x ∗)). (4)
Then, fclass is used to compute: ŷ = fclass(fenc(x̂
∗)). (5) The input adversarial examples x∗ are not classified directly, but are first fed to the generative model for reconstruction. This reconstruction loop improves the accuracy of the classifier by 60% on average against the adversarial attacks we examined. The predicted label ŷ after the reconstruction feedback loop is compared with the attack target yt to determine if the adversarial example successfully reconstructed to the target class. If the precision and recall of fclass are sufficiently high on yt, fclass can be used to filter out most of the failed adversarial examples while keeping most of the good ones.
We derive two metrics from classifier predictions after one reconstruction feedback loop. The first metric is ASignore−target, the attack success rate ignoring targeting, i.e., without requiring the output class of the adversarial example to match the target class:
ASignore−target = 1
N N∑ i=1 1ŷi 6=yi (6)
N is the total number of reconstructed adversarial examples; 1ŷi 6=yi is 1 when ŷi, the classification of the reconstruction for image i, does not equal yi, the ground truth classification of the original image, and 0 otherwise. The second metric is AStarget, the attack success rate including targeting (i.e., requiring the output class of the adversarial example to match the target class), which we define similarly as:
AStarget = 1
N N∑ i=1 1ŷi=yit . (7)
Both metrics are expected to be higher for more successful attacks. Note that AStarget ≤ ASignore−target. When computing these metrics, we exclude input examples that have the same ground truth class as the target class.
5 EVALUATION
We evaluate the three attacks on MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011) and CelebA (Liu et al., 2015), using the standard training and validation set splits. The VAE and VAEGAN architectures are implemented in TensorFlow (Abadi & et al., 2015). We optimized using Adam with learning rate 0.001 and other parameters set to default values for both the generative model and the classifier. For the VAE, we use two architectures: a simple architecture with a single fully-connected hidden layer with 512 units and ReLU activation function; and a convolutional architecture taken from the original VAE-GAN paper Larsen et al. (2015) (but trained with only the VAE loss). We use the same architecture trained with the additional GAN loss for the VAE-GAN model, as described in that work. For both VAE and VAE-GAN we use a 50-dimensional latent representation on MNIST, a 1024-dimensional latent representation on SVHN and 2048-dimensional latent representation on CelebA.
6 Note that fclass here is being used in a different manner than when we use it to generate adversarial examples. However, the network itself is identical, so we don’t distinguish between the two uses in the notation.
In this section we only show results where no sampling from latent space has been performed. Instead we use the mean vector µ as the latent representation z. As sampling can have an effect on the resulting reconstructions, we evaluated it separately. We show the results with different number of samples in Figure 22 in the Appendix. On most examples, the visible change is small and in general the attack is still successful.
5.1 MNIST
Both VAE and VAE-GAN by themselves reconstruct the original inputs well as show in Figure 9, although the quality from the VAE-GAN is noticeably better. As a control, we also generate random noise of the same magnitude as used for the adversarial examples (see Figure 13), to show that random noise does not cause the reconstructed noisy images to change in any significant way. Although we ran experiments on both VAEs and VAE-GANs, we only show results for the VAE-GAN as it generates much higher quality reconstructions than the corresponding VAE.
5.1.1 CLASSIFIER ATTACK
We use a simple classifier architecture to help generate attacks on the VAE and VAE-GAN models. The classifier consists of two fully-connected hidden layers with 512 units each, using the ReLU activation function. The output layer is a 10 dimensional softmax. The input to the classifier is the 50 dimensional latent representation produced by the VAE/VAE-GAN encoder. The classifier achieves 98.05% accuracy on the validation set after training for 100 epochs.
To see if there are differences between classes, we generate targeted adversarial examples for each MNIST class and present the results per-class. For the targeted attacks we used the optimization method with lambda 0.001, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L2 norm of the difference between original images and generated adversarial examples using the classifier attack is 3.36, while the mean RMSD is 0.120.
Numerical results in Table 2 show that the targeted classifier attack successfully fools the classifier. Classifier accuracy is reduced to 0%, while the matching rate (the ratio between the number of predictions matching the target class and the number of incorrectly classified images) is 100%, which means that all incorrect predictions match the target class. However, what we are interested in (as per the attack definition from Section 3.2) is how the generative model reconstructs the adversarial examples. If we look at the images generated by the VAE-GAN for class 0, shown in Figure 4, the targeted attack is successful on some reconstructed images (e.g. one, four, five, six and nine are reconstructed as zeroes). But even when the classifier accuracy is 0% and matching rate is 100%, an incorrect classification does not always result in a reconstruction to the target class, which shows that the classifier is fooled by an adversarial example more easily than the generative model.
Reconstruction feedback loop. The reconstruction feedback loop described in Section 4.5 can be used to measure how well a targeted attack succeeds in making the generative model change the
reconstructed classes. Table 4 in the Appendix shows ASignore−target and AStarget for all source and target class pairs. A higher value signifies a more successful attack for that pair of classes. It is interesting to observe that attacking some source/target pairs is much easier than others (e.g. pair (4, 0) vs. (0, 8)) and that the results are not symmetric over source/target pairs. Also, some pairs do well in ASignore−target, but do poorly in AStarget (e.g., all source digits when targeting 4). As can be seen in Figure 11, the classifier adversarial examples targeting 4 consistently fail to reconstruct into something easily recognizable as a 4. Most of the reconstructions look like 5, but the adversarial example reconstructions of source 5s instead look like 0 or 3.
5.1.2 LVAE ATTACK
For generating adversarial examples using the LVAE attack, we used the optimization method with λ = 1.0, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L2 norm of the difference between original images and generated adversarial examples with this approach is 3.68, while the mean RMSD is 0.131.
We show ASignore−target and AStarget of the LVAE attack in Table 5 in the Appendix. Comparing with the numerical evaluation results of the latent attack (below), we can see that both methods achieve similar results on MNIST.
5.1.3 LATENT ATTACK
To generate adversarial examples using the latent attack, we used the optimization method with λ = 1.0, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L2 norm of the difference between original images and generated adversarial examples using this approach is 2.96, while the mean RMSD is 0.105.
Table 3 shows ASignore−target and AStarget for all source and target class pairs. Comparing with the numerical evaluation results of the classifier attack we can see that the latent attack performs much better. This result remains true when visually comparing the reconstructed images, shown in Figure 5.
We also tried an untargeted version of the latent attack, where we change Equation 2 to maximize the distance in latent space between the encoding of the original image and the encoding of the adversarial example. In this case the loss we are trying to minimize is unbounded, since the L2 distance can always grow larger, so the attack normally fails to generate a reasonable adversarial example.
Additionally, we also experimented with targeting latent representations of specific images from the training set instead of taking the mean, as described in Section 4.3. We show the numerical results in Table 3 and the generated reconstructions in Figure 15 (in the Appendix). It is also interesting to compare the results with LVAE, by choosing the same image as the target. Results for LVAE for the same target images as in Table 3 are shown in Table 6 in the Appendix. The results are identical between the two attacks, which is expected as the target image is the same – only the loss function differs between the methods.
5.2 SVHN
The SVHN dataset consists of cropped street number images and is much less clean than MNIST. Due to the way the images have been processed, each image may contain more than one digit; the target digit is roughly in the center. VAE-GAN produces high-quality reconstructions of the original images as shown in Figure 17 in the Appendix.
For the classifier attack, we set λ = 10−5 after testing a range of values, although we were unable to find an effective value for this attack against SVHN. For the latent and LVAE attacks we set λ = 10. In Table 10 we show ASignore−target and AStarget for the L2 optimization latent attack. The evaluation metrics are less strong on SVHN than on MNIST, but it is still straightforward for an attacker to find a successful attack for almost all source/target pairs. Figure 2 supports this evaluation. Visual inspection shows that 11 out of the 12 adversarial examples reconstructed as 0, the target digit. It is worth noting that 2 out of the 12 adversarial examples look like zeros (rows 1 and 11), and two others look like both the original digit and zero, depending on whether the viewer focuses on the light or dark areas of the image (rows 4 and 7). The L2 optimization latent attack achieves much better results than the LVAE attack (see Table 11 and Figure 6) on SVHN, while both attacks work equally well on MNIST.
5.3 CELEBA
The CelebA dataset consists of more than 200,000 cropped faces of celebrities, each annotated with 40 different attributes. For our experiments, we further scale the images to 64x64 and ignore the attribute annotations. VAE-GAN reconstructions of original images after training are shown in Figure 19 in the Appendix.
Since faces don’t have natural classes, we only evaluated the latent and LVAE attacks. We tried lambdas ranging from 0.1 to 0.75 for both attacks. Figure 20 shows adversarial examples generated
using the latent attack and a lambda value of 0.5 (L2 norm between original images and generated adversarial examples 9.78, RMSD 0.088) and the corresponding VAE-GAN reconstructions. Most of the reconstructions reflect the target image very well. We get even better results with the LVAE attack, using a lambda value of 0.75 (L2 norm between original images and generated adversarial examples 8.98, RMSD 0.081) as shown in Figure 21.
5.4 SUMMARY OF DIFFERENT ATTACK METHODS
Table 1 shows a comparison of the mean distances between original images and generated adversarial examples for the three different attack methods. The larger the distance between the original image and the adversarial perturbation, the more noticeable the perturbation will tend to be, and the more likely a human observer will no longer recognize the original input, so effective attacks keep these distances small while still achieving their goal. The latent attack consistently gives the best results in our experiments, and the classifier attack performs the worst.
We also measure the time it takes to generate 1000 adversarial examples using the given attack method. The LVAE attack is by far the slowest of the three, due to the fact that it requires computing full reconstructions at each step of the optimizer when generating the adversarial examples. The other two attacks do not need to run the reconstruction step during optimization of the adversarial examples.
6 CONCLUSION
We explored generating adversarial examples against generative models such as VAEs and VAEGANs. These models are also vulnerable to adversaries that convince them to turn inputs into surprisingly different outputs. We have also motivated why an attacker might want to attack generative models. Our work adds further support to the hypothesis that adversarial examples are a general phenomenon for current neural network architectures, given our successful application of adversarial attacks to popular generative models. In this work, we are helping to lay the foundations for understanding how to build more robust networks. Future work will explore defense and robustification in greater depth as well as attacks on generative models trained using natural image datasets such as CIFAR-10 and ImageNet.
ACKNOWLEDGMENTS
This material is in part based upon work supported by the National Science Foundation under Grant No. TWC-1409915. Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
A APPENDIX
A.1 MEAN LATENT VECTOR TARGETED ATTACK
A variant of the single latent vector targeted attack described in Section 4.3, that was not explored in previous work to our knowledge is to take the mean latent vector of many target images and use that vector as xt. This variant is more flexible, in that the attacker can choose different latent properties to target without needing to find the ideal input. For example, in MNIST, the attacker may wish to have a particular line thickness or slant in the reconstructed digit, but may not have such an image available. In that case, by choosing some images of the target class with thinner lines or less slant, and some with thicker lines or more slant, the attacker can find a target latent vector that closely matches the desired properties.
In this case, the attack starts by using fenc to produce the target latent vector, zt, from the chosen target images, x(t).
zt = 1
|x(t)| |x(t)|∑ i=0 fenc(x i (t)). (8)
In this work, we choose to reconstruct “ideal” MNIST digits by taking the mean latent vector of all of the training digits of each class, and using those vectors as xt. Given a target class yt, a set of examples X and their corresponding ground truth labels y, we create a subset x(t) as follows:
x(t) = {xi|xi ∈ X ∧ yi = yt} . (9)
Both variants of this attack appear to be similarly effective, as shown in Figure 15 and Figure 5. The trade-off between the two in these experiments is between the simplicity of the first attack and the flexibility of the second attack.
A.2 EVALUATION RESULTS

After the rebuttal:
The paper contains an interesting set of results (mainly produced after the initial submission), but novelty is limited, and presentation is suboptimal.
For me now the biggest problem now is that the title and the content do not correspond. The authors clearly attack deterministic encoder-decoder models (as described in 3.2), which are not at all the same as generative models, even though many generative models make use of this architecture. A small experiment with sampling is interesting, but does not change the overall focus of the paper. This inconsistency in not acceptable. The whole issue could be resolved for example by simply replacing "generative models" by "encoder-decoder networks" in the title. Then I would tend towards recommending acceptance.
------
Initial review:
The paper describes three approaches to generating adversarial examples for deep encoder-decoder generative networks (trained as VAE or VAE-GAN), and shows a comparative analysis of these. While the phenomenon of adversarial examples in discriminative models is widely known and relatively well studied, I am not aware of previous work on adversarial examples for generative networks, so this work is novel (there is a concurrent work by Tabacof et al. which should be cited, though).  The paper has significantly improved since the initial submission; still, I have a number of remarks on presentation and experimental evaluation. I am in the borderline mode, and may change my rating during the discussion phase.
Detailed comments:
1) The paper is 13 pages long - significantly over the recommended page limit of 8 pages. Reviewers have to read multiple papers, multiple versions of each, it is a lot of work. Large portions of the paper should be shortened and/or moved to the appendix. It is job of the authors to make the paper concise and readable. "in our attempts to be thorough, we have had a hard time keeping the length down" is a bad excuse - it may be hard, but has to be done.
2) I intentionally avoided term "generative model" above because it is not obvious to me if the attacks described by the authors indeed attack generative models. To clarify, the authors train encoder-decoders as generative models (VAE or VAE-GAN), but then remove all stochasticity (sampling) and prior on the latent variables: that is, they treat the models as deterministic encoders-decoders. It is not a big surprise that a deterministic deep network can be easily tricked; it would be much more interesting to see if the probabilistic aspect of generative models makes them more robust to such attacks. Am I missing something? I would like the authors to clarify their view on this and adjust the claims in the paper if necessary.
3) The paper is motivated by possible attacks on a data channel which uses a generative network for compressing information. Description of the attack scenario in 3.1 does not look convincing to me. It takes a huge amount of space and I do not think it adds much to the paper. First, experiments on natural images are necessary to judge if the proposed attack could succeed in a realistic scenario and second, I am not aware of any existing practical applications of VAEs to image compression: attacking JPEG would be much more practical.
4) Experiments are limited to MNIST and, in the latest version, SVHN (which is very nice). While no good generative models of general natural images exist, it is common to evaluate generative models on datasets of faces, so this would be another very natural domain for testing the proposed approach.
Smaller remarks:
1) Usage of "Oracle" in 3.2 does not look appropriate - oracle typically has access to (part of) ground truth, which is not the case here as far as I understand.
2) Beginning of section 4: "All three methods work for any generative architecture that relies on a learned latent representation z" - "are technically applicable to" would be more correct than "work for".
3) 4.1: "confidentally"


TOWARDS INFORMATION-SEEKING AGENTS
We develop a general problem setting for training and testing the ability of agents to gather information efficiently. Specifically, we present a collection of tasks in which success requires searching through a partially-observed environment, for fragments of information which can be pieced together to accomplish various goals. We combine deep architectures with techniques from reinforcement learning to develop agents that solve our tasks. We shape the behavior of these agents by combining extrinsic and intrinsic rewards. We empirically demonstrate that these agents learn to search actively and intelligently for new information to reduce their uncertainty, and to exploit information they have already acquired.
1 INTRODUCTION
Humans possess an innate desire to know and to understand. We seek information actively through behaviors both simple (glancing at a billboard) and elaborate (conducting scientific experiments) (Gottlieb et al., 2013). These qualities equip us to deal with a complex, ever-changing environment. Artificial agents could benefit greatly from similar capacities. Discussion of information seeking behavior in artificial agents dates back at least 25 years (Schmidhuber, 1991). We contribute to this discussion by reformulating and implementing some of the associated ideas, aided by 10-20 years worth of algorithmic and computational advances. To that end, we present a general problem setting for examining the ability of models to seek information efficiently, and show that our models can apply generic information seeking behavior to improve performance in goal-oriented tasks.
Consider the game 20 Questions. The objective is to guess the identity of some arbitrary item by asking no more than twenty yes-or-no questions (that is, collecting no more than 20 bits of information). At each turn, the questioner seeks to split the set of all viable items along some dimension, thereby shrinking the set. The “optimal” question changes from turn to turn, depending heavily on the questions asked previously. The rules of this game demand efficient seeking. Almost all “guessing games”, from 20 Questions to Battleship to Hangman, seem expressly designed to train efficient information seeking, or at least to exploit our intrinsic joy in exercising this skill.
With this in mind, we develop a collection of tasks that can be solved only through efficient acquisition of information. Our tasks vary in difficulty and complexity, but all involve searching an environment iteratively for salient fragments of information (clues) towards the fulfilment of some goal. To necessitate efficient search we impose restrictions on the information that can be acquired at each turn and on the total number of turns that an agent can take. Agents must synthesize separately acquired clues into a more complete representation of their environment in order to succeed. Our tasks are built upon several existing datasets, including cluttered MNIST (Mnih et al., 2014) and CelebA (Liu et al., 2015), as well as a Blockworld dataset of our own design. Through these tasks, using techniques from deep learning and reinforcement learning, we demonstrate how neural agents can be trained to seek information efficiently.
We make several contributions in this paper. First, we promote a subtle but meaningful shift in perspective regarding attention. Models that choose “where to look” (Mnih et al., 2014; Ranzato, 2014) are implicitly asking the world “what’s over there?”, perhaps at the loss of information received by directing the attention to another location. In contrast to this purely observational form of
questioning, we advocate a perspective that supports acquiring information from the world more actively. Rather than asking simply “what’s happening?”, an information-seeking model should be able to ask questions such as “what would happen if...?”. Second, we develop agents that learn to exploit the information they have acquired and to look for more information when they are uncertain about the environment. Third, we show that simple task-agnostic heuristics related to the notion of information gain can be used to improve task-specific performance.
The rest of this paper is organized as follows. In Section 2 we further discuss our motivation and the relations between our problem setting and prior work. In Section 3 we formally describe the problem and the models we have devised to realize information-seeking behavior. Section 4 details our experimental results, with analysis. We conclude in Section 5.
2 RELATED WORK
Information seeking has been studied from a variety of perspectives, including behavioral science, psychology, neuroscience, and machine learning. In neuroscience, for instance, information-seeking strategies are often explained by biases toward novel, surprising, or uncertain events (Ranganath & Rainer, 2003). Information seeking is also a key component in formal notions of fun, creativity, and intrinsic motivation (Schmidhuber, 2010). Information seeking is closely related to the concept of attention. Both mechanisms are uniquely acts of intelligent agents, in that they do not affect the external world per se; rather, they alter an agent’s epistemic state (Gottlieb et al., 2013). Rensink (2000) points out that humans focus attention selectively to acquire information when and where it is needed, and combine attentive fixations to build an internal representation of the environment. Similarly, attention can improve efficiency by ignoring irrelevant features outside of attended regions (Mnih et al., 2014). In this sense, attention can be considered a strategy for information seeking.
Our work thus overlaps with, and draws from, work on neural attention models – a subject which has become prominent in recent years (Larochelle & Hinton, 2010; Bahdanau et al., 2015; Ranzato, 2014; Gregor et al., 2015; Mnih et al., 2014; Sordoni et al., 2016). Larochelle & Hinton (2010), Gregor et al. (2015), and Mnih et al. (2014), for example, develop neural models that “learn where to look” to improve their understanding of visual scenes. Our work relates most closely to Mnih et al. (2014) and Gregor et al. (2015). In the RAM model (Mnih et al., 2014), visual attention is investigated through the problem of maneuvering a small sensor around a larger image in order to perform digit classification in noisy settings. DRAW (Gregor et al., 2015) uses visual attention to improve the performance of a generative model. In our work, we put tighter constraints on the amount of information that can be gathered from the environment and consider more closely whether this restricted capacity is used efficiently. We show that our model can achieve improved classification performance while operating on a significantly tighter information budget than either RAM or DRAW.1
Empirically, we find that a model’s task-specific performance can be improved by adding a taskagnostic objective which encourages it to 1. formulate hypotheses about the state of the environment and 2. ask questions which effectively test the most uncertain hypotheses. This objective, stated more formally in Sec. 3.2, encourages the model to select questions whose answers most significantly reduce the error in the model’s predictions about the answers to other questions that it might ask. In effect, this objective trains the model to simultaneously classify and reconstruct an image.
In a sense, training our models with this objective encourages them to maximize the rate at which they gather information about the environment. There exists a vast literature on applications of information gain measures for artificial curiosity, intrinsically-motivated exploration, and other more precise goals (Storck et al., 1995; Schmidhuber, 2005; Still & Precup, 2012; Hernández-Lobato et al., 2014; Mohamed & Rezende, 2015; Houthooft et al., 2016). One contribution of the current paper is to revisit some of these ideas in light of more powerful algorithmic and computational tools. Additionally, we use these ideas as a means of bootstrapping and boosting task-specific performance. I.e., we treat information seeking and curiosity-driven behavior as a developing ground for fundamental skills that a model can then apply to goal-oriented tasks.
1RAM and DRAW were not optimized for information efficiency, which biases this comparison in our favor. We’re unaware of other existing models against which we can compare the performance of our models on the sorts of tasks we consider.
Current attention models assume that the environment is fully observable and focus on learning to ignore irrelevant information. Conceptually, the information-seeking approach reverses this assumption: the agent exists in a state of incomplete knowledge and must gather information that can only be observed through a restricted set of interactions with the world.
3 PROBLEM DEFINITION AND MODEL DESCRIPTION
We address the information seeking problem by developing a family of models which ask sequences of simple questions and combine the resulting answers in order to minimize the amount of information consumed while solving various tasks. The ability to actively integrate received knowledge into some sort of memory is potentially extremely useful, but we do not focus on that ability in this paper. Presently, we focus strictly on whether a model can effectively reason about observed information in a way that reduces the number of questions asked while solving a task. We assume that a model records all previously asked questions and their corresponding answers, perhaps in a memory whose structure is well-suited to the task-at-hand.
3.1 AN OBJECTIVE FOR INFORMATION-SEEKING AGENTS
We formulate our objective as a sequential decision making problem. At each decision step, the model considers the information it has received up until the current step, and then selects a particular question from among a set of questions which it can ask of the environment. Concurrently, the model formulates and refines a prediction about some unknown aspect(s) of the environment. E.g., while sequentially selecting pixels to observe in an image, the model attempts to predict whether or not the person in the image is wearing a hat.
For the current paper, we make two main simplifying assumptions: 1. the answer to a given question will not change over time, and 2. we can precisely remember previous questions and answers. Assumption 1. holds (at least approximately) in many useful settings and 2. is easily achieved with modern computers. Together, these assumptions allow us to further simplify the problem by eliminating previously-asked questions from consideration at subsequent time steps.
While the precise objectives we consider vary from task-to-task, they all follow the same pattern:
maximize θ E (x,y)∼D
[ E
{(q1,a1),...,(qT ,aT )}∼(πθ,O,x) [ T∑ t=1 Rt(fθ(q1, a1, ..., qt, at), x, y) ]] . (1)
In Eqn. 1, θ indicates the model parameters and (x, y) denotes an observable/unobservable data pair sampled from some distribution D. We assume questions can be asked about x and can not be asked about y. {(q1, a1), ..., (qT , aT )} indicates a sequence of question/answer pairs generated by allowing the policy πθ to ask T questions qt about x, with the answers at provided by an observation function O(x, at)2. Rt(fθ(·), x, y) indicates a (possibly) non-stationary task-specific reward function which we assume to be a deterministic function of the model’s belief state fθ(·) at time t, and the observable/unobservable data x/y. For our tasks, Rt is differentiable with respect to fθ, but this is not required in general. Intuitively, Eqn. 1 says the agent should ask questions about x which most quickly allow it to make good predictions about x and/or y, as measured by Rt.
As a concrete example, (x, y) could be an image/annotation pair sampled from D ≡ CelebA, each question qt could indicate a 4x4 block of pixels in x, O(x, qt) could provide the value of those pixels, and Rt could be the log-likelihood which the model’s belief state fθ assigns to the true value of y after observing the pixels requested by questions {q1, ..., qt} (i.e. {a1, ..., at}).
3.2 TRAINING
We train our models using Generalized Advantage Estimation (Schulman et al. (2016), abbr. GAE), TD(λ) (Sutton & Barto (1998)), and standard backpropagation. We use GAE to train our models how to make better decisions, TD(λ) to train the value function approximators required by GAE, and backpropagation to train our models how to cope with their decisions. When the observation
2I.e., we may be unable to backprop through O(x, a), though its derivatives could be useful when available.
function O(x, qt) is differentiable w.r.t. qt and the policy πθ(qt|h:t) has a suitable form, GAE and TD(λ) can be replaced with a significantly lower variance estimator based on the “reparametrization trick” (Kingma & Welling, 2014; Rezende et al., 2014; Silver et al., 2014).
We train our models by stochastically ascending an approximation of the gradient of Eqn. 1. Considering a fixed (x, y) pair – incorporating the expectation over (x, y) ∼ D is simple – the gradient of Eqn. 1 w.r.t. θ can be written:
∇θ E {(qt,at)}∼(πθ,O,x) [ T∑ t=1 Rt(fθ(q1, a1, ..., qt, at), x, y) ] =
E {(qt,at)}∼(πθ,O,x) [ T∑ t=1 (∇θ log πθ(qt|h:t)Rt: +∇θRt(fθ(h:t+1), x, y)) ] , (2)
in which Rt: refers to the total reward received after asking question qt, and h:t indicates the history of question/answer pairs {(q1, a1), ..., (qt−1, at−1)} observed prior to asking question qt. The gradient in Eqn. 2 can be interpreted as comprising two parts:
∇πθ ≡ E{(qt,at)}∼(πθ,O,x) [ T∑ t=1 ∇θ log πθ(qt|h:t)(Rt: − Vθ(h:t)) ] and (3)
∇fθ ≡ E{(qt,at)}∼(πθ,O,x) [ T∑ t=1 ∇θRt(fθ(h:t+1), x, y) ] , (4)
where we have introduced the approximate value function (i.e. baseline) Vθ(h:t). Roughly speaking, Vθ(h:t) provides an estimate of the expectation of Rt: and is helpful in reducing variance of the gradient estimator in Eqn. 3 (Sutton & Barto, 1998). Intuitively, ∇πθ modifies the distribution of question/answer sequences {(q1, a1), ..., (qT , aT )} experienced by the model, and ∇fθ makes the model better at predicting given the current distribution of experience. Respectively, ∇πθ and ∇ f θ train the model to make better decisions and to cope with the decisions it makes.
We estimate∇fθ directly using standard backpropagation and Monte Carlo integration of the required expectation. In contrast, obtaining useful estimates of ∇πθ is quite challenging and a subject of ongoing research. We use the GAE estimator presented by Schulman et al. (2016), which takes a weighted average of all possible k-step actor-critic estimates of Rt:. Details are available in the supplementary material.
3.3 EXTRINSIC AND INTRINSIC REWARD
The specification of the reward functionRt is a central aspect of sequential decision making problems. Extrinsic rewards incorporate any external feedback useful to solve the problem at hand. For example, Rt may reflect the log-likelihood the model assigns to the true value of the unknown target y, as in Mnih et al. (2014), or some performance score obtained from the external environment, as in Silver et al. (2014). Because extrinsic rewards may be sparse, intrinsically motivated reinforcement learning (Chentanez et al., 2004; Mohamed & Rezende, 2015) aims to provide agents with reward signals that are task-agnostic and motivated rather by internal drives like curiosity.
In our work, we use a reward function Rt(.) = rEt (.) + r I t (.), which sums an extrinsic and intrinsic reward function respectively. We suppose that the belief state fθ comprises a probabilistic model q(x|fθ(.)) of the unobserved world x ∼ D. Therefore, we use an intrinsic reward given by the negative cross-entropy rIt = Ex∼D[log q(x|fθ(h:t))], which encourages the model to form an accurate belief about the world distribution D. Instead of using the same intrinsic reward for each question that has been asked, we reward each question that the model asks by the difference in the rewards rIt+1 − rIt , which is the difference in cross-entropy between the model beliefs after the question has been asked and those prior to the question qt. Intuitively, this intrinsic reward has the effect of more explicitly favoring the questions whose answers provide the most useful information about the underlying world x.
3.4 MODEL ARCHITECTURES FOR INFORMATION SEEKING
We use deep neural networks to represent the functions πθ, fθ, and Vθ described in the preceding sections. Our networks share parameters extensively. Figure 1 illustrates the specific architectures we developed for tasks involving images and generic data. For the tasks we examine, the number of possible questions is moderately sized, i.e. < 1000, and the possible answers to each question can be represented by small vectors, or perhaps even single scalars. Additionally, the response to a particular question will always have the same type, i.e. if question qt produces a 4d answer vector at ≡ O(x, qt) for some x, then it produces a 4d answer vector for all x. Given these assumptions, we can train neural networks whose inputs are tables of (qrepr, answer) tuples, where each tuple provides the answer to a question (if it has been asked), and a question representation, which provides information about the question which was asked. E.g., a simple question representation might be a one-hot vector indicating which question, among a fixed set of questions, was asked. Our networks process the tables summarizing questions and answers which they have observed so far (i.e. a trial history h:t) by vectorizing them and feeding them through one or more hidden layers. We compute quantities used in training from one or more of these hidden layers – i.e. the value function estimate Vθ(h:t), the policy πθ(qt|h:t), and the belief state fθ(h:t). The architecture in Fig. 1a first evaluates a bottom-up network comprising a sequence of convolutional layers topped by a fully-connected layer. Each convolutional layer in this network performs 2x downsampling via strided convolution. For the fully-connected layer we use an LSTM (Hochreiter & Schmidhuber, 1997), which maintains internal state from step to step during a trial. After computing the output of each layer in the bottom-up network, we then evaluate a sequence of layers making up the top-down network. Each layer in the top-down network receives input both from the preceding layer in the top-down network and a partner layer in the bottom-up network (see arrows in (a)). Each convolutional layer in the top-down network performs 2x upsampling via strided convolution.
The input to the bottom-up network is an image, masked to reveal only the pixels whose value the model has previously queried, and a bit mask indicating which pixels are visible. The output of the top-down network has the same spatial shape as the input image, and provides both a reconstruction of the complete input image and the values used to compute πθ(qt|h:t). The value function Vθ(h:t) at time t is computed as a linear function of the bottom-up network’s LSTM state. For labelling tasks, the class prediction fyθ (h:t) is computed similarly. The input reconstruction f x θ (h:t) is taken from the
top-down network’s output. The policy πθ(h:t) is computed by summing pixel-level values from a single channel in the top-down network’s output over regions whose pixels the model has yet to ask about. A softmax is applied to the summed pixel-level values to get the final probabilities for which pixels the model will ask about next.
The fully-connected architecture in Fig. 1b performs similar computations to the convolutional architecture. The input to this model is an observation vector, masked to reveal only the features which have already been requested by the model, and a mask vector indicating which features have been requested. These vectors are concatenated and fed through a sequence of four shared hidden layers linked by shortcut connections. The values fxθ (h:t), f y θ (h:t), Vθ(h:t), and πθ(qt|h:t) are all computed as linear functions of the output of the final shared layer, with a softmax used to make πθ(qt|h:t) a distribution over which feature to request next. We use Leaky ReLU activation functions throughout our networks, and apply layer normalization in all hidden layers to deal with the variable input magnitude caused by large variation in the number of visible pixels/features over the course of a trial. Layer weights are all initialized with a basic orthogonal init. We use the ADAM optimizer (Kingma & Ba, 2015) in all our tests, while working with minibatches of 100 examples each.
4 EXPERIMENTS
4.1 TASK 1: CLUTTERED MNIST
The first task we examine is cluttered MNIST classification as proposed by Mnih et al. (2014). In this task, a model must classify an MNIST digit placed uniformly at random on a 104x104 canvas3. Note that an MNIST digit is 28x28, so the salient information for classifying the digit occupies at most 7.3% of the full canvas. To make the task more challenging, eight pieces of clutter are randomly distributed over the canvas. Each piece is an 8x8 patch extracted from a random location in a randomly selected MNIST digit. We did not include the intrinsic reward signal during these tests.
3The original task uses a 100x100 canvas. We wanted the canvas size to be divisible by 8.
The lowest test errors previously reported on this task were 8.11% with eight four-scale glimpses of size 12x12 for RAM, and 3.36% for DRAW with eight one-scale glimpses of size 12x12. Respectively, these results consume 4608 and 1152 scalar values worth of information from the image. Both methods can cut this information use in half with about 1% increase in error.
The first model we applied to this task was structured precisely as shown in Fig. 1a. At each step of a trial, the model predicted the class of the digit in the image and selected a 4x4 block of pixels whose values it wanted to observe. These pixels were made visible to the model at the beginning of the next step. We allowed the model to view up to 41 4x4 pixel blocks. This makes 656 pixels worth of information, which is roughly comparable to the amount consumed by DRAW with four 12x12 peeks. After 41 peeks our model had 4.5% error on held-out examples from the test set. Qualitatively, this model learned a reasonably efficient search strategy (see Fig. 2) but occasionally failed to find the true digit before time ran out.
Based on the observation that sometimes purely-local information, due simply to bad luck, can be inadequate for solving this task quickly, we provided our model with an additional source of “summary” information. This summary was fed into the model by linearly downsampling the full 104x104 canvas by 8x, and then appending the downsampled image channel-wise to the inputs for appropriately-shaped layers in both the bottom-up and top-down networks. This summary comprised 13x13=169 scalar values. Provided with this information, the task became nearly trivial for our model. It was able to succesfully integrate the summary information into its view of the world and efficiently allocate its low-level perceptual resources to solve the task-at-hand. After just 10 4x4 peeks, this model had 2.5% test error. By 20 peeks, this dropped to 1.5%. After 10 peeks the model had consumed just 329 pixels worth of information – about half that of the most efficient DRAW model.
4.2 TASK 2: POSITIONAL REASONING FOR BLOCKWORLD
We designed BlockWorld to train and test inference capabilities in information-seeking agents. This synthetic dataset consists of 64x64-pixel images that depict elementary shapes of different colors and sizes. Shapes are drawn from the set S = {triangle, square, cross, diamond} and can be colored as C = {green, blue, yellow, red}. The scale of the shapes may vary, with the longest border fixed to be either 12 pixels or 16 pixels.
Distinct “worlds” – the environments for our agents – are generated by sampling three objects at random and placing them on the 64x64 image canvas such that they do not intersect. Objects in each world are uniquely identifiable – we enforce that an image does not contain two objects with the same shape and color. An agent’s goal in Blockworld is to estimate whether a specific positional statement holds between two objects in a given image. Relation statements are structured as triples F = {(s1, r, s2)}, where s1, s2 ∈ S × C are shape descriptions and r is a positional relation from R = {above, below, to the right of, to the left of}. The query (yellow triangle, above, yellow sphere) is an example. Generation of true statements is straightforward: it is sufficient to pick two objects in the image and compare their coordinates to find one relation that holds. We generate negative statements by corrupting the positive ones. Given a statement triple with positive answer, we either change the relation r or one property (either colour or shape) of s1 or s2. Thus, a statement may be false if the relation between two shapes in the world does not hold or if one of the shapes does not appear in the world (but a similar one does).
The input to our model during training is a triple (x, s, y), where s is a 20-dimensional multi-hot vector encoding the particular statement (16 dimensions for color and shape for the two objects and 4 dimensions for the relation) and y is a binary variable encoding its truth value in image x. We approach the task by estimating a conditional policy πθ(qt|h:t, a). Here, each dimension of the statement vector s is treated as an additional channel in the bottom-up and the top-down convolutions. In practice we condition the convolutional layers at a coarser resolution (in order to limit computational costs) as follows: we form a 10x16x16 "statement" feature map by repeating s along the dimensions of the image down-sampled at 4x resolution; then we concatenate these additional channels onto the output of the 4x down-sampling (up-sampling) layer in the bottom-up (top-down) convolutional stacks.
Figure 3 illustrates 13 questions asked by the model in order to assess the truth of the statement “the blue triangle is above the yellow diamond”, with respect to the world pictured in the first row (see
figure caption for more details). At first, the model exhibits exploratory behavior, until it finds the red triangle at the 5th iteration (see the reconstructions in the second row). As it appears on the top of the frame, the model becomes confident of the truth value of the statement. At its 8th question, the yellow cross is found, which entails a drop in the confidence that the statement is true. The model finishes the prediction by tracing the objects in the image. In 10 steps, the model has consumed only 4% of the total information contained in the image (each question is worth 16 pixels). Our model with 20 questions (8% of the original image) achieves an accuracy of 96%, while a similar bottom-up convolutional architecture that takes the whole image as input (upper-bound) gets an accuracy of 98.2%. The cases in which the model makes mistakes correspond to unfruitful exploration, i.e. when the model cannot find the object of interest in the image. We also ran our model but with a random policy of question asking (lower-bound) and got an accuracy of 71%.
4.3 TASK 3: CONDITIONAL CLASSIFICATION FOR CELEBA
We also test our model on a real-word image dataset. CelebA (Liu et al., 2015) is a corpus of thousands of celebrity face images, where each image is annotated with 40 binary facial attributes (e.g., “attractive”, “male”, “gray hair”, “bald”, “has beard”, “wearing lipstick”). We devise a conditional classification task based on this corpus wherein the agent’s goal is to determine whether a face has a given attribute. Through training, an agent should learn to adapt its questioning policy to the distinct attributes that may be queried.
In order to ensure that the learned conditional information-seeking strategy is interpretable, we exclude from the task those attributes whose presence might be ambiguous (such as ‘is old’ and ‘pointy nose’) and query only a subset of 10 attributes that can be discriminated from a specific image region. Our query attributes are “eyeglasses”, “wearing hat”, “narrow eyes”, “wearing earrings”, “mouth slightly open”, “male”, “wearing lipstick”, “young”, “wavy hair”, “bald”. As is common in previous works (Radford et al., 2015), we center-crop and resize images to 64 by 64. In order to condition our policy, we adopt an approach similar to the previous section.
We show the behavior of the model in Figure 4. In this case, the model is pretty effective as it reaches an accuracy of 83.1% after 2 questions (32 pixels, less than 1% of the image), 85.9% after 5 questions (90 pixels) and 87.1% after 20 questions, while the random policy scores 76.5%. The “upper-bound” architecture having access to all the image scores 88.3%.
4.4 TASK 4: GUESSING CHARACTERS IN HANGMAN
We also test our model in a language-based task inspired by the well-known text game Hangman. We sample sub-sequences of 16 characters from the Text8 corpus and train our model to guess all the
characters in the input sequence. At each step, the model guesses a character and the environment shows all the positions in which the selected character appears. If the model asks for a character that is not in the sequence, it suffers a fixed loss of -1. On the contrary, if the character is present, the model gets a reward of +1. The main objective for the model is to maximize the expected reward.
We adapt our architecture by substituting the 2-D convolutions with 1-D convolutions, similar to Zhang et al. (2015). In Figure 5, we report the cumulative distribution of completed games with respect to the number of wrong guessed characters. rand is equipped with a random policy; freq follows the unigram distribution of characters in the training corpus; seek is our model after 10K, 100K and 250K updates respectively.
5 CONCLUSION
In this work we developed a general setting to train and test the ability of agents to seek information. We defined a class of problems in which success requires searching, in a partially-observed environment, for fragments of information that must be pieced together to achieve some goal. We then demonstrated how to combine deep architectures with techniques from reinforcement learning to build agents that solve these tasks. It was demonstrated empirically that these agents learn to search actively and intelligently for new information to reduce their uncertainty, and to exploit information they have already acquired. Information seeking is an essential capability for agents acting in complex, ever-changing environments.

This paper proposes a setting to learn models that will seek information (e.g., by asking question) in order to solve a given task. They introduce a set of tasks that were designed for that goal. They show that it is possible to train models to solve these tasks with reinforcement learning.
One key motivation for the tasks proposed in this work are the existence of games like 20Q or battleships where an agent needs to ask questions to solve a given task. It is quite surprising that the authors do not actually consider these games as potential tasks to explore (beside the Hangman). It is also not completely clear how the tasks have been selected. A significant amount of work has been dedicated in the past to understand the property of games like 20Q (e.g., Navarro et al., 2010) and how humans solve them.  It would interesting to see how the tasks proposed in this work distinguish themselves from the ones studied in the existing literature, and how humans would perform on them.  In particular, Cohen & Lake, 2016m have recently studied the 20 questions games in their paper “Searching large hypothesis spaces by asking questions” where they both evaluate the performance of humans and computer. I believe that this paper would really benefits from a similar study.
Developing the ability of models to actively seek for information to solve a task is a very interesting but challenging problem. In this paper, all of the  tasks require the agent to select a questions from a finite set of clean and informative possibilities. This allows a simpler analysis of how a given agent may perform but at the cost of a reducing the level of noise that would appear in more realistic settings.
This paper also show that by using a relatively standard mix of deep learning models and reinforcement learning, they are able to train agents that can solve these tasks in the way it was intended to. This validates their empirical setting but also may exhibit some of the limitation of their approach; using relatively toy-ish settings with perfect information and a fixed number of questions may be too simple.
While it is interesting to see that their agent are able to perform well on all of their tasks, the absence of baselines limit the conclusions we can draw from these experiments. For example in the Hangman experiment, it seems that the frequency based model obtains promising performance. It would interesting to see how good are baselines that may use the co-occurrence of letters or the frequency of character n-grams.
Overall, this paper explores a very interesting direction of research and propose a set of promising tasks to test the capability of a model to learn from asking question. However, the current analysis of the tasks is a bit limited, and it is hard to draw any conclusion from them. It would be good if the paper would focus more on how humans perform on these tasks, on strong simple baselines and on more tasks related to natural language (since it is one of the motivation of this work) rather than on solving them with relatively sophisticated models.