EXTRACTIVE DOCUMENT SUMMARIZATION
We present two novel and contrasting Recurrent Neural Network (RNN) based architectures for extractive summarization of documents. The Classifier based architecture sequentially accepts or rejects each sentence in the original document order for its membership in the final summary. The Selector architecture, on the other hand, is free to pick one sentence at a time in any arbitrary order to piece together the summary. Our models under both architectures jointly capture the notions of salience and redundancy of sentences. In addition, these models have the advantage of being very interpretable, since they allow visualization of their predictions broken up by abstract features such as information content, salience and redundancy. We show that our models reach or outperform state-of-the-art supervised models on two different corpora. We also recommend the conditions under which one architecture is superior to the other based on experimental evidence.
1 INTRODUCTION
Document summarization is an important problem that has many applications in information retrieval and natural language understanding. Summarization techniques are mainly classified into two categories: extractive and abstractive. Extractive methods aim to select salient snippets, sentences or passages from documents, while abstractive summarization techniques aim to concisely paraphrase the information content in the documents.
A vast majority of the literature on document summarization is devoted to extractive summarization. Traditional methods for extractive summarization can be broadly classified into greedy approaches (e.g., Carbonell & Goldstein (1998)), graph based approaches (e.g., Radev & Erkan (2004)) and constraint optimization based approaches (e.g., McDonald (2007)).
Recently, neural network based approaches have become popular for extractive summarization. For example, Kageback et al. (2014) employed the recursive autoencoder (Socher et al. (2011)) to summarize documents, producing best performance on the Opinosis dataset (Ganesan et al. (2010)). Yin & Pei (2015) applied Convolutional Neural Networks (CNN) to project sentences to continuous vector space and then select sentences by minimizing the cost based on their ‘prestige’ and ‘diverseness’, on the task of multi-document extractive summarization. Another related work is that of Cao et al. (2016), who address the problem of query-focused multi-document summarization using query-attention-weighted CNNs.
Recently, with the emergence of strong generative neural models for text Bahdanau et al. (2014), abstractive techniques are also becoming increasingly popular (Rush et al. (2015), Nallapati et al. (2016b) and Nallapati et al. (2016a)). Despite the emergence of abstractive techniques, extractive techniques are still attractive as they are less complex, less expensive, and generate grammatically
and semantically correct summaries most of the time. In a very recent work, Cheng & Lapata (2016) proposed an attentional encoder-decoder for extractive single-document summarization and trained it on Daily Mail corpus, a large news data set, achieving state-of-the-art performance. Like Cheng & Lapata (2016), our work also focuses only on sentential extractive summarization of single documents using neural networks.
2 TWO ARCHITECTURES
Our architectures are motivated by two intuitive strategies that humans tend to adopt when they are tasked with extracting salient sentences in a document. The first strategy, which we call Classify, involves reading the whole document once to understand its contents, and then traversing through the sentences in the original document order and deciding whether or not each sentence belongs to the summary. The other strategy that we call Select involves memorizing the whole document once as before, and then picking sentences that should belong to the summary one at a time, in any order of one’s choosing. Qualitatively, the latter strategy appears to be a better one since it allows us to make globally optimal decisions at each step. While it may be harder for humans to follow this strategy since we are forgetful by nature, one may expect that the Select strategy could deliver an advantage for the machines, since ’forgetfulness’ is not a real ‘concern’ for them. In this work, we will explore both the strategies empirically and make a recommendation on which strategy is optimal under what conditions.
Broadly, our Classify architecture involves an RNN based sequence classification model that sequentially classifies each sentence into 0/1 binary labels, while the Select architecture involves a generative model that sequentially generates the indices of the sentences that should belong to the summary. We will first discuss the components shared by both the architectures and then we will present each architecture separately.
Shared Building Blocks: Both architectures begin with word-level bidirectional Gated Recurrent Unit (GRU) based RNNs (Chung et al. (2014)) run independently over each sentence in the document, where each time-step of the RNN corresponds to a word index in the sentence. The average pooling of the concatenated hidden states of this bidirectional RNN is then used as an input to another bidirectional RNN whose time steps correspond to sentence indices in the document. The concatenated hidden states ‘h’ from the forward and backward layers of this second layer of bidirectional RNN at each time step are used as corresponding sentence representations. We also use the average pooling of the sentence representations as the document representation ‘d’. Both architectures also maintain a dynamic summary representation ‘s’ whose estimation is architecture dependent. Models under each architecture compute a score for each sentence towards its summary membership. Motivated by the need to build humanly interpretable models, we compute this score by explicitly modeling abstract features such as salience, novelty and information content as shown below:
score(hj , sj ,d,pj) = wcσ(W T c hj) #(content richness)
+wsσ(cos(hj ,d)) #(salience w.r.t. document)
+wpσ(W T p pj) #(positional importance)
−wrσ(cos(hj , sj)) #(redundancy w.r.t. summary) +b), #(bias term) (1)
where j is the index of the sentence in the document, pj is the positional embedding of the sentence computed by concatenation of embeddings corresponding to forward and backward position indices of the sentence in the document; cos(a,b) is the standard cosine similarity between the two vectors a and b; Wc and Wp are parameter vectors to model content richness and positional importance of sentences respectively; and wc, ws, wp and wr are scalar weights to model relative importance of various abstract features, and are learned automatically. In the equation above, the abstract feature that each term represents is printed against the term in comments. In other words, assuming the importance weights are positive, in order for a sentence to score high for summary membership, it needs to be highly salient, content rich and occupy important positions in the document, while being least redundant with respect to the summary generated till that point. Note that our formulation of the scoring function simultaneously captures both salience of the sentence hj with respect to the document d as well as its redundancy with respect to the current summary representation sj . In
the next subsection, we will describe the estimation of dynamic summary representation sj and the formulation of the cost function for training in each architecture. We will also present shallow and deep models under each architecture.
2.1 CLASSIFIER ARCHITECTURE
In this architecture, we sequentially visit each sentence in the original document order and binaryclassify the sentence in terms of whether it belongs to the summary. The probability of the sentence belonging to the summary, P (yj = 1) is given as follows:
P (yj = 1|hj , sj ,d,pj) = σ(score(hj , sj ,d,pj) (2) The objective function to minimize at training is the negative log-likelihood of the training data labels:
`(W,w,b) = − N∑
d=1 Nd∑ j=1 (ydj logP (y d j = 1|hdj , sdj ,dd) + (1− ydj ) log(1− P (ydj = 1|hdj , sdj ,dd))
where N is the size of the training corpus and Nd is the number of sentences in the document d. Now the only detail that remains is how the dynamic summary representation sj is estimated. This is where the shallow and deep models under this architecture differ, and we describe them below.
Shallow Model: In the shallow model, we estimate the dynamic summary representation as the running sum of the representations of the sentences visited so far weighted by their probability of being in the summary.
sj = j−1∑ i=1 hiyi #(training time)
sj = j−1∑ i=1 hiP (yi = 1|hi, si,d) #(test time)
(3)
In other words, at training time, since the summary membership of sentences is known, the probabilities are binary, whereas at test time we use a weighted pooling based on the estimated probability
that each sentence belongs to the summary. There is no need to normalize the summary representations since the cosine similarity metric we use in the scoring function of Eq. (1) automatically normalizes them.
Deep Model: In the deep model, we introduce an additional layer of unidirectional sentence-level GRU-RNN that takes as input the sentence representations hj at each time-step. The hidden state of the new GRU ĥj = GRU(hj) is used as a replacement for sentence representation hj in computing summary membership scores using Eq. (1) as well as in computing the dynamic summary representation using Eq. (3). The main idea behind using this additional layer of GRU is to allow a greater degree of non-linearity in computing the summary representation.
The graphical representations of the shallow and deep models under the Classifier architecture are displayed in Figure 1 with their full set of dependencies.
2.2 SELECTOR ARCHITECTURE
In this architecture, the models do not make decisions in the sequence of sentence ordering; instead, they pick one sentence at a time in an order that they deem fit. The act of picking a sentence is cast as a sequential generative model in which one sentence-index is emitted at each time step that maximizes the score in Eq. 1. Accordingly, the probability of picking a sentence with index I(j) = k ∈ {1, . . . , Nd} at time-step j is given by the softmax over the scoring function:
P (I(j) = k|sj ,hk,d) = exp(score(hk, sj ,d,pk))∑
l∈{1,...,Nd} exp(score(hl, sj ,d,pl)) (4)
The loss function in this case is the negative log-likelihood of the selected sentences in the ground truth data as shown below.
`(W,w,b) = − N∑
d=1 Md∑ j=1 logP (I(j)(d)|hI(j)(d) , sdj ,dd) (5)
where Md is the number of sentences selected in the ground truth of document d, {I(1)(d), . . . , I(Md)(d)} is the ordered list of selected sentence indices in the ground truth of document d. The dependence of the loss function on the order of the selected sentences can be gauged by the fact that the probability of selecting a sentence at time step j depends on the dynamic summary representation sj , which is estimated based on the all sentences selected up to time step j − 1. At test time, at each time-step, the model emits the index of the sentence that has the best score given the current summary representation as shown below.
I(j) = arg max k∈{1,...,Nd} score(hk, sj ,d,pk) (6)
The estimation of dynamic summary representation is done differently for the shallow and deep selector models as described below.
Shallow Model: In this model, we sum the representations of the selected sentences until the time step j as the dynamic summary representation. This is true for both training time and test time.
sj = j−1∑ i=1 hI(i). (7)
Deep Model: In the deep model, we introduce an additional GRU-RNN whose time steps correspond to the sentence index emission events. At each time-step, it takes as input the representation of the previously selected sentence hI(j−1), and computes a new hidden state ĥj = GRU(hI(j−1)). Unlike the shallow model that maintains a separate vector for summary representation sj , we use ĥj as the summary representation sj at time step j. This makes sense for the case of the Selector architecture since both at training and test time we make hard decisions of sentence selection, with the effect that the hidden state of the new GRU can capture a non-linear aggregation of the sentences selected until time step j − 1.
Fig. 2 shows the graphical representation of the Selector architecture with all the dependencies between the nodes. The architecture is the same for both shallow and deep models with the only difference being that the simple summary representation in the former is replaced with a gated recurrent unit in the latter.
3 RELATED WORK
Previous researcher such as Shen et al. (2007) have proposed modeling extractive document summarization as a sequence classification problem using Conditional Random Fields. Our approach is different from theirs in the sense that we use RNNs in our model that do not require any handcrafted features for representing sentences and documents.
The Selector architecture broadly involves ranking of sentences by some criterion, therefore does correspond to traditional methods for extractive summarization such as TextRank (Mihalcea & Tarau (2004)) that also involve ranking of sentences by salience and novelty. However, to the best of our knowledge, our Selector framework is a novel deep learning framework for extractive summarization. Broader efforts are being made in the deep learning community to build more sophisticated sequence to sequence models towards the objective of automatically learning complex tasks such as sorting sequences (Oriol Vinyals (2015); Graves et al. (2014)), but their utility for extractive summarization remains to be explored.
In the deep learning framework, the extractive summarization work of Cheng & Lapata (2016) is the closest to our work. Their model is based on an encoder-decoder approach where the encoder learns the representation of sentences and documents while the decoder classifies each sentence using an attention mechanism. Broadly, their model is also in the Classifier framework, but architecturally, our approaches are different. While their approach can be termed as a multi-pass approach where both the encoder and decoder consume the same sentence representations, our approach is a deep one where the representations learned by the bidirectional GRU encoder are in turn consumed by the Classifier or Selector models. Another key difference between our work and theirs is that unlike our unsupervised greedy approach to convert abstractive summaries to extractive labels, Cheng & Lapata (2016) chose to train a separate supervised classifier using manually created labels on a subset of the data. This may yield more accurate gold extractive labels which may help boost the performance of their models, but incurs additional annotation costs.
4 EXPERIMENTS AND RESULTS
Pseudo ground-truth generation: In order to train our extractive Classifier and Selector models, for each document we need ground truth in the form of sentence-level binary labels and ordered list of selected sentences respectively. However, most summarization corpora only contain human written abstractive summaries as ground truth. To solve this problem, we use an unsupervised approach to convert the abstractive summaries to extractive labels. Our approach is based on the idea that the selected sentences from the document should be the ones that maximize the Rouge score with respect to gold abstractive summaries. Since it is computationally expensive to find a globally optimal subset of sentences that maximizes the Rouge score, we employ a greedy approach, where we add one sentence at a time incrementally to the summary, such that the Rouge score of the current set of selected sentences is maximized with respect to the entire gold summary. We stop adding sentences when either none of the remaining candidate sentences improves the Rouge score upon addition to the current summary set or when the maximum summary length is reached. We return this ordered list of sentences as the ground-truth for the Selector architecture. The ordered list is converted into binary summary-membership labels that are consumed by the Classifier architecture for training.
We note that similar approaches have been employed by other researchers such as Svore et al. (2007) to handle the problem of converting abstractive summaries to extractive ground truth. We would also like to point readers to a recent work by Cao et al. (2015) that proposes an ILP based approach to solve this problem optimally. Since this is not the focus of this work, we chose a simple greedy algorithm.
Corpora: For our experiments, we used the Daily Mail corpus originally constructed by Hermann et al. (2015) for the task of passage-based question answering, and re-purposed for the task of document summarization as proposed in Cheng & Lapata (2016) for extractive summarization and Nallapati et al. (2016a) for abstractive summarization. Overall, we have 196,557 training documents, 12,147 validation documents and 10,396 test documents from the Daily Mail corpus. On average, there are about 28 sentences per document in the training set, and an average of 3-4 sentences in the reference summaries. The average word count per document in the training set is 802.
We also used the DUC 2002 single-document summarization dataset1 consisting of 567 documents as an additional out-of-domain test set to evaluate our models.
Evaluation: In our experiments below, we evaluate the performance of our models using different variants of the Rouge metric2 computed with respect to the gold abstractive summaries. Following Cheng & Lapata (2016), we use limited length Rouge recall at 75 bytes of summary as well as 275 bytes on the Daily Mail corpus. On DUC 2002 corpus, following the official guidelines, we use limited length Rouge recall at 75 words. We report the scores from Rouge-1, Rouge-2 and RougeL, which are computed using matches of unigrams, bigrams and longest common subsequences respectively, with the ground truth summaries.
Baselines: On all datasets, we use Lead-3 model, which simply produces the leading three sentences of the document as the summary, as a baseline. On the Daily Mail and DUC 2002 corpora, we also report performance of LReg, a feature-rich logistic classifier used as a baseline by Cheng & Lapata (2016). On DUC 2002 corpus, we report several baselines such as Integer Linear Programming based approach (Woodsend & Lapata (2010)), and graph based approaches such as TGRAPH (Parveen et al. (2015)) and URANK (Wan (2010)) which achieve very high performance on this corpus. In addition, we also compare with the state-of-the art deep learning supervised extractive model from Cheng & Lapata (2016).
Experimental Settings: We used 100-dimensional word2vec (Mikolov et al. (2013)) embeddings trained on the Daily Mail corpus as our embedding initialization. We limited the vocabulary size to 150K and the maximum sentence length to 50 words, to speed up computation. We fixed the model hidden state size at 200. We used a batch size of 32 at training time, and employed adadelta (Zeiler (2012)) to train our model. We employed gradient clipping and L-2 regularization to prevent overfitting and an early stopping criterion based on validation cost.
1http://www-nlpir.nist.gov/projects/duc/guidelines/2002.html 2http://www.berouge.com/Pages/default.aspx
At test time, for the Classifier models we pick sentences sorted by the predicted probabilities until we exceed the length limit as determined by the Rouge metric. Likewise, we allow the Selector models to emit sentence indices until the desired summary length is reached. For the Selector model, we also make sure the emitted sentence ids are not repeated across time steps by traversing down the sorted predicted probabilities of the softmax layer at each time step until we reach a sentence-id that was not emitted before.
We note that it is possible to optimize the Classifier performance at test time using the Viterbi algorithm to compute the best sequence of labels, subject to the Markovian assumptions of the architecture and model. Similarly, it is also possible to further boost the Selector’s performance by using beam search at test time. However, in this work we used greedy classification/selection for inference since our primary interest is in comparing the two architectures, and our choice allows us to make a fair apples-to-apples comparison.
Results on Daily Mail corpus: Table 1 shows the performance comparison of our models with state-of-the-art model of Cheng & Lapata (2016) and other baselines on the DailyMail corpus using Rouge recall at two different summary lengths.
The results show that contrary to our initial expectation, the Classifier architecture is superior to the Selector architecture. Within each architecture, the deeper models are better performing than the shallower ones. Our deep classifier model outperforms Cheng & Lapata (2016) with a statistically significant margin at 75 bytes, while matching their model at 275 bytes. One potential reason our models do not consistently outperform the extractive model of Cheng & Lapata (2016) is the additional supervised training they used to create sentence-level extractive labels to train their model. Our models instead use an unsupervised greedy approximation to create extractive labels from abstractive summaries, and as a result, may generate noisier ground truth than theirs.
Results on the Out-of-Domain DUC 2002 corpus: We also evaluated the models trained on the DailyMail corpus on the out-of-domain DUC 2002 set as shown in Table 2. The performance trend is similar to that on Daily Mail. Our best model, Deep Classifier is again statistically on par with the model of Cheng & Lapata (2016). However, both models perform worse than graph-based TGRAPH (Parveen et al. (2015)) and URANK (Wan (2010)) algorithms, which are the state-of-the-art models on this corpus. Deep learning based supervised models such as ours and that of Cheng & Lapata (2016) perform very well on the domain they are trained on, but may suffer from domain adaptation issues when tested on a different corpus such as DUC 2002.
5 DISCUSSION
Impact of Document Structure: In all our experiments thus far, the classifier architecture has proven superior to the selector architecture. We conjecture that decision making in the same sequence as the original sentence ordering is perhaps advantageous in document summarization since there is a smooth sequential discourse structure in news stories starting with the main highlights of the story in the beginning, more elaborate description in the middle and ending with conclusive remarks. If this is true, then in scenarios where sentence ordering is less structured, the selector
architecture should be superior since it has freedom to select salient sentences in any arbitrary order. Such scenarios actually do occur in practice, e.g., summarization of a cluster of tweets on a topic where there is no specific discourse structure between individual tweets, or in multi-document summarization where a pair of sentences across document boundaries have no specific ordering. In order to test this hypothesis, we simulated such data in the Daily Mail corpus by randomly shuffling the sentences in each document in the training set and retraining models under both the architectures, and evaluating them on the original test sets. The results, summarized in Table 3, show that the Classifier architecture suffers bigger losses than the Selector architecture when the document structure is destroyed. In fact, the Selector architecture performs slightly better than the Classifier architecture when trained on the shuffled data, indicating that our hypothesis may indeed be true.
Qualitative Analysis: One of the advantages of our model design is teasing out various abstract features for the sake of interpretability of system predictions. In the appendix, we present a visualization (see Fig. 3 in the Appendix) of the system predictions based on the scores for various abstract features listed in Eq. (1). We also present the learned importance weights of these features in Table 4. A few representative documents are also presented in the appendix highlighting the sentences chosen by our models for summarization.
6 CONCLUSION AND FUTURE WORK
In this work, we propose two neural architectures for extractive summarization. Our proposed models under these architectures are not only very interpretable, but also achieve state-of-the-art performance on two different data sets. We also empirically compare our two frameworks and suggest conditions under which each of them can deliver optimal performance.
As part of our future work, we plan to further investigate the applicability of the novel Selector architecture to relatively less structured summarization problems such as summarization of multiple documents or topical clusters of tweets. In addition, we also intend to perform additional experiments on the Daily Mail dataset such as incorporating beam search in both model inference as well in pseudo ground truth generation that may result in further performance improvements.
7 APPENDIX
In this section, we will present some additional qualitative and quantitative analysis of our models that we hope will shed some light on their behavior.
7.1 VISUALIZATION OF MODEL OUTPUT
In addition to being state-of-the-art performers, our models have the additional advantage of being very interpretable. The clearly separated terms in the scoring function (see Eqn. 1) allow us to tease out various factors responsible for the classification/selection of each sentence. This is illustrated in Figure 3, where we display a representative document from our validation set along with normalized scores from each abstract feature from the deep classifier model. Such visualization is especially useful in explaining to the end-user the decisions made by the system.
7.2 LEARNED IMPORTANCE WEIGHTS
We display in Table 4 the learned importance weights corresponding to various abstract features for deep sentence selector. Confirming our intuition, the model learns that salience and redundancy are the most important predictive features for summary membership of a sentence, followed by positional feature and content based feature. Further, when the same model is trained on documents with randomly shuffled sentences, it learns very small weight for the positional features, which is exactly what one expects.
7.3 ABLATION EXPERIMENTS
We evaluated the performance of the deep selector and deep classifier models on the validation set by deleting one abstract feature at a time from the model, with replacement. The performance numbers, displayed in Table 5, show that removing any of the features results in a small loss in performance. Note that the priority of features in the ablation experiments need not correspond to their priority in terms of learned weights in Table 4, since feature correlations may affect the two metrics differently. For the deep classifier, content and redundancy seem to matter the most while for the deep selector, dropping positional features hurts the most. Based on this analysis, we plan to investigate more thoroughly the reasons behind the poor ablation performance of salience and redundancy in the classifier and selector models respectively.
7.4 REPRESENTATIVE DOCUMENTS AND EXTRACTIVE SUMMARIES
We display a couple of representative documents, one each from the Daily Mail and DUC corpora, highlighting the sentences chosen by deep classifier and comparing them with the gold summaries in Table 6. The examples demonstrate qualitatively that the model performs a reasonably good job in identifying the key messages of a document.

This paper presents two RNN architectures for extractive document summarization. The first one, Classifier, takes into account the order in which sentences appear in the original document, whereas the second one, Selector, chooses sentences in an arbitrary order. For each architecture, the concatenated RNN hidden state from a sentence forward and backward pass  is used as features to compute a score that captures content richness, salience, positional importance, and redundancy. Both models are trained in a supervised manner, so the authors used "pseudo-ground truth generation" to create training data from abstractive summaries. Experiments show that the Classifier model performs better, and it achieves near state-of-the-art performance for some evaluation metrics.
The proposed model is in general an extension of Cheng and Lapata, 2016. Unfortunately, the performance is only slightly better or sometimes even worse. The authors mentioned that one key difference how they transform abstractive summaries to become gold labels for the supervised method. However, in the experiment results, the authors described that one potential reason their models do not consistently outperform the extractive model of Cheng & Lapata, 2016 is that the unsupervised greedy approximation may generate noisier ground truth labels than Cheng & Lapata. Is there a reason to construct the training data similar to Cheng & Lapata, if that turns out to be a better method?
In order for the proposed models to be convincing, they need to outperform this baseline that's very similar to the proposed methods more consistently, since the main contribution is improved neural architectures for extractive document summarization.


DEEP UNSUPERVISED LEARNING THROUGH SPATIAL CONTRASTING
Convolutional networks have marked their place over the last few years as the best performing model for various visual tasks. They are, however, most suited for supervised learning from large amounts of labeled data. Previous attempts have been made to use unlabeled data to improve model performance by applying unsupervised techniques. These attempts require different architectures and training methods. In this work we present a novel approach for unsupervised training of Convolutional networks that is based on contrasting between spatial regions within images. This criterion can be employed within conventional neural networks and optimized using standard techniques such as SGD and backpropagation, thus complementing supervised methods.
1 INTRODUCTION
For the past few years convolutional networks (ConvNets, CNNs) LeCun et al. (1998) have proven themselves as a successful model for vision related tasks Krizhevsky et al. (2012) Mnih et al. (2015) Pinheiro et al. (2015) Razavian et al. (2014). A convolutional network is composed of multiple convolutional and pooling layers, followed by a fully-connected affine transformations. As with other neural network models, each layer is typically followed by a non-linearity transformation such as a rectified-linear unit (ReLU). A convolutional layer is applied by cross correlating an image with a trainable weight filter. This stems from the assumption of stationarity in natural images, which means that parameters learned for one local region in an image can be shared for other regions and images.
Deep learning models, including convolutional networks, are usually trained in a supervised manner, requiring large amounts of labeled data (ranging between thousands to millions of examples per-class for classification tasks) in almost all modern applications. These models are optimized using a variant of stochastic-gradient-descent (SGD) over batches of images sampled from the whole training dataset and their ground truth-labels. Gradient estimation for each one of the optimized parameters is done by back propagating the objective error from the final layer towards the input. This is commonly known as ”backpropagation” Rumelhart et al..
In early works, unsupervised training was used as a part of pre-training procedure to obtain an effective initial state of the model. The network was later fine-tuned in a supervised manner as displayed by Hinton (2007). Such unsupervised pre-training procedures were later abandoned, since they provided no apparent benefit over other initialization heuristics in more careful fully supervised training regimes. This led to the de-facto almost exclusive usage of neural networks in supervised environments.
In this work we will present a novel unsupervised learning criterion for convolutional network based on comparison of features extracted from regions within images. Our experiments indicate that by
using this criterion to pre-train networks we can improve their performance and achieve state-ofthe-art results.
2 PREVIOUS WORKS
Using unsupervised methods to improve performance have been the holy grail of deep learning for the last couple of years and vast research efforts have been focused on that. We hereby give a short overview of the most popular and recent methods that tried to tackle this problem.
AutoEncoders and reconstruction loss These are probably the most popular models for unsupervised learning using neural networks, and ConvNets in particular. Autoencoders are NNs which aim to transform inputs into outputs with the least possible amount of distortion. An Autoencoder is constructed using an encoder G(x;w1) that maps an input to a hidden compressed representation, followed by a decoder F (y;w2), that maps the representation back into the input space. Mathematically, this can be written in the following general form:
x̂ = F (G(x;w1);w2)
The underlying encoder and decoder contain a set of trainable parameters that can be tied together and optimized for a predefined criterion. The encoder and decoder can have different architectures, including fully-connected neural networks, ConvNets and others. The criterion used for training is the reconstruction loss, usually the mean squared error (MSE) between the original input and its reconstruction Zeiler et al. (2010)
min‖x− x̂‖2
This allows an efficient training procedure using the aforementioned backpropagation and SGD techniques. Over the years autoencoders gained fundamental role in unsupervised learning and many modification to the classic architecture were made. Ng (2011) regularized the latent representation to be sparse, Vincent et al. (2008) substituted the input with a noisy version thereof, requiring the model to denoise while reconstructing. Kingma et al. (2014) obtained very promising results with variational autoencoders (VAE). A variational autoencoder model inherits typical autoencoder architecture, but makes strong assumptions concerning the distribution of latent variables. They use variational approach for latent representation learning, which results in an additional loss component which required a new training algorithm called Stochastic Gradient Variational Bayes (SGVB). VAE assumes that the data is generated by a directed graphical model p(x|z) and require the encoder to learn an approximation qw1(z|x) to the posterior distribution pw2(z|x) where w1 and w2 denote the parameters of the encoder and decoder. The objective of the variational autoencoder in that case has the following form:
L(w1, w2, x) = −DKL(qw1(z|x)||pw2(z)) + Eqw1 (z|x) ( log pw2(x|z) ) Recently, a stacked set of denoising autoencoders architectures showed promising results in both semi-supervised and unsupervised tasks. A stacked what-where autoencoder by Zhao et al. (2015) computes a set of complementary variables that enable reconstruction whenever a layer implements a many-to-one mapping. Ladder networks by Rasmus et al. (2015) - use lateral connections and layer-wise cost functions to allow the higher levels of an autoencoder to focus on invariant abstract features.
Exemplar Networks: The unsupervised method introduced byDosovitskiy et al. (2014) takes a different approach to this task and trains the network to discriminate between a set of pseudo-classes. Each pseudo-class is formed by applying multiple transformations to a randomly sampled image patch. The number of pseudo-classes can be as big as the size of the input samples. This criterion ensures that different input samples would be distinguished while providing robustness to the applied transformations. In this work we will explore an alternative method with a similar motivation.
Context prediction Another method for unsupervised learning by context was introduced by Doersch et al. (2015). This method uses an auxiliary criterion of predicting the location of an image patch given another from the same image. This is done by classification to 1 of 9 possible locations. Although the work of Doersch et al. (2015) and ours both use patches from an image to perform unsupervised learning, the methods are quite different. Whereas the former used a classification criterion over the spatial location of each patch within a single image, our work is concerned with comparing patches from several images to each other. We claim that this encourages discriminability between images (which we feel to be important aspect of feature learning), and was not an explicit goal in previous work.
Adversarial Generative Models: This a recently introduced model that can be used in an unsupervised fashion Goodfellow et al. (2014). Adversarial Generative Models uses a set of networks, one trained to discriminate between data sampled from the true underlying distribution (e.g., a set of images), and a separate generative network trained to be an adversary trying to confuse the first network. By propagating the gradient through the paired networks, the model learns to generate samples that are distributed similarly to the source data. As shown by Radford et al. (2015),this model can create useful latent representations for subsequent classification tasks.
Sampling Methods: Methods for training models to discriminate between a very large number of classes often use a noise contrasting criterion. In these methods, roughly speaking, the posterior probability P (t|yt) of the ground-truth target t given the model output on an input sampled from the true distribution yt = F (x) is maximized, while the probability P (t|yn) given a noise measurement y = F (n) is minimized. This was successfully used in a language domain to learn unsupervised representation of words. The most noteworthy case is the word2vec model introduced by Mikolov et al. (2013). When using this setting in language applications, a natural contrasting noise is a smooth approximation of the Unigram distribution. A suitable contrasting distribution is less obvious when data points are sampled from a high dimensional continuous space, such as the case of image patches.
2.1 PROBLEMS WITH CURRENT APPROACHES
Only recently the potential of ConvNets in an unsupervised environment began to bear fruit, still we believe it is not fully uncovered.
The majority of unsupervised optimization criteria currently used are based on variations of reconstruction losses. One limitation of this fact is that a pixel level reconstruction is non-compliant with the idea of a discriminative objective, which is expected to be agnostic to low level information in the input. In addition, it is evident that MSE is not best suited as a measurement to compare images, for example, viewing the possibly large square-error between an image and a single pixel shifted copy of it. Another problem with recent approaches such as Rasmus et al. (2015); Zeiler et al. (2010) is their need to extensively modify the original convolutional network model. This leads to a gap between unsupervised method and the state-of-the-art, supervised, models for classification - which can hurt future attempt to reconcile them in a unified framework, as well as efficiently leverage unlabeled data with otherwise supervised regimes.
3 LEARNING BY COMPARISONS
The most common way to train NN is by defining a loss function between the target values and the network output. Learning by comparison approaches the supervised task from a different angle. The main idea is to use distance comparisons between samples to learn useful representations. For example, we consider relative and qualitative examples of the form X1 is closer to X2 than X1 is to X3. Using a comparative measure with neural network to learn embedding space was introduced in the “Siamese network” framework by Bromley et al. (1993) and later used in the works of Chopra et al. (2005). One use for this methods is when the number of classes is too large or expected to vary over time, as in the case of face verification, where a face contained in an image has to compared against another image of a face. This problem was recently tackled by Schroff et al. (2015) for training a convolutional network model on triplets of examples. There, one image served as an anchor x, and an additional pair of images served as a positive example x+ (containing an instance
of the face of the same person) together with a negative example x−, containing a face of a different person. The training objective was on the embedded distance of the input faces, where the distance between the anchor and positive example is adjusted to be smaller by at least some constant α from the negative distance. More precisely, the loss function used in this case was defined as
L(x, x+, x−) = max {‖F (x)− F (x+)‖2 − ‖F (x)− F (x−)‖2 + α, 0} (1)
where F (x) is the embedding (the output of a convolutional neural network), and α is a predefined margin constant. Another similar model used by Hoffer & Ailon (2015) with triplets comparisons for classification, where examples from the same class were trained to have a lower embedded distance than that of two images from distinct classes. This work introduced a concept of a distance ratio loss, where the defined measure amounted to:
L(x, x+, x−) = e−‖F (x)−F (x+)‖2
e−‖F (x)−F (x+)‖2 + e−‖F (x)−F (x−)‖2 (2)
This loss has a flavor of a probability of a biased coin flip. By ‘pushing’ this probability to zero, we express the objective that pairs of samples coming from distinct classes should be less similar to each other, compared to pairs of samples coming from the same class. It was shown empirical by Balntas et al. (2016) to provide better feature embeddings than the margin based distance loss 1
4 OUR CONTRIBUTION: SPATIAL CONTRASTING
One implicit assumption in convolutional networks, is that features are gradually learned hierarchically, each level in the hierarchy corresponding to a layer in the network. Each spatial location within a layer corresponds to a region in the original image. It is empirically observed that deeper layers tend to contain more ‘abstract’ information from the image. Intuitively, features describing different regions within the same image are likely to be semantically similar (e.g. different parts of an animal), and indeed the corresponding deep representations tend to be similar. Conversely, regions from two probably unrelated images (say, two images chosen at random) tend to be far from each other in the deep representation. This logic is commonly used in modern deep networks such as Szegedy et al. (2015) Lin et al. (2013) He et al. (2015), where a global average pooling is used to aggregate spatial features in the final layer used for classification.
Our suggestion is that this property, often observed as a side effect of supervised applications, can be used as a desired objective when learning deep representations in an unsupervised task. Later, the resulting representation can be used, as typically done, as a starting point or a supervised learning task. We call this idea which we formalize below Spatial contrasting. The spatial contrasting criterion is similar to noise contrasting estimation Gutmann & Hyvärinen (2010) Mnih & Kavukcuoglu (2013), in trying to train a model by maximizing the expected probability on desired inputs, while minimizing it on contrasting sampled measurements.
4.1 FORMULATION
We will concern ourselves with samples of images patches x̃(m) taken from an image x. Our convolutional network model, denoted by F (x), extracts spatial features f so that f (m) = F (x̃(m)) for an image patch x̃(m). We will also define P (f1|f2) as the probability for two features f1, f2 to occur together in the same image. We wish to optimize our model such that for two features representing patches taken from the same image x̃(1)i , x̃ (2) i ∈ xi for which f (1) i = F (x̃ (1) i ) and f (2) i = F (x̃ (2) i ), P (f (1) i |f (2) i ) will be maximized. This means that features from a patch taken from a specific image can effectively predict, under our model, features extracted from other patches in the same image. Conversely, we want our model to minimize P (fi|fj) for i, j being two patches taken from distinct images. Following the logic presented before, we will need to sample contrasting patch x̃(1)j from a different image xj such that P (f (1) i |f (2) i ) > P (f (1) j |f (2) i ), where f (1) j = F (x̃ (1) j ). In order to obtain contrasting samples, we use regions from two random images in the training set. We will use a distance ratio, described earlier
in Eq. (2) for the supervised case, to represent the probability two feature vectors were taken from the same image. The resulting training loss for a pair of images will be defined as
LSC(x1, x2) = − log e−‖f (1) 1 −f (2) 1 ‖2
e−‖f (1) 1 −f (2) 1 ‖2 + e−‖f (1) 1 −f (1) 2 ‖2
(3)
Effectively minimizing a log-probability under the SoftMax measure. This formulation is portrayed in figure 4.1. Since we sample our contrasting sample from the same underlying distribution, we can evaluate this loss considering the image patch as both patch compared (anchor) and contrast symmetrically. The final loss will be the average between these estimations:
L̂SC(x1, x2) = 1
2 [LSC(x1, x2) + LSC(x2, x1)]
4.2 METHOD
Convolutional network are usually trained using SGD over mini-batch of samples, therefore we can extract patches and contrasting patches without changing the network architecture. Each image serves as both anchor and positive patches, for which the corresponding features should be closer, as well as contrasting samples for other images in that batch. For a batch of N images, two samples from each image are taken, and N2 different distance comparisons are made. The final loss is defined as the average distance ratio for all images in the batch:
LSC({x}Ni=1) = 1
N N∑ i=1 LSC(xi, {x}j 6=i) = − 1 N N∑ i=1 log e−‖f (1) i −f (2) i ‖2∑N j=1 e −‖f(1)i −f (2) j ‖2
(4)
Since the criterion is differentiable with respect to its inputs, it is fully compliant with standard methods for training convolutional network and specifically using backpropagation and gradient descent. Furthermore, SC can be applied to any layer in the network hierarchy. In fact, SC can be used at multiple layers within the same convolutional network. The spatial properties of the
features means that we can sample directly from feature space f̃ (m) ∈ f instead of from the original image. Therefore SC has a simple implementation which doesn’t require substation amount of computation. The complete algorithm for batch training is described in Algorithm (1). Similar to the batch normalization (BN) layer Ioffe & Szegedy (2015), a recent usage for batch statistics in neural networks, SC also uses the batch statistics. While BN normalize the input based on the batch statistics, SC sample from it. This can be viewed as a simple sampling from the space of possible features describing a patch of image.
Algorithm 1 Calculation the spatial contrasting loss Require: X = {x}Ni=1 # Training on batches of images
# Get the spatial features for the whole batch of images # Size: N ×Wf ×Hf × C {f}Ni=1 ← ConvNet(X)
# Sample spatial features and calculate embedded distance between all pairs of images for i = 1 to N do f̃ (1) i ← sample(fi)
for j = 1 to N do f̃ (2) j ← sample(fj) Dist(i, j)← ‖f̃ (1)i − f̃ (2) j ‖2
end for end for
# Calculate log SoftMax normalized distances di ← − log e
−Dist(i,i)∑N k=1 e −Dist(i,k)
# Spatial contrasting loss is the mean of distance ratios return 1N ∑N i=1 di
5 EXPERIMENTS
In this section we report empirical results showing that using SC loss as an unsupervised pretraining procedure can improve state-of-the-art performance on subsequent classification. We experimented with MNIST, CIFAR-10 and STL10 datasets. We used modified versions of well studied networks such as those of Lin et al. (2013) and Rasmus et al. (2015). A detailed description of our architecture can be found in 4.
In each one of the experiments, we used the spatial contrasting criterion to train the network on the unlabeled images. In each usage of SC criterion, patch features were sampled from the preceding layer in uniform. We note that spatial size of sampled patches ranged between datasets, where on STL10 and Cifar10 it covered about 30% of the image, MNIST required the use of larger patches covering almost the entire image.Training was done by using SGD with an initial learning rate of 0.1 that was decreased by a factor of 10 whenever the measured loss stopped decreasing. After convergence, we used the trained model as an initialization for a supervised training on the complete labeled dataset. The supervised training was done following the same regime, only starting with a lower initial learning rate of 0.01. We used mild data augmentations, such as small translations and horizontal mirroring. The datasets we used are:
• STL10 (Coates et al. (2011)). This dataset consists of 100, 000 96× 96 colored, unlabeled images, together with another set of 5, 000 labeled training images and 8, 000 test images . The label space consists of 10 object classes.
• Cifar10 (Krizhevsky & Hinton (2009)). The well known CIFAR-10 is an image classification benchmark dataset containing 50, 000 training images and 10, 000 test images. The
All experiments were conducted using the Torch7 framework by Collobert et al. (2011). Code reproducing these results will by available at https://github.com/eladhoffer/ SpatialContrasting.
5.1 RESULTS ON STL10
Since STL10 dataset is comprised of mostly unlabeled data, it is most suitable to highlight the benefits of the spatial contrasting criterion. The initial training was unsupervised, as described earlier, using the entire set of 105, 000 samples (union of the original unlabeled set and labeled training set). The representation outputted by the training, was used to initialize supervised training on the 5, 000 labeled images. Evaluation was done on a separate test set of 8, 000 samples. Comparing with state of the art results, we see an improvement of 7% in test accuracy over the best model by Zhao et al. (2015), setting the SC as best model at 81.3% test classification accuracy (see Table (1)). We note that the results of Dosovitskiy et al. (2014) are achieved with no fine-tuning over labeled examples, which may be unfair to this work. We also compare with the same network, but without SC initialization, which achieves a lower classification of 72.6%. This is an indication that indeed SC managed to leverage unlabeled examples to provide a better initialization point for the supervised model.
5.2 RESULTS ON CIFAR10
For Cifar10 dataset, we use the same setting as Coates & Ng (2012) and Hui (2013) to test a model’s ability to learn from unlabeled images. Here, only 4, 000 samples out of 50, 000 are used with their label annotation, and the rest of the samples can be used only in an unsupervised manner. The final test accuracy is measured on the entire 10, 000 test set. In our experiments, we trained our model using SC criterion on the entire dataset, and then used only 400 labeled samples per class (for a total of 4000) in a supervised regime over the initialized network. The results are compared with previous efforts in Table (2). Using the SC criterion allowed an improvement of 6.8% over a non-initialized model, and achieved a final test accuracy of 79.2%. This is a competitive result with current state-of-the-art models.
5.3 RESULTS ON MNIST
The MNIST dataset is very different in nature from the Cifar10 and STL10 datasets, we experimented earlier. The biggest difference, relevant to this work, is that spatial regions sampled from MNIST images usually provide very little, or no information. Thus, SC is much less suited for MNIST dataset, and was conjured to have little benefit. We still, however, experimented with initializing a model with SC criterion and continuing with a fully-supervised regime over all labeled
examples. We found again that this provided benefit over training the same network without preinitialization, improving results from 0.63% to 0.34% error on test set. As mentioned previously, the effective compared patches of MNIST covered almost the entire image area. This can be attributed to the fact that MNIST requires global features to differentiate between digits. The results, compared with previous attempts are included in Table (3).
6 CONCLUSIONS AND FUTURE WORK
In this work we presented spatial contrasting - a novel unsupervised criterion for training convolutional networks on unlabeled data. Its is based on comparison between spatial features sampled from a number of images. We’ve shown empirically that using spatial contrasting as a pretraining technique to initialize a ConvNet, can improve its performance on a subsequent supervised training. In cases where a lot of unlabeled data is available, such as the STL10 dataset, this translates to state-of-the-art classification accuracy in the final model.
Since the spatial contrasting loss is a differentiable estimation that can be computed within a network parallel to supervised losses, in future work we plan to embed it as a semi-supervised model. This usage will allow to create models that can leverage both labeled an unlabeled data, and can be compared to similar semi-supervised models such as the ladder network Rasmus et al. (2015). It is is also apparent that contrasting can occur in dimensions other than the spatial, the most straightforward is the temporal dimension. This suggests that similar training procedure can be applied on segments of sequences to learn useful representation without explicit supervision.
7 APPENDIX

This paper proposes an unsupervised training objective based on patch contrasting for visual representation learning using deep neural networks. In particular, the feature representations of the patches from the same image are encouraged to be closer than the those from different images. The distance ratios of positive training pairs are optimized. The proposed method are empirically shown to be effective as an initialization method for supervised training.
Strengths:
- The training objective is reasonable. In particular, high-level features show translation invariance.
- The proposed methods are effective for initializing neural networks for supervised training on several datasets.
Weaknesses:
- The methods are technically similar to the “exemplar network” (Dosovitskiy 2015). Cropping patches from a single image can be taken as a type of data augmentation, which is comparable to the data augmentation of positive sample (the exemplar) in (Dosovitskiy 2015).
- The paper is experimentally misleading.
The results reported in this paper are based on fine-tuning the whole network with supervision. However, in Table 2, the results of exemplar convnets (Dosovitskiy 2015) is from unsupervised feature learning (the network is not finetuned with labeled samples, and only a classifier is trained upon the features). Therefore, the comparison is not fair. I suspect that exemplar convnets (Dosovitskiy 2015) would achieve similar improvements from fine-tuning; so, without such comparisons (head-to-head comparison with and without fine-tuning based on the same architecture except for the loss), the experimental results are not fully convincing.
Regarding the comparison to “What-where” autoencoder (Zhao et al, 2015), it will be interesting to compare against it in large-scale settings, as shown by Zhang et al, ICML 2016 (Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-Scale Image Classification). Training an AlexNet is not very time-consuming with latest (e.g., TITAN-X level) GPUs.
The proposed method seems useful only for natural images where different patches from the same image can be similar to each other.


We ask whether neural networks can learn to use secret keys to protect information from other neural networks. Specifically, we focus on ensuring confidentiality properties in a multiagent system, and we specify those properties in terms of an adversary. Thus, a system may consist of neural networks named Alice and Bob, and we aim to limit what a third neural network named Eve learns from eavesdropping on the communication between Alice and Bob. We do not prescribe specific cryptographic algorithms to these neural networks; instead, we train end-to-end, adversarially. We demonstrate that the neural networks can learn how to perform forms of encryption and decryption, and also how to apply these operations selectively in order to meet confidentiality goals.
1 INTRODUCTION
As neural networks are applied to increasingly complex tasks, they are often trained to meet endto-end objectives that go beyond simple functional specifications. These objectives include, for example, generating realistic images (e.g., (Goodfellow et al., 2014a)) and solving multiagent problems (e.g., (Foerster et al., 2016a;b; Sukhbaatar et al., 2016)). Advancing these lines of work, we show that neural networks can learn to protect their communications in order to satisfy a policy specified in terms of an adversary.
Cryptography is broadly concerned with algorithms and protocols that ensure the secrecy and integrity of information. Cryptographic mechanisms are typically described as programs or Turing machines. Attackers are also described in those terms, with bounds on their complexity (e.g., limited to polynomial time) and on their chances of success (e.g., limited to a negligible probability). A mechanism is deemed secure if it achieves its goal against all attackers. For instance, an encryption algorithm is said to be secure if no attacker can extract information about plaintexts from ciphertexts. Modern cryptography provides rigorous versions of such definitions (Goldwasser & Micali, 1984).
Adversaries also play important roles in the design and training of neural networks. They arise, in particular, in work on adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014b) and on generative adversarial networks (GANs) (Goodfellow et al., 2014a). In this latter context, the adversaries are neural networks (rather than Turing machines) that attempt to determine whether a sample value was generated by a model or drawn from a given data distribution. Furthermore, in contrast with definitions in cryptography, practical approaches to training GANs do not consider all possible adversaries in a class, but rather one or a small number of adversaries that are optimized by training. We build on these ideas in our work.
Neural networks are generally not meant to be great at cryptography. Famously, the simplest neural networks cannot even compute XOR, which is basic to many cryptographic algorithms. Nevertheless, as we demonstrate, neural networks can learn to protect the confidentiality of their data from other neural networks: they discover forms of encryption and decryption, without being taught specific algorithms for these purposes.
Knowing how to encrypt is seldom enough for security and privacy. Interestingly, neural networks can also learn what to encrypt in order to achieve a desired secrecy property while maximizing utility. Thus, when we wish to prevent an adversary from seeing a fragment of a plaintext, or from estimating a function of the plaintext, encryption can be selective, hiding the plaintext only partly.
∗Visiting from Carnegie Mellon University.
The resulting cryptosystems are generated automatically. In this respect, our work resembles recent research on automatic synthesis of cryptosystems, with tools such as ZooCrypt (Barthe et al., 2013), and contrasts with most of the literature, where hand-crafted cryptosystems are the norm. ZooCrypt relies on symbolic theorem-proving, rather than neural networks.
Classical cryptography, and tools such as ZooCrypt, typically provide a higher level of transparency and assurance than we would expect by our methods. Our model of the adversary, which avoids quantification, results in much weaker guarantees. On the other hand, it is refreshingly simple, and it may sometimes be appropriate.
Consider, for example, a neural network with several components, and suppose that we wish to guarantee that one of the components does not rely on some aspect of the input data, perhaps because of concerns about privacy or discrimination. Neural networks are notoriously difficult to explain, so it may be hard to characterize how the component functions. A simple solution is to treat the component as an adversary, and to apply encryption so that it does not have access to the information that it should not use. In this respect, the present work follows the recent research on fair representations (Edwards & Storkey, 2015; Louizos et al., 2015), which can hide or remove sensitive information, but goes beyond that work by allowing for the possibility of decryption, which supports richer dataflow structures.
Classical cryptography may be able to support some applications along these lines. In particular, homomorphic encryption enables inference on encrypted data (Xie et al., 2014; Gilad-Bachrach et al., 2016). On the other hand, classical cryptographic functions are generally not differentiable, so they are at odds with training by stochastic gradient descent (SGD), the main optimization technique for deep neural networks. Therefore, we would have trouble learning what to encrypt, even if we know how to encrypt. Integrating classical cryptographic functions—and, more generally, integrating other known functions and relations (e.g., (Neelakantan et al., 2015))—into neural networks remains a fascinating problem.
Prior work at the intersection of machine learning and cryptography has focused on the generation and establishment of cryptographic keys (Ruttor, 2006; Kinzel & Kanter, 2002), and on corresponding attacks (Klimov et al., 2002). In contrast, our work takes these keys for granted, and focuses on their use; a crucial, new element in our work is the reliance on adversarial goals and training. More broadly, from the perspective of machine learning, our work relates to the application of neural networks to multiagent tasks, mentioned above, and to the vibrant research on generative models and on adversarial training (e.g., (Goodfellow et al., 2014a; Denton et al., 2015; Salimans et al., 2016; Nowozin et al., 2016; Chen et al., 2016; Ganin et al., 2015)). From the perspective of cryptography, it relates to big themes such as privacy and discrimination. While we embrace a playful, exploratory approach, we do so with the hope that it will provide insights useful for further work on these topics.
Section 2 presents our approach to learning symmetric encryption (that is, shared-key encryption, in which the same keys are used for encryption and for decryption) and our corresponding results. Appendix A explains how the same concepts apply to asymmetric encryption (that is, public-key encryption, in which different keys are used for encryption and for decryption). Section 3 considers selective protection. Section 4 concludes and suggests avenues for further research. Appendix B is a brief review of background on neural networks.
2 LEARNING SYMMETRIC ENCRYPTION
This section discusses how to protect the confidentiality of plaintexts using shared keys. It describes the organization of the system that we consider, and the objectives of the participants in this system. It also explains the training of these participants, defines their architecture, and presents experiments.
2.1 SYSTEM ORGANIZATION
A classic scenario in security involves three parties: Alice, Bob, and Eve. Typically, Alice and Bob wish to communicate securely, and Eve wishes to eavesdrop on their communications. Thus, the desired security property is secrecy (not integrity), and the adversary is a “passive attacker” that can intercept communications but that is otherwise quite limited: it cannot initiate sessions, inject messages, or modify messages in transit.
We start with a particularly simple instance of this scenario, depicted in Figure 1, in which Alice wishes to send a single confidential message P to Bob. The message P is an input to Alice. When Alice processes this input, it produces an output C. (“P ” stands for “plaintext” and “C” stands for “ciphertext”.) Both Bob and Eve receive C, process it, and attempt to recover P . We represent what they compute by PBob and PEve, respectively. Alice and Bob have an advantage over Eve: they share a secret key K. We treat K as an additional input to Alice and Bob. We assume one fresh key K per plaintext P , but, at least at this abstract level, we do not impose that K and P have the same length.
For us, Alice, Bob, and Eve are all neural networks. We describe their structures in Sections 2.4 and 2.5. They each have parameters, which we write θA, θB , and θE , respectively. Since θA and θB need not be equal, encryption and decryption need not be the same function even if Alice and Bob have the same structure. As is common for neural networks, Alice, Bob, and Eve work over tuples of floating-point numbers, rather than sequences of bits. In other words, K, P , PBob, PEve, and C are all tuples of floating-point numbers. Note that, with this formulation, C, PBob, and PEve may consist of arbitrary floating-point numbers even if P and K consist of 0s and 1s. In practice, our implementations constrain these values to the range (−1, 1), but permit the intermediate values. We have explored alternatives (based on Williams’ REINFORCE algorithm (Williams, 1992) or on Foerster et al.’s discretization technique (Foerster et al., 2016b)), but omit them as they are not essential to our main points.
This set-up, although rudimentary, suffices for basic schemes, in particular allowing for the possibility that Alice and Bob decide to rely on K as a one-time pad, performing encryption and decryption simply by XORing the key K with the plaintext P and the ciphertext C, respectively. However, we do not require that Alice and Bob function in this way—and indeed, in our experiments in Section 2.5, they discover other schemes. For simplicity, we ignore the process of generating a key from a seed. We also omit the use of randomness for probabilistic encryption (Goldwasser & Micali, 1984). Such enhancements may be the subject of further work.
2.2 OBJECTIVES
Informally, the objectives of the participants are as follows. Eve’s goal is simple: to reconstruct P accurately (in other words, to minimize the error between P and PEve). Alice and Bob want to communicate clearly (to minimize the error between P and PBob), but also to hide their communication from Eve. Note that, in line with modern cryptographic definitions (e.g., (Goldwasser & Micali, 1984)), we do not require that the ciphertext C “look random” to Eve. A ciphertext may even contain obvious metadata that identifies it as such. Therefore, it is not a goal for Eve to distinguish C from a random value drawn from some distribution. In this respect, Eve’s objectives contrast with common ones for the adversaries of GANs. On the other hand, one could try to reformulate Eve’s goal in terms of distinguishing the ciphertexts constructed from two different plaintexts.
Given these objectives, instead of training each of Alice and Bob separately to implement some known cryptosystem (Dourlens, 1996), we train Alice and Bob jointly to communicate successfully and to defeat Eve without a pre-specified notion of what cryptosystem they may discover for this
purpose. Much as in the definitions of GANs, we would like Alice and Bob to defeat the best possible version of Eve, rather than a fixed Eve. Of course, Alice and Bob may not win for every plaintext and every key, since knowledge of some particular plaintexts and keys may be hardwired into Eve. (For instance, Eve could always output the same plaintext, and be right at least once.) Therefore, we assume a distribution on plaintexts and keys, and phrase our goals for Alice and Bob in terms of expected values.
We write A(θA, P,K) for Alice’s output on input P,K, write B(θB , C,K) for Bob’s output on input C,K, and write E(θE , C) for Eve’s output on input C. We introduce a distance function d on plaintexts. Although the exact choice of this function is probably not crucial, for concreteness we take the L1 distance d(P, P ′) = Σi=1,N |Pi − P ′i | where N is the length of plaintexts. We define a per-example loss function for Eve:
LE(θA, θE , P,K) = d(P,E(θE , A(θA, P,K)))
Intuitively, LE(θA, θE , P,K) represents how much Eve is wrong when the plaintext is P and the key is K. We also define a loss function for Eve over the distribution on plaintexts and keys by taking an expected value:
LE(θA, θE) = EP,K(d(P,E(θE , A(θA, P,K))))
We obtain the “optimal Eve” by minimizing this loss:
OE(θA) = argminθE (LE(θA, θE))
Similarly, we define a per-example reconstruction error for Bob, and extend it to the distribution on plaintexts and keys:
LB(θA, θB , P,K) = d(P,B(θB , A(θA, P,K),K))
LB(θA, θB) = EP,K(d(P,B(θB , A(θA, P,K),K)))
We define a loss function for Alice and Bob by combining LB and the optimal value of LE :
LAB(θA, θB) = LB(θA, θB)− LE(θA, OE(θA))
This combination reflects that Alice and Bob want to minimize Bob’s reconstruction error and to maximize the reconstruction error of the “optimal Eve”. The use of a simple subtraction is somewhat arbitrary; below we describe useful variants. We obtain the “optimal Alice and Bob” by minimizing LAB(θA, θB):
(OA, OB) = argmin(θA,θB)(LAB(θA, θB))
We write “optimal” in quotes because there need be no single global minimum. In general, there are many equi-optimal solutions for Alice and Bob. As a simple example, assuming that the key is of the same size as the plaintext and the ciphertext, Alice and Bob may XOR the plaintext and the ciphertext, respectively, with any permutation of the key, and all permutations are equally good as long as Alice and Bob use the same one; moreover, with the way we architect our networks (see Section 2.4), all permutations are equally likely to arise.
Training begins with the Alice and Bob networks initialized randomly. The goal of training is to go from that state to (OA, OB), or close to (OA, OB). We explain the training process next.
2.3 TRAINING REFINEMENTS
Our training method is based upon SGD. In practice, much as in work on GANs, our training method cuts a few corners and incorporates a few improvements with respect to the high-level description of objectives of Section 2.2. We present these refinements next, and give further details in Section 2.5.
First, the training relies on estimated values calculated over “minibatches” of hundreds or thousands of examples, rather than on expected values over a distribution.
We do not compute the “optimal Eve” for a given value of θA, but simply approximate it, alternating the training of Eve with that of Alice and Bob. Intuitively, the training may for example proceed roughly as follows. Alice may initially produce ciphertexts that neither Bob nor Eve understand at all. By training for a few steps, Alice and Bob may discover a way to communicate that allows Bob
to decrypt Alice’s ciphertexts at least partly, but which is not understood by (the present version of) Eve. In particular, Alice and Bob may discover some trivial transformations, akin to rot13. After a bit of training, however, Eve may start to break this code. With some more training, Alice and Bob may discover refinements, in particular codes that exploit the key material better. Eve eventually finds it impossible to adjust to those codes. This kind of alternation is typical of games; the theory of continuous games includes results about convergence to equilibria (e.g., (Ratliff et al., 2013)) which it might be possible to apply in our setting.
Furthermore, in the training of Alice and Bob, we do not attempt to maximize Eve’s reconstruction error. If we did, and made Eve completely wrong, then Eve could be completely right in the next iteration by simply flipping all output bits! A more realistic and useful goal for Alice and Bob is, generally, to minimize the mutual information between Eve’s guess and the real plaintext. In the case of symmetric encryption, this goal equates to making Eve produce answers indistinguishable from a random guess. This approach is somewhat analogous to methods that aim to prevent overtraining GANs on the current adversary (Salimans et al., 2016, Section 3.1). Additionally, we can tweak the loss functions so that they do not give much importance to Eve being a little lucky or to Bob making small errors that standard error-correction could easily address.
Finally, once we stop training Alice and Bob, and they have picked their cryptosystem, we validate that they work as intended by training many instances of Eve that attempt to break the cryptosystem. Some of these instances may be derived from earlier phases in the training.
2.4 NEURAL NETWORK ARCHITECTURE
The Architecture of Alice, Bob, and Eve Because we wish to explore whether a general neural network can learn to communicate securely, rather than to engineer a particular method, we aimed to create a neural network architecture that was sufficient to learn mixing functions such as XOR, but that did not strongly encode the form of any particular algorithm.
To this end, we chose the following “mix & transform” architecture. It has a first fully-connected (FC) layer, where the number of outputs is equal to the number of inputs. The plaintext and key bits are fed into this FC layer. Because each output bit can be a linear combination of all of the input bits, this layer enables—but does not mandate—mixing between the key and the plaintext bits. In particular, this layer can permute the bits. The FC layer is followed by a sequence of convolutional layers, the last of which produces an output of a size suitable for a plaintext or ciphertext. These convolutional layers learn to apply some function to groups of the bits mixed by the previous layer, without an a priori specification of what that function should be. Notably, the opposite order (convolutional followed by FC) is much more common in image-processing applications. Neural networks developed for those applications frequently use convolutions to take advantage of spatial locality. For neural cryptography, we specifically wanted locality—i.e., which bits to combine—to be a learned property, instead of a pre-specified one. While it would certainly work to manually pair each input plaintext bit with a corresponding key bit, we felt that doing so would be uninteresting.
We refrain from imposing further constraints that would simplify the problem. For example, we do not tie the parameters θA and θB , as we would if we had in mind that Alice and Bob should both learn the same function, such as XOR.
2.5 EXPERIMENTS
As a proof-of-concept, we implemented Alice, Bob, and Eve networks that takeN -bit random plaintext and key values, and produce N -entry floating-point ciphertexts, for N = 16, 32, and 64. Both plaintext and key values are uniformly distributed. Keys are not deliberately reused, but may reoccur because of random selection. (The experiments in Section 3 consider more interesting distributions and also allow plaintext and key values to have different sizes.)
We implemented our experiments in TensorFlow (Abadi et al., 2016a;b). We ran them on a workstation with one GPU; the specific computation platform does not affect the outcome of the experiments. We plan to release the source code for the experiments.
Network Structure Our networks follow the “mix & transform” pattern described in Section 2.4. The Alice network concatenates two N -bit inputs (the plaintext and the key) into a 2N -entry vector,
using−1 and 1 to represent the bit values. This vector is processed through a 2N×2N FC layer, then sent through a succession of four 1-D convolutional layers. The convolutional layers are described in terms of their window size, input depth, and output depth. Each has a “stride”—the amount by which the window is shifted at each step. The concrete layers are [4, 1, 2], [2, 2, 4], [1, 4, 4], and [1, 4, 1], with strides of 1, 2, 1, 1. Intuitively, the first layer slides a size-4 window across the 2N output elements from the FC layer, outputting two numbers (output depth 2), with stride 1. We use a sigmoid nonlinear unit after each layer except the final one. After the final layer, where the output is reduced to N elements, we use a tanh nonlinear unit. (Recall that tanh has an output domain between [−1, 1], bringing the values back to a range that can map to binary values.) The Bob network is identical to the Alice network The Eve network takes only the ciphertext as input, and thus has a first N × 2N FC layer.
Training To train the networks, we use a “minibatch” size ranging from 256 to 4096 entries. Unless otherwise specified, we present the results from the training runs with 4096 entries. We use TensorFlow’s Adam (Kingma & Ba, 2014) optimizer with a learning rate of 0.0008. We do not reduce the learning rate over time because we wish Alice, Bob, and Eve to be able to continue to respond strongly to changes in other components, until Alice and Bob—hopefully—achieve a solution that is robust to additional changes in Eve.
Training alternates between Alice/Bob and Eve, with Alice/Bob training for one minibatch, and then Eve training for two minibatches. We chose this ratio in order to give a slight computational edge to the adversary Eve without training it so much that it becomes excessively specific to the exact current parameters of Alice and Bob. Empirically, this value has worked well.
Loss Functions Eve’s loss function is exactly as described above: the L1 distance between Eve’s guess and the input plaintext. The loss function for Alice and Bob is more complex, as indicated in Sections 2.2 and 2.3. This function has two components, related to Bob’s reconstruction error and to the eavesdropper’s success. The first component is simply the L1 distance between Bob’s output and the input plaintext. The latter component, on the other hand, is (N/2− Eve L1 error)2/(N/2)2. This definition expresses the goal, described in Section 2.3, that Eve should not do better than random guessing. Accordingly, this component is minimized when half of the message bits are wrong and half are right. We choose a quadratic formula in order to place more emphasis on making Eve have a large error, and to impose less of a penalty when Eve guesses a few bits correctly, as should happen occasionally even if Eve’s guesses are effectively random. Adopting this formulation allowed us to have a meaningful per-example loss function (instead of looking at larger batch statistics), and improved the robustness of training. Its cost is that our final, trained Alice and Bob typically allow Eve to reconstruct slightly more bits than purely random guessing would achieve. We have not obtained satisfactory results for loss functions that depend linearly (rather than quadratically) on Eve’s reconstruction error. The best formulation remains an open question.
Post-training Evaluation After successfully training the networks so that they cross an accuracy threshold (e.g., at most 0.05 bits of reconstruction error for Alice and Bob, with Eve achieving only 1-2 bits more than random guessing would predict), we reset the Eve network and train it from scratch 5 times, each for up to 250,000 steps, recording the best result achieved by any Eve. An Alice/Bob combination that fails to achieve the target thresholds within 150,000 steps is a training failure. If the retrained Eves obtain a substantial advantage, the solution is non-robust. Otherwise, we consider it a successful training outcome.
Results Figure 2 shows, for one successful run, the evolution of Bob’s reconstruction error and Eve’s reconstruction error vs. the number of training steps for N = 16 bit plaintext and key values, using a minibatch size of 4096. Each point in the graph is the mean error across 4096 examples. An ideal result would have Bob’s reconstruction error drop to zero and Eve’s reconstruction error reach 8 (half the bits wrong). In this example, both reconstruction errors start high. After a period of time, Alice and Bob start to communicate quite effectively, but in a way that allows Eve to improve its understanding as well, gradually. Then, around step 10,000, Alice and Bob counter Eve’s progress. By about step 15,000, the training goals are effectively achieved. The remaining steps merely increase Eve’s reconstruction error slightly.
This training graph does not look like a typical result for neural networks, where monotonicity in the number of steps is generally expected. Instead, the dynamics of this adversarial training appear somewhat more reminiscent of evolutionary processes. These dynamics appear to depend on somewhat random-seeming change to cause the bits to mix slightly, but once there is some mixing, the gradient descent can rapidly drive it farther.
Supporting this interpretation is the observation that training is not always successful. WithN = 16, six of twenty initial runs were failures that never got Bob’s reconstruction error under the 0.05 threshold, or failed to drive Eve’s reconstruction error above 7.3 bits (of 16). In order to test the robustness of the other fourteen Alice/Bob combinations, we retrained Eve five times, and obtained reconstruction errors for Eve that ranged from 4.67 to 6.97 bits, with a mean of 6.1. Figure 3 shows the final reconstruction errors of Bob and of the most effective retrained Eve for those fourteen Alice/Bob combinations. If we somewhat arbitrarily define success as maintaining Bob’s reconstruction error at or under 0.05 bits, and requiring that Eve get at least 6 bits wrong, on average, then training succeeded half of the time (ten of twenty cases).
Although training with an adversary is often unstable (Salimans et al., 2016), we suspect that some additional engineering of the neural network and its training may be able to increase this overall success rate. With a minibatch size of only 512, for example, we achieved a success rate of only 1/3 (vs. the 1/2 that we achieved with a minibatch size of 4096). In the future, it may be worth studying the impact of minibatch sizes, and also that of other parameters such as the learning rate.
Analogous results hold in general for N = 32 and N = 64-bit keys and plaintexts; training appears to be successful somewhat more often for N = 64. Basically, the experiments for N = 32 and N = 64 indicate that there is nothing special about N = 16 which, to a cryptographer, may look suspiciously tiny. We focus our presentation on the case of N = 16 because, first, the experiments run more rapidly, and second, it is modestly easier to examine their behavior.
For one successful training run, we studied the changes in the ciphertext induced by various plaintext/key pairs. Although we did not perform an exhaustive analysis of the encryption method, we did make a few observations. First, it is key-dependent: changing the key and holding the plaintext constant results in different ciphertext output. It is also plaintext-dependent, as required for successful communication. However, it is not simply XOR. In particular, the output values are often floating-point values other than 0 and 1. Moreover, the effect of a change to either a key bit or a plaintext bit is spread across multiple elements in the ciphertext, not constrained to a single bit as it would be with XOR. A single-bit flip in the key typically induces significant changes in three to six of the 16 elements in the ciphertext, and smaller changes in other elements. Plaintext bits are similarly diffused across the ciphertext.
3 LEARNING WHAT TO PROTECT
Building on the results of Section 2, we study selective protection. In other words, we consider the question of whether neural networks can learn what information to protect, given confidentiality objectives described in terms of an adversary. In the simplest case, a plaintext may have several components; if we wish to keep the adversary from seeing one of the components, it may suffice to encrypt it. More generally, we may wish to publish some values correlated with the plaintext but prevent the adversary from calculating other values. In such scenarios, protecting information selectively while maximizing utility is both challenging and interesting.
3.1 THE PROBLEM
To test these ideas and to demonstrate the viability of selective protection, we focus on a concrete experiment. We construct an artificial dataset consisting of tuples of four values, 〈A, B, C, D〉. We aim to build and train a system that outputs two predictions of D, given the first three values as inputs: a “true” prediction of D (that is, the most accurate possible estimate of D given 〈A, B, C〉), and a “privacy-preserving” estimate of D, which we call D-public, defined as the best possible estimate of D that does not reveal any information about the value of C.
The system organization is essentially that of Figure 1, with Alice and Bob sharing a key, but here Alice receives 〈A, B, C〉 and produces D-public in addition to a ciphertext; both Bob and Eve have ac-
cess to Alice’s outputs; Bob uses them for producing an improved estimate of D, while Eve attempts to recover C. Thus, our goal is to demonstrate that the adversarial training permits approximating D without revealing C, and that this approximation can be combined with encrypted information and with a key in order to obtain a better approximation of D.
The distributions of A, B, C, and D are constructed so that there is substantial correlation between them. In particular, knowing D exactly would convey some information about the likely values of C. In our experiments, A, B, C, and D are normally distributed floating-point values with a correlation matrix imposed upon them. The specifics of how we generate A, B, C, and D are standard: we compute the Cholesky decomposition LL∗ of a 4 × 4 correlation matrix that specifies the pairwise correlations, we generate vectors of 4 normally distributed floating-point values, and then we multiply these vectors by L in order to generate values of A, B, C, and D.
Although A, B, C, and D do not correspond to any real-world data, these experiments are loosely inspired by real-world tasks. For example, A, B, and C may represent attributes of a user, which may in general be correlated, and D may represent a decision about the user, a prediction of the user’s behavior, or a recommendation to the user.
3.2 NEURAL NETWORK ARCHITECTURE AND TRAINING GOALS
In the experiments that follow, we use an augmented version of the neural network architecture of Section 2.4. The inputs first go into a new FC layer (12 inputs—eight key bits and four values—and 12 outputs); the outputs of that first layer are fed into a network with the architecture of Section 2.4. Intuitively, we chose this augmented architecture because a single FC layer should be capable of predicting D from A, B, and C, as well as making a prediction decorrelated with C; and the architecture of Section 2.4 suffices to encrypt any of the output of the first layer under the key. We therefore believed this augmented architecture would be sufficient to accomplish its task, though it may be more than is necessary to do so.
We train Eve with the goal of minimizing the squared error of its estimate of C. We train Alice and Bob with the goal of minimizing a linear combination of three terms: the squared error of each of the two estimates of D, and the absolute value of the covariance of Eve’s estimate of C with respect to the actual value of C. We compute this covariance on a batch of training examples, and define the loss function for Alice and Bob batch-wise.
A human engineer would naturally try to design the system somewhat differently—likely first computing the desired quantities, and then encrypting the values to be hidden, relying on the encryption and decryption components developed in Section 2 as modules. We prefer instead an end-to-end approach, because our goal is to understand whether a neural network can learn to hide information selectively, given only a high-level specification (“do not leak information about C, but let people with the right key learn as much as possible about D”). In this end-to-end approach, the system does not have distinct, pre-assigned units for encryption and decryption, and all components are trained jointly. In particular, although we benefit from some of the insights from Section 2, we do not rely on the components developed there. End-to-end approaches are sometimes preferred in machine learning because they can more efficiently use resources and information. In our case, this feature may mean, in particular, that the cryptographic operations are optimized for the given distribution of inputs. Nevertheless, our choices have the primary goal of maximizing insight, not efficiency.
3.3 RESULTS
To understand whether the system is learning to hide information properly, we train a separate evaluator that we call “Blind Eve”, which is aware of the distribution of C. Blind Eve tries to guess C relying only upon this baseline information, whereas the real Eve also knows the intermediate output (D-public) and the ciphertext. If Eve’s reconstruction error becomes equal to that of Blind Eve, we know that Eve is not successfully extracting information from the public estimate and the ciphertext.
Figure 4 shows the results of adversarial training to learn to hide C. The red “public error” line shows the average error of D-public vs. D. The green “Bob error” line shows Bob’s average reconstruction error, which drops over time until Bob is making the best possible prediction of D given the input. The public error becomes low, but is slightly higher than Bob’s. The blue dots show Eve’s average error improvement over Blind Eve. After a few steps of training, Eve’s error is substantially lower
than that of Blind Eve (by 0.1–0.2). As time proceeds, Eve’s relative advantage becomes zero. Eve is not able to reconstruct any more information about C than would be possible by simply knowing the distribution of values of C.
4 CONCLUSION
In this paper, we demonstrate that neural networks can learn to protect communications. The learning does not require prescribing a particular set of cryptographic algorithms, nor indicating ways of applying these algorithms: it is based only on a secrecy specification represented by the training objectives. In this setting, we model attackers by neural networks; alternative models may perhaps be enabled by reinforcement learning.
There is more to cryptography than encryption. In this spirit, further work may consider other tasks, for example steganography, pseudorandom-number generation, or integrity checks. Finally, neural networks may be useful not only for cryptographic protections but also for attacks. While it seems improbable that neural networks would become great at cryptanalysis, they may be quite effective in making sense of metadata and in traffic analysis.
ACKNOWLEDGMENTS
We are grateful to Samy Bengio, Laura Downs, Úlfar Erlingsson, Jakob Foerster, Nando de Freitas, Ian Goodfellow, Geoff Hinton, Chris Olah, Ananth Raghunathan, and Luke Vilnis for discussions on the matter of this paper.
A LEARNING ASYMMETRIC ENCRYPTION
Paralleling Section 2, this section examines asymmetric encryption (also known as public-key encryption). It presents definitions and experimental results, but omits a detailed discussion of the objectives of asymmetric encryption, of the corresponding loss functions, and of the practical refinements that we develop for training, which are analogous to those for symmetric encryption.
A.1 DEFINITIONS
In asymmetric encryption, a secret is associated with each principal. The secret may be seen as a seed for generating cryptographic keys, or directly as a secret key; we adopt the latter view. A public key can be derived from the secret, in such a way that messages encrypted under the public key can be decrypted only with knowledge of the secret.
We specify asymmetric encryption using a twist on our specification for symmetric encryption, shown in Figure 5. Instead of directly supplying the secret encryption key to Alice, we supply the secret key to a public-key generator, the output of which is available to every node. Only Bob has access to the underlying secret key. Much as in Section 2, several variants are possible, for instance to support probabilistic encryption.
The public-key generator is itself a neural network, with its own parameters. The loss functions treats these parameters much like those of Alice and Bob. In training, these parameters are adjusted at the same time as those of Alice and Bob.
A.2 EXPERIMENTS
In our experiments on asymmetric encryption, we rely on the same approach as in Section 2.5. In particular, we adopt the same network structure and the same approach to training.
The results of these experiments are intriguing, but much harder to interpret than those for symmetric encryption. In most training runs, the networks failed to achieve a robust outcome. Often, although it appeared that Alice and Bob had learned to communicate secretly, upon resetting and retraining Eve, the retrained adversary was able to decrypt messages nearly as well as Bob was.
However, Figure 6 shows the results of one training run, in which even after five reset/retrain cycles, Eve was unable to decrypt messages between Alice and Bob.
Our chosen network structure is not sufficient to learn general implementations of many of the mathematical concepts underlying modern asymmetric cryptography, such as integer modular arithmetic. We therefore believe that the most likely explanation for this successful training run was that Alice and Bob accidentally obtained some “security by obscurity” (cf. the derivation of asymmetric schemes from symmetric schemes by obfuscation (Barak et al., 2012)). This belief is somewhat reinforced by the fact that the training result was fragile: upon further training of Alice and Bob, Eve was able to decrypt the messages. However, we cannot rule out that the networks trained into some set of hard-to-invert matrix operations resulting in “public-key-like” behavior. Our results suggest that this issue deserves more exploration.
Further work might attempt to strengthen these results, perhaps relying on new designs of neural networks or new training procedures. A modest next step may consist in trying to learn particular asymmetric algorithms, such as lattice-based ciphers, in order to identify the required neural network structure and capacity.
B BACKGROUND ON NEURAL NETWORKS
Most of this paper assumes only a few basic notions in machine learning and neural networks, as provided by general introductions (e.g., LeCun et al. (2015)). The following is a brief review.
Neural networks are specifications of parameterized functions. They are typically constructed out of a sequence of somewhat modular building blocks. For example, the input to Alice is a vector of bits that represents the concatenation of the key and the plaintext. This vector (x) is input into a “fully-connected” layer, which consists of a matrix multiply (by A) and a vector addition (with b): Ax+ b. The result of that operation is then passed into a nonlinear function, sometimes termed an “activation function”, such as the sigmoid function, or the hyperbolic tangent function, tanh. In classical neural networks, the activation function represents a threshold that determines whether a neuron would “fire” or not, based upon its inputs. This threshold, and matrices and vectors such as A and b, are typical neural network “parameters”. “Training” a neural network is the process that finds values of its parameters that minimize the specified loss function over the training inputs.
Fully-connected layers are powerful but require substantial amounts of memory for a large network. An alternative to fully-connected layers are “convolutional” layers. Convolutional layers operate much like their counterparts in computer graphics, by sliding a parameterized convolution window across their input. The number of parameters in this window is much smaller than in an equivalent
fully-connected layer. Convolutional layers are useful for applying the same function(s) at every point in an input.
A neural network architecture consists of a graph of these building blocks (often, but not always, a DAG), specifying what the individual layers are (e.g., fully-connected or convolutional), how they are parameterized (number of inputs, number of outputs, etc.), and how they are wired.

The paper deals with an interesting application of adversarial training to encryption. It considers the standard scenario of Alice, Eve and Bob, where A and B aim to exchange messages conditioned on a shared key, while Eve should be unable to encrypt the message. Experiments are performed in a simple symmetric 16 bit encryption task, and an application on privacy. The concepts, ideas and previous literature are quite nicely and carefully presented.
The only major concern I have - and I apologize to the authors for not raising this earlier - are the experiments in section 3. In particular, I don't quite get the scenario. The reasoning here seems to be as follows: given information < A, B, C, D >, I want to give the public the value of D (e.g. movies watched) without releasing information about C (e.g. gender). In this scenario, Eve would need to be able to reconstruct D as good as possible without gaining information about C. What is described in section 3, however, is that D and D-public are both reconstructed by Bob, but why would Bob reconstruct the latter (he is not public, in particular because he is allowed to reconstruct C, which is not tested here)? Also, Eve only tries to estimate C, thus rendering the scenario not different in any way to the scenario considered in section 2.
I have two more minor concerns:
1) As raised in the pre-review, Eve should actually be stronger then Alice and Bob in order to be able to compensate for the missing key. The authors noted they have been doing these experiments and are going to add the results.
2) In any natural encryption case I would expect the length of the key to be much shorter then the length of the message. This, however, could potentially make the scenario much easier for Eve (although I doubt any of the results will change if the key is long enough).
I like the creative application of adversarial training to a completely different domain, and I believe it could be the starting point of a very interesting direction in cryptographic systems or in privacy applications (although it is unclear whether the weak guarantees of neural network based approaches can ever be overcome). At the same time the application in the privacy setting leaves me quite confused, and the symmetric encryption example is not particularly strong either. I'd appreciate if the authors could address the major concern I raised above, and I will be quite happy to raise the score in case this confusion can be resolved.