VOCABULARY SELECTION STRATEGIES FOR NEURAL MACHINE TRANSLATION
Classical translation models constrain the space of possible outputs by selecting a subset of translation rules based on the input sentence. Recent work on improving the efficiency of neural translation models adopted a similar strategy by restricting the output vocabulary to a subset of likely candidates given the source. In this paper we experiment with context and embedding-based selection methods and extend previous work by examining speed and accuracy trade-offs in more detail. We show that decoding time on CPUs can be reduced by up to 90% and training time by 25% on the WMT15 English-German and WMT16 English-Romanian tasks at the same or only negligible change in accuracy. This brings the time to decode with a state of the art neural translation system to just over 140 words per seconds on a single CPU core for English-German.
1 INTRODUCTION
Neural Machine Translation (NMT) has made great progress in recent years and improved the state of the art on several benchmarks (Jean et al., 2015b; Sennrich et al., 2016b; Zhou et al., 2016). However, neural systems are typically less efficient than traditional phrase-based translation models (PBMT; Koehn et al. 2003), both at training and decoding time.
The efficiency of neural models depends on the size of the target vocabulary and previous work has shown that vocabularies of well over 50k word types are necessary to achieve good accuracy (Jean et al., 2015a; Zhou et al., 2016). Neural translation systems compute the probability of the next target word given both the previously generated target words as well as the source sentence. Estimating this conditional distribution is linear in the size of the target vocabulary which can be very large for many language pairs (Grave et al., 2016). Recent work in neural translation has adopted sampling techniques from language modeling which do not leverage the input sentence (Mikolov et al., 2011; Jean et al., 2015a; Chen et al., 2016; Zhou et al., 2016).
On the other hand, classical translation models generate outputs in an efficient two-step selection procedure: first, a subset of promising translation rules is chosen by matching rules to the source sentence, and by pruning them based on local scores such as translation probabilities. Second, translation hypotheses are generated that incorporate non-local scores such as language model probabilities. Recently, Mi et al. (2016) proposed a similar strategy for neural translation: a selection method restricts the target vocabulary to a small subset, specific to the input sentence. The subset is then scored by the neural model. Their results demonstrate that vocabulary subsets that are only about 1% of the original size result in very little to no degradation in accuracy.
This paper complements their study by experimenting with additional selection techniques and by analyzing speed and accuracy in more detail. Similar to Mi et al. (2016), we consider selecting target words based either on a dictionary built from Viterbi word alignments, or by matching phrases in a traditional phrase-table, or by using the k most frequent words in the target language. In addition, we investigate methods that do not rely on a traditional phrase-based translation model or alignment model to select target words. We investigate bilingual co-occurrence counts, bilingual embeddings as well as a discriminative classifier to leverage context information via features extracted from the entire source sentence (§2).
∗Gurvan was interning at Facebook for this work.
Our experiments show speed-ups in CPU decoding by up to a factor of 10 at very little degradation in accuracy. Training speed on GPUs can be increased by a factor of 1.33. We find that word alignments as the sole selection method is sufficient to obtain good accuracy. This is in contrast to Mi et al. (2016) who used a combination of the 2, 000 most frequent words, word alignments as well as phrase-pairs. Selection methods often fall short in retrieving all words of the gold standard human translation. However, we find that with a reduced vocabulary of ∼ 600 words they can recover over 99% of the words that are actually chosen by a model that decodes over the entire vocabulary. Finally, the speed-ups obtained by vocabulary selection become even more significant if faster encoder models are used, since selection removes the burden of scoring large vocabularies (§4).
2 VOCABULARY SELECTION STRATEGIES
This section presents different selection strategies inspired by phrase-based translation. We improve on a simple word co-occurrence method by estimating bilingual word embeddings with Hellinger PCA and then by using word alignments instead of co-occurrence counts. Finally, we leverage richer context in the source via bilingual phrases from a phrase-table or by using the entire sentence in a Support Vector Machine classifier.
2.1 WORD CO-OCCURRENCES
This is the simplest approach we consider. We estimate a co-occurrence table which counts how many times each source word s co-occurs with each target word t in the training bitext. The table allows us to estimate the joint distribution P (s, t). Next, we create a list of the k target words that co-occur most with each source word, i.e., the words t which maximize P (s, t) for a given s. Vocabulary selection then simply computes the union of the target word lists associated with each source word in the input.
We were concerned that this strategy over-selects frequent target words which have higher cooccurrence counts than rare words, regardless of the source word. Therefore, we experimented with selecting target words maximizing point-wise mutual information (PMI) instead, i.e.,
PMI(s, t) = P (s, t)
P (s)P (t)
However, this estimate was deemed too unreliable for low P (t) in preliminary experiments and did not perform better than just P (s, t).
2.2 BILINGUAL EMBEDDINGS
We build bilingual embeddings by applying Hellinger Principal Component Analysis (PCA) to the bilingual co-occurrence count matrix Mi,j = P (t = i|s = j); this extends the work on monolingual embeddings of Lebret & Collobert (2014) to the bilingual case. The resulting low rank estimate of the matrix can be more robust for rare counts. Hellinger PCA has been shown to produce embeddings which perform similarly to word2vec but at higher speed (Mikolov et al., 2013; Gouws et al., 2015).
For selection, the estimated co-occurrence can be used instead of the raw counts as described in the above section. This strategy is equivalent to using the low rank representation of each source word (source embedding, i.e., column vectors from the PCA) and finding the target word with the closest low rank representation (target embeddings, i.e., row vectors from the PCA).
2.3 WORD ALIGNMENTS
This strategy uses word alignments learned from a bilingual corpus (Brown et al., 1993). Word alignment introduces latent variables to model P (t|s), the probability of source word t given target word s. Latent variables indicate the source position corresponding to each target position in a sentence pair (Koehn, 2010). We use FastAlign, a popular reparameterization of IBM Model 2 (Dyer et al., 2013). For each source word s, we build a list of the top k target words maximizing P (t|s). The candidate target vocabulary is the union of the lists for all source words.
Compared to co-occurrence counts, this strategy avoids selecting frequent target words when conditioning on a rare source word. Word alignments will only link a frequent target word to a rare source word if no better explanation is present in the source sentence.
2.4 PHRASE PAIRS
This strategy relies on a phrase translation table, i.e., a table pairing source phrases with corresponding target phrases. The phrase table is constructed by reading off all bilingual phrases that are consistent with the word alignments according to an extraction heuristic (Koehn, 2010). For selection, we restrict the phrase table to the phrases present in the source sentence and consider the union of the word types appearing in all corresponding target phrases (Mi et al., 2016). Compared to word alignments, we hope this strategy to fetch more relevant target words as it can rely on longer source phrases to leverage richer source context.
2.5 SUPPORT VECTOR MACHINES
Support Vector Machines (SVMs) for vocabulary selection have been previously proposed in (Bangalore et al., 2007). The idea is to determine a target vocabulary based on the entire source sentence rather than individual words or phrases. In particular, we train one SVM per target word taking as input a sparse vector encoding the source sentence as a bag of words. The SVM then predicts whether the considered target word is present or absent from the target sentence.
This classifier-based method has several advantages compared to phrase alignments: the input is not restricted to a few contiguous source words and can leverage all words in the source sentence. The model can express anti-correlation with negative weights, marking that the presence of a source word is a negative indicator for the presence of a target word. A disadvantage of this approach is that we need to feed the source sentence to all SVMs in order to get scores, instead of just reading from a pre-computed table. However, SVMs can be evaluated efficiently since (i) the features are sparse, and (ii) only features corresponding to words from the source sentence are used at each evaluation. Finally, this framework formulates the selection of each target word as an independent binary classification problem which might not favor competition between target words.
2.6 COMMON WORDS
Following Mi et al. (2016), we consider adding the k most frequent target words to the above selection methods. This set includes conjunctions, determiners, prepositions and frequent verbs. Pruning any such word through restrictive vocabulary selection may adversely affect the system output and is addressed by this technique.
3 RELATED WORK
The selection of a limited target vocabulary from the source sentence is classical topic in machine translation. It is often referred to as lexical selection. As mentioned above, word-based and phrasebased systems perform implicit lexical selection by building a word or phrase table from alignments to constrain the possible target words. Other approaches to lexical selection include discriminative models such as SVMs and Maximum Entropy models (Bangalore et al., 2007) as well as rule-based systems (Tufis, 2002; Tyers et al., 2012).
In the context of neural machine translation, vocabulary size has always been a concern. Various strategies have been proposed to improve training and decoding efficiency. Approaches inspired by importance sampling reduce the vocabulary for training (Jean et al., 2015a), byte pair encoding segment words into more frequent sub-units (Sennrich et al., 2016a), while (Luong & Manning, 2016) proposes to segment words into characters. Related work in neural language modeling is also relevant (Bengio et al., 2003; Mnih & Hinton, 2008; Chen et al., 2016). One can refer to (Sennrich, 2016) for further references.
Closer to our work, recent work (Mi et al., 2016) presents preliminary results on using lexical selection techniques in an NMT system. Compared to this work, we investigate more selection methods
(SVM, PCA, co-occurrences) and analyze the speed/accuracy trade-offs at various operating points. We report efficiency gains and distinguish the impact of selection in training and decoding.
4 EXPERIMENTS & RESULTS
This section presents our experimental setup, then discusses the impact of vocabulary selection at decoding time and then during training time.
4.1 EXPERIMENTAL SETUP
We use a an encoder-decoder style neural machine translation system based on Torch.1 Our encoder is a bidirectional recurrent neural network (Long Short Term Memory, LSTM) and an LSTM decoder with attention. The resulting context vector is fed to an LSTM decoder which generates the output (Bahdanau et al., 2015; Luong et al., 2015a). We use a single layer setup both in the encoder and as well as the decoder, each with 512 hidden units. Decoding experiments are run on a CPU since this is the most common type of hardware for inference. For training we use GPUs which are the most common hardware for neural network fitting. Specifically, we rely on 2.5GHz Intel Xeon 5 CPUs and Nvidia Tesla M40 GPUs. Decoding times are based on a single CPU core and training times are based on a single GPU card.
Word alignments are computed with FastAlign (Dyer et al., 2013) in both language directions and then symmetrized with ’grow-diag-final-and’. Phrase tables are computed with Moses (Koehn et al., 2007) and we train support vector machines with SvmSgd (Bottou, 2010). We also use the Moses preprocessing scripts to tokenize the training data.
We experiment on two language pairs. The majority of experiments are on WMT-15 English to German data (Bojar et al., 2015); we use newstest2013 for validation and newstest2010-2012 as well as newstest2014,2015 to present final test results. Training is restricted to sentences of no more than 50 words which results in 3.6m sentence pairs. We chose the vocabulary sizes following the same methodology. We use the 100k most frequent words both for the source and target vocabulary. At decoding time we use a standard beam search with a beam width of 5 in all experiments. Unknown output words are simply replaced with the source word whose attention score is largest (Luong et al., 2015b).
We also experiment with WMT-16 English to Romanian data using a similar setting but allowing sentences of up to 125 words (Bojar et al., 2016). Since the training set provided by WMT is limited to 600k sentence pairs, we add the synthetic training data provided by Sennrich et al. (2016b). This results in a total of 2.4m sentence pairs. Our source vocabulary comprises the 200k most frequent words and the target vocabulary contains 50k words.
4.2 SELECTION FOR EFFICIENT DECODING
Decoding efficiency of neural machine translation is still much lower than for traditional phrasebased translation. For NMT, the running time of beam search on a CPU is dominated by the last linear layer that computes a score for each target word. Vocabulary selection can therefore have a large impact on decoding speed. Figure 1 (left) shows that a reduced vocabulary of ∼ 460 types (144 msec) can achieve a 10X speedup over using the full 100k-vocabulary (∼ 1, 600 msec). Next we investigate the impact of reduced vocabularies on accuracy. Figure 1 (right) compares BLEU for the various selection strategies on a wide range of vocabulary sizes. Broadly, there are two groups of techniques: first, co-occurrence counts and bilingual embeddings (PCA) are not able to match the baseline performance (Full 100k) even with over 5k candidate words per sentence. Second, even with fewer than 1, 000 candidates per sentence, word alignments, phrase pairs and SVMs nearly match the full vocabulary accuracy.
1We will release the code with the camera ready.
Although co-occurrence counts and PCA have shown useful to measure semantic relatedness (Brown et al., 1992; Lebret & Collobert, 2014), it seems that considering the whole source sentence as the explanation of a target word without latent alignment variables undermines their selection ability. Overall, word alignments work as well or better than the other techniques relying on a wider input context (phrase pairs and SVMs). Querying a word table is also more efficient than querying a phrase-table or evaluating SVMs. We therefore use word alignment-based selection for the rest of our analysis.
Mi et al. (2016) suggest that adding common words to a selection technique could have a positive impact. We therefore consider adding the most frequent k words to our word alignment-based selection. Figure 2 shows that this actually has little impact on BLEU in our setting. In fact, the overlap of the results for n = 0 and 50 indicates that most of the top 50 words are already selected, even with small candidate sets.
Next we try to get a better sense of how precise selection is with respect to the words used by a human translator or with respect to the translations generated by the full vocabulary model. We use word alignments for this experiment. Figure 3 shows coverage with respect to the reference (left) and with respect to the output of the full vocabulary system (right). We do not count unknown words (UNK) in all settings, even if they may later be replaced by a source word (§4.1). Not counting UNKs is the reason why the full vocabulary models do not achieve 100% coverage in either setting. The two graphs show different trends: On the left, coverage with respect to the reference for the full vocabulary is 95.1%, while selection achieves 87.5% with a vocabulary of 614 words (3rd point on graph). However, when coverage is measured with respect to the full vocabulary system output,
the coverage of selection is very close to the full vocabulary model with respect to itself, i.e., when unknown words are not counted. In fact, the selection model covers over 99% of the non-UNK words in the full vocabulary output. This result shows that selection can recover almost all of the words which are effectively selected by a full vocabulary model while discarding many words which are not chosen by the full model.
What is the exact speed and accuracy trade-off when reducing the output vocabulary? Figure 4 plots BLEU against decoding speed. We pick a number of operating points from this graph for our final test set experiments (Table 1). For our best methods (word alignments, phrase alignments and SVMs) we pick points such that vocabularies are kept small while maintaining good accuracy compared to the full vocabulary setting. For co-occurrence counts and bilingual PCA we choose settings with comparable speed.
Our test results (Table 1) confirm the validation experiments. On English-German translation we achieve more than a 10-fold speed-up over the full vocabulary setting. Accuracy for the word alignment-based selection matches the full vocabulary setting on most test sets or decreases only slightly. For example with word alignment selection, the largest drop is on newstest2015 which
achieves 22.2 BLEU compared to 22.5 BLEU for the full setting on English-German; the best singlesystem neural setup at WMT15 achieved 22.4 BLEU on this dataset (Jean et al., 2015b). On EnglishRomanian, we achieve a speed-up of over 5 times with word alignments at 28.1 BLEU versus 27.9 BLEU for the full vocabulary baseline. This matches the state-of-the-art on this dataset (Sennrich et al., 2016b) from WTM16. The smaller speed-up on English-Romanian is due to the smaller vocab of the baseline in this setting which is 50k compared to 100k for English-German.
4.3 SELECTION FOR BETTER TRAINING
So far our evaluation focused on vocabulary selection for decoding, relying on a model trained with the full vocabulary. Next we address the question of whether the efficiency advantages observed for decoding translate to training as well. Selection at training may impact generalization performance either way: it assimilates training and testing conditions which could positively impact accuracy. However, training with a reduced vocabulary could result in worse parameter estimates, especially for rare words which would receive much fewer updates because they would be selected less often.
We run training experiments on WMT English to German with word alignment-based selection. In addition to the selected words, we include the target words of the reference and train a batch with the union of the sentence-specific vocabularies of all samples (Mi et al., 2016).
Figure 5 compares validation accuracy of models trained with selection or with the full vocabulary. Selection in both training and decoding gives a small accuracy improvement. However, this improvements disappears for vocabulary sizes of 500 and larger; we found the same pattern on other test sets. Similar to our decoding experiments, adding common words during training did not improve accuracy. Table 2 shows the impact of selection on training speed. Our bi-directional LSTM model (BLSTM) can process the same number of samples in 25% less time on a GPU with a batch size of 32 sentences. We do not observe changes in the number of epochs required to obtain the best validation BLEU.
The speed-ups for training are significantly smaller than for decoding (Table 1). This is because training scores the vocabulary exactly once per target position, while beam search has to score multiple hypotheses at each generation step.
We suspect that training is now dominated by the bi-directional LSTM encoder. To confirm this, we replaced the encoder with a simple average pooling model which encodes source words as the mean of word and position embeddings over a local context (Ranzato et al., 2016). Table 2 shows
that in this setting the efficiency gains of vocabulary selection are more substantial (40% less time per epoch). This model is not as accurate and achieves only 18.5 BLEU on newstest2015 compared to 22.5 for the bi-directional LSTM encoder. However, it shows that improving the efficiency of the encoder is a promising future work direction.
5 CONCLUSIONS
This paper presents a comprehensive analysis of vocabulary selection techniques for neural machine translation. Vocabulary selection constrains the output words to be scored to a small subset relevant to the current source sentence. The idea is to avoid scoring a high number of unlikely candidates with the full model which can be ruled out by simpler means.
We extend previous work by considering a wide range of simple and complex selection techniques including bilingual word co-occurrence counts, bilingual embeddings built with Hellinger PCA, word alignments, phrase pairs, and discriminative SVM classifiers. We explore the trade-off between speed and accuracy for different vocabulary sizes and validate results on two language pairs and several test sets.
Our experiments show that decoding speed-up can be reduced by up to 90% without compromising accuracy. Word alignments, bilingual phrases and SVMs can achieve high accuracy, even when considering fewer than 1, 000 word types per sentence.
At training time, we achieve a speed-up of up to 1.33 with a bi-directional LSTM encoder and 1.66 with a faster alternative. Efficiency increases are less pronounced during training because of two combined factors. First, vocabulary scoring at the final layer of the model is a smaller part of the computation compared to beam search. Second, state-of-the-art bi-directional LSTM encoders (Bahdanau et al., 2015) are relatively costly compared to scoring the vocabulary on GPU hardware. Efficiency gains from vocabulary selection highlight the importance of progress towards efficient, accurate encoder and decoder architectures.

This paper compares several strategies for guessing a short list of vocabulary for the target language in neural machine translation. The primary findings are that word alignment dictionaries work better than a variety of other techniques.
My take on this paper is that to have a significant impact, it needs to make the case for why one might want vocabulary rather than characters or sub word units like BPE. I think there are likely many very good reasons to do this that could be argued for (synthesize morphology, deal with transliteration, etc), but most of these would suggest some particular models and experiments, which are of course not in this paper. As it is, I think this paper is a useful but minor contribution that shows that word alignment is a good way of getting short lists, but it does not strongly make the case that we should abandon work in other directions.
Minor comments:
In addition to the SVM approach for modeling vocabulary, the discriminative word lexicon of Mauser et al. (2009) and the neural version of Ha et al. (2014) are also worth mentioning.
It would be useful to know what the coverage rate of the actual full vocabulary would be (rather than the 100k “full vocabulary”). Since presumably this technique could be used to work with much larger vocabularies.
When reducing the vocabulary size for training, the Mi et al. (2016) technique of taking the union of all the vocabularies in a mini batch seems like a rather strange objective. If the vocabulary of a single sentence is used, the probabilistic semantics of the translation model can still be preserved since p(e | f, vocab(f)) = p(e | f) if p(vocab(f) | f) = 1, i.e., is deterministic, which it is here. Whereas the objective is no longer a sensible probability model in the mini batch vocabulary case. Thus, while it may be a bit more difficult to implement, it seems like it would at least be a sensible comparison to make.


This paper aims to reduce test-time computational load of a deep neural network. Unlike previous methods which factorize a weight matrix into multiple real-valued matrices, our method factorizes both weights and activations into integer and noninteger components. In our method, the real-valued weight matrix is approximated by a multiplication of a ternary matrix and a real-valued co-efficient matrix. Since the ternary matrix consists of three integer values, {−1, 0,+1}, it only consumes 2 bits per element. At test-time, an activation vector that passed from a previous layer is also transformed into a weighted sum of binary vectors, {−1,+1}, which enables fast feed-forward propagation based on simple logical operations: AND, XOR, and bit count. This makes it easier to deploy a deep network on low-power CPUs or to design specialized hardware. In our experiments, we tested our method on three different networks: a CNN for handwritten digits, VGG-16 model for ImageNet classification, and VGG-Face for large-scale face recognition. In particular, when we applied our method to three fully connected layers in the VGG-16, 15× acceleration and memory compression up to 5.2% were achieved with only a 1.43% increase in the top-5 error. Our experiments also revealed that compressing convolutional layers can accelerate inference of the entire network in exchange of slight increase in error.
1 INTRODUCTION
It is widely believed that deeper networks tend to achieve better performance than shallow ones in various computer vision tasks. As a trade-off of such impressive improvements, deeper networks impose heavy computational load both in terms of processing time and memory consumption due to an enormous amount of network parameters. For example, VGG-16 model (Simonyan & Zisserman, 2015) requires about 528 MBytes to store the network weights where fully connected layers account for 89% of them. A large number of multiplications and additions must also be processed at each layer which prevent real-time processing, consume vast amounts of electricity, and require a large number of logic gates when implementing a deep network on a FPGA or ASIC.
This article addresses the above issues. Specifically, we aimed to reduce the test-time computational load of a pre-trained network. Since our approach does not depend on a network configuration (e.g. a choice of an activation function, layer structures, and a number of neurons) and acts as a post-processing of network training, pre-trained networks shared in a download site of MatConvNet (Vedaldi & Lenc, 2015) and Model Zoo (BVLC) can be compressed and accelerated. Our method is outlined in Figure 1. The main idea is to factorize both weights and activations into integer and non-integer components. Our method is composed of two building blocks, as shown below.
Ternary weight decomposition for memory compression: We introduce a factored representation where the real-valued weight matrix is approximated by a multiplication of a ternary basis matrix and a real-valued co-efficient matrix. While the ternary basis matrix is sufficiently informative to reconstruct the original weights, it only consumes 2 bits per element. The number of rows of the coefficient matrix is also smaller than that of the original weight matrix. These compact representations result in efficient memory compression.
Binary activation encoding for fast feed-forward propagation: It has been reported that an inner product between a ternary and binary vector can be computed extremely fast by using three logical operations: AND, XOR, and bit count (Ambai & Sato, 2014). To use this technique, we approximate the activation vector by a weighted sum of binary vectors. This binary encoding must be processed as fast as possible at test-time. To overcome this issue, we use a fast binary encoding method based on a small lookup table.
1.1 RELATED WORK
There have been extensive studies on accelerating and compressing deep neural networks, e.g., on an FFT-based method (Mathieu et al., 2014), re-parameterization of a weight matrix (Yang et al., 2015), pruning network connection (Han et al., 2015; 2016), and hardware-specific optimization (Vanhoucke et al., 2011). In the following paragraphs, we only review previous studies that are intimately connected to ours.
It was pointed out by Denil et al. (2013) that network weights have a significant redundancy. Motivated by this fact, researchers have been involved in a series of studies on matrix/tensor factorization (Jaderberg et al., 2014; Zhang et al., 2015). In these studies, a weight matrix (or tensor) was factorized by minimizing an approximation error of original weights or activations. Jaderberg et al. (2014) exploited 1-D separable filter decomposition to accelerate feed-forward propagation. Zhang et al. (2015) proposed low-rank approximation based on generalized SVD to compress an entire deep network. Taking into account the lessons learned from these best practices, we also exploit the redundancy of the weights.
There is an another series of studies, integer decomposition (Hare et al., 2012; Yuji et al., 2014; Ambai & Sato, 2014), which involved accelerating test-time speed of a classifier by using fast logical operations. Although their contributions are limited to a shallow architecture such as a linear SVM, they achieved a noticeable acceleration. In these approaches, a real-valued weight vector is approximated by a weighted sum of a few binary or ternary basis vectors. To use fast logical operations, they extracted binary features from an image. Hare et al. (2012) and Yuji et al. (2014) exploited binary basis vectors, and Ambai & Sato (2014) investigated a case of ternary basis to improve approximation quality.
In a manner of speaking, our method is a unified framework of matrix/tensor factorization and integer decomposition reviewed in the above and inherits both their advantages. While the weight matrix is factorized to exploit low-rank characteristics, the basis matrix is restricted to take only three integer values, {−1, 0,+1}. In contrast to recent binary weighted networks such as XNOR-Net (Rastegari et al., 2016) which quantizes both activations and weights during backpropagation, it is not necessary for our method to change training algorithms at all. We can benefit from recent sophisticated training techniques, e.g. batch normalization (Ioffe & Szegedy, 2015), in combination with our method. Furthermore, our method does not need (iterative) end-to-end retraining which is needed for several previous studies such as network pruning (Han et al., 2015; 2016) and distillation (Hinton et al., 2014).
2 NETWORK COMPRESSION MODEL
In this section, we introduce our compression model and discuss time and space complexity. We consider a convolutional layer with a filter size of wx × wy × c, where wx and wy are the spacial size and c is a number of input channels. If wx = wy = 1, we can regard this layer as a fully connected layer. This three dimensional volume is reshaped to form a DI dimensional vector where DI = wx×wy×c. The filter weights and biases can be formulated by W ∈ RDI×DO and b ∈ RDO , where DO is a number of output channels. Let x ∈ RDI denote an activation vector obtained by
vectorizing the corresponding three dimensional volume. In test-time, we need to compute W>x+b followed by a non-linear activation function.
In our compressed network, W is decomposed into two matrices before test-time as follows:
W ≈MwCw, (1)
where Mw ∈ {−1, 0,+1}DI×kw is a ternary basis matrix, Cw ∈ Rkw×DO is a co-efficient matrix, and kw is the number of basis vectors, respectively. Since Mw only takes the three values, it consumes only 2 bits per element. Setting a sufficiently small value to kw further reduces total memory consumption. From the viewpoint of approximation quality, it should be noted that a large number of elements in W takes close to zero values. To fit them well enough, a zero value must be included in the basis. The ternary basis satisfies this characteristic. In practice, the ternary basis gives better approximation than the binary basis, as we discuss in Section 3.
The activation vector x is also factored to the following form:
x ≈Mxcx + bx1, (2)
where Mx ∈ {−1,+1}DI×kx is a binary basis matrix, cx ∈ Rkx is a real-valued co-efficient vector, bx ∈ R is a bias, and kx is the number of basis vectors, respectively. Since elements of x are often biased, e.g., activations from ReLU take non-negative values and have a non-zero mean, bx is added to this decomposition model. While cx and bx reflect a range of activation values, Mx determines approximated activation values within the defined range. This factorization must be computed at test-time because the intermediate activations depend on an input to the first layer. However, in practice, factorizing x into Mx, cx, and bx requires an iterative optimization, which is very slow. Since a scale of activation values within a layer is almost similar regardless of x, we pre-computed canonical cx and bx in advance and only optimized Mx at test-time. As we discuss in Section 4, an optimal Mx under fixed cx and bx can be selected using a lookup table resulting in fast factorization.
Substituting Eqs.(1) and (2) into W>x+ b, approximated response values are obtained as follows:
W>x+ b ≈ (MwCw)>(Mxcx + bx1) + b = C>wM > wMxcx + bxC > wM > w1+ b. (3)
A new bias bxC>wM > w1+ b in Eq.(3) is pre-computable in advance, because Cw,Mw, and bx are fixed at test-time. It should be noted that M>wMx is a multiplication of the ternary and binary matrix, which is efficiently computable using three logical operations: XOR, AND, and bit count, as previously investigated (Ambai & Sato, 2014). After computing M>wMx, the two co-efficient components, cx and Cw, are multiplied from the right and left in this order. Since cx and Cw are much smaller than W, the total number of floating point computations is drastically reduced.
The time and space complexity are summarized in Tables 1 and 2. As can be seen from Table 1, most of the floating operations are replaced with logical operations. In this table, B means the bit width of a variable used in the logical operations, e.g., B = 64 if a type of unsigned long long is used in C/C++ language. Table 2 suggests that if kw is sufficiently smaller than DI and DO, the total size of Mw and Cw is reduced compared to the original parameterization.
Algorithm 1 Decompose W into Mw and Cw Require: W, kw Ensure: factorized components Mw and Cw.
1: R←W 2: for i← 1 to kw do 3: Initialize m(i)w by three random values {−1, 0,+1}. 4: Minimize ||R−m(i)w c(i)w ||2F by repeating the following two steps until convergence. 5: [Step 1] c(i)w ←m(i)>w R/m(i)>w m(i)w 6: [Step 2] mij ← arg min
α∈{−1,0,+1} ||rj − αc(i)w ||22, for j = 1, · · · , DI
7: R← R−m(i)w c(i)w 8: end for
3 TERNARY WEIGHT DECOMPOSITION
To factorize W, we need to solve the following optimization problem.
Jw = min Mw,Cw
||W −MwCw||2F . (4)
However, the ternary constraint makes this optimization very difficult. Therefore, we take an iterative approach that repeats rank-one approximation one by one, as shown in Algorithm 1. Let m (i) w ∈ {−1, 0,+1}DI×1 denote an i-th column vector of Mw and c(i)w ∈ R1×DO denote an i-th row vector of Cw. Instead of directly minimizing Eq. (4), we iteratively solve the following rank-one approximation,
J (i)w = min m(i)w ,c(i)w ||R−m(i)w c(i)w ||2F , (5)
where R is a residual matrix initialized by W. We applied alternating optimization to obtain m(i)w and c(i)w . If m (i) w is fixed, c (i) w can be updated using a least squares method, as shown in line 5 of Algorithm 1. If c(i)w is fixed, mij , the j-th element of m (i) w , can be independently updated by exhaustively verifying three choices {−1, 0,+1} for each j = 1, · · · , DI , as shown in line 6 of Algorithm 1, where rj is a j-th row vector of R. After the alternating optimization is converged, R is updated by subtracting m(i)w c (i) w and passed to the next (i+ 1)-th iteration. Comparison of binary constraints with ternary constraints can be seen in Appendix A.
4 BINARY ACTIVATION ENCODING
Binary decomposition for a given activation vector x can be performed by minimizing
Jx(Mx, cx, bx;x) = ||x− (Mxcx + bx1)||22. (6)
In contrast to the case of decomposing W, a number of basis vectors kx can be set to a very small value (from 2 to 4 in practice) because x is not a matrix but a vector. This characteristic enables an exhaustive search for updating Mx. Algorithm 2 is an alternating optimization with respect to Mx, cx, and bx. By fixing Mx, we can apply a least squares method to update cx and bx (in lines 3-4 of Algorithm 2). If cx and bx are fixed, m (j) x , the j-th row vector of Mx, is independent of any other m (j′) x , j′ 6= j. We separately solve DI sub-problems formulated as follows:
m(j)x = arg min β∈{−1,+1}1×kx (xj − (βcx + bx))2, j = 1, · · · , DI , (7)
where xj is a j-th element of x. Since kx is sufficiently small, 2kx possible solutions can be exhaustively verified (in line 5 of Algorithm 2).
Our method makes this decomposition faster by pre-computing canonical cx and bx from training data and only optimizing Mx at test-time using lookup table. This compromise is reasonable because of the following two reasons: (1) scale of activation values is similar regardless of vector elements
Algorithm 2 Decompose x into Mx, cx, and bx Require: x, kx Ensure: factorized components Mx, cx, and bx.
1: Initialize Mx by three random values {−1,+1}. 2: Minimize ||x− (Mxcx + bx1)||22 by repeating the following two steps until convergence. 3: [Step 1] Update cx and bx using a least squares method. 4: cx ← (M>xMx)−1M > x (x− bx1), bx ← 1>(x−Mxcx)/DI 5: [Step 2] Update m(j)x for each j = 1, · · ·DI by an exhaustive search that minimizes Eq.(7).
within a layer, and (2) cx and bx reflect a scale of approximated activation values. Knowing these properties, cx and bx are obtained by minimizing Jx(M̂x, cx, bx; x̂) ,where x̂ is constructed as follows. First, NT different activation vectors T ∈ {xi}NTi=1 are collected from randomly chosen NT training data. Second, n elements are randomly sampled from xi. The sampled nNT elements are concatenated to form a vector x̂ ∈ RnNT . We use cx and bx as constants at test-time, and discard M̂x.
At test-time, we only need to solve the optimization of Eq. (7) for each xj . This can be regarded as the nearest neighbour search in one-dimensional space. We call βcx + bx a prototype. There are 2kx possible prototypes because β takes 2kx possible combinations. The nearest prototype to xj and an optimal solution m(j)x can be efficiently found using a lookup table as follows.
Preparing lookup table: We define L bins that evenly divide one-dimensional space in a range from the smallest to largest prototype. Let x̂l denote a representative value of the l-th bin. This is located at the center of the bin. For each x̂l, we solve Eq. (7) and assign the solution to the bin.
Activation encoding: At test-time, xj is quantized into L-levels. In other words, xj is transformed to an index of the lookup table. Let pmax and pmin denote the largest and smallest prototype, respectively. We transform xj as follows:
q = (L− 1)(xj − pmin)/(pmax − pmin) + 1, (8) l̂ = min(max(bq + 1/2c, 1), L). (9)
The range from pmin to pmax is linearly mapped to the range from 1 to L by Eq. (8). The term q is rounded and truncated from 1 to L by the max and min function in Eq. (9). If L is sufficiently large, the solution assigned to the l̂-th bin can be regarded as a nearly optimal solution because the difference between xj and the center of the bin x̂l̂ becomes very small. We found that L = 4096 is sufficient. The time complexity of this encoding is O(DI).
5 EXPERIMENTS
We tested our method on three different convolutional neural networks: CNN for handwritten digits (LeCun et al., 1998), VGG-16 for ImageNet classification (Simonyan & Zisserman, 2015), and VGGFace for large-scale face recognition (Parkhi et al., 2015). To compute memory compression rate, a size of W and a total size of Mw and Cw were compared. To obtain a fair evaluation of computation time, a test-time code of forward propagation was implemented without using any parallelization scheme, e.g., multi-threading or SIMD, and was used for both compressed and uncompressed networks. The computation time includes both binary activation encoding and calculation of Eq. (3). We used an Intel Core i7-5500U 2.40-GHz processor.
5.1 CNN FOR HANDWRITTEN DIGITS
MNIST is a database of handwritten digits which consists of 60000 training and 10000 test sets of 28× 28 gray-scale images with ground-truth labels from 0 to 9. We trained our CNN by using an example code in MatConvNet 1.0-beta18 (Vedaldi & Lenc, 2015). Our architecture is similar to LeNet-5 (LeCun et al., 1998) but has a different number of input and output channels. Each layer’s configuration is shown below:
(conv5-20)(maxpool)(conv5-64)(maxpool)(fc1024-640)(relu)(fc640-10)(softmax), (10)
where the parameters of a convolutional layer are denoted as (conv<receptive field size>-<number of output channels>), and parameters of a fully connected layer are denoted as (fc<number of input channels>-<number of output channels>). The (maxpool) is 2×2 subsampling without overlapping. The error rate of this network is 0.86%.
We applied our method to the first fully connected layer (fc1024-640) and set n = 10 andNT = 1000 to learn cx and bx from randomly chosen nNT activations. The cases of kx = 1, 2, 3, 4 and kw = DO, DO/2, DO/5 were tested. This means that kw was set to 640, 320, and 128.
Figures 2(a) and (b) show the relationships among the increases in error rates, memory compression rates, and acceleration rates. It was observed that error rates basically improved along with increasing kx and saturated at kx = 4. It is interesting that kx = 2, only 2 bits per element for encoding an activation x, still achieved good performance. While the smaller kw achieved better compression and acceleration rate, error rates rapidly increased when kw = DO/5. One of the well balanced parameters was (kx, kw) = (4, DO/2) which resulted in 1.95× faster processing and a 34.4% memory compression rate in exchange of a 0.19% increase in the error rate.
5.2 VGG-16 FOR IMAGENET CLASSIFICATION TASK
A dataset of ILSVRC2012 (Russakovsky et al., 2015) consists of 1.2 million training, 50,000 validation, and 100,000 test sets. Each image represents one of 1000 object categories. In this experiment, we used a network model of VGG-16 (model D in (Simonyan & Zisserman, 2015)) that consists of 13 convolutional layers and 3 fully connected layers followed by a softmax layer. The architecture is shown below:
(input) · · · (fc25088-4096)(relu)(fc4096-4096)(relu)(fc4096-1000)(softmax), (11) where layers before the first fully connected layer are omitted.
First, all three fully connected layers were compressed with our algorithm. We set n = 10 and NT = 1000 to learn cx and bx from randomly chosen nNT activations. The cases of kx = 2, 3, 4 and kw = DO/2, DO/4, DO/8, DO/16 were tested. The case of kx = 1 was omitted because this setting resulted in a very high error rate. Note that each of the fully connected layers has different DO. The kw was independently set for each layer according to its DO. The top-5 error rates were evaluated on the validation dataset. The top-5 error rate of the original network is 13.4%.
The three lines with circles in Figure 3 show these results. It should be noted that much higher acceleration rates and smaller compression rates with small loss of accuracies were achieved than the case of the network for MNIST. Interestingly, the case of kw = DO/4 still performed well due to the low-rank characteristics of weights in the VGG-16 network.
Although the error rates rapidly increased when kw took much smaller values, we found that this could be improved by tuning kw of the third layer. More specifically, we additionally tested the
following cases. While kw was set to DO/2, DO/4, DO/8, and DO/16 for the first and second layers, kw was fixed to DO for the third layer. The kx was set to 4. This is plotted with a red line in Figure 3. In this way, the memory compression rate and acceleration rate noticeably improved. Setting appropriate parameters for each layer is important to improve the total performance. Table 3 shows the details of the best balanced case in which 15× faster processing and 5.2% compression rate were achieved in exchange of a 1.43% increase in error rate.
Next, we also tested to compress convolutional layers. In this experiment, kw and kx were set to DO and 4. This setting accelerates each of the layers averagely 2.5 times faster. Table 4 shows positions of compressed layers, top-5 errors, and acceleration rates of the entire network. Although kw and kx must be larger than those of fully connected layers to avoid error propagation, it is still beneficial for entire acceleration. In summary, while compressing fully connected layers is beneficial for reducing memory, compressing convolutional layers is beneficial for reducing entire computation time.
5.3 VGG-FACE FOR FACE RECOGNITION TASK
The VGG-Face (Parkhi et al., 2015) is a model for extracting a face descriptor. It consists of a similar structure to VGG-16. The difference is that VGG-Face has only two fully connected layers, as shown below. (input) · · · (fc25088-4096)(relu)(fc4096-4096). (12) This network outputs a 4096-dimensional descriptor. We can verify whether two face images are identical, by evaluating the Euclidean distance of two l2-normalized descriptors extracted from
them. In our experiment, we did not apply a descriptor embedding technique based on triplet loss minimization (Parkhi et al., 2015). Following the evaluation protocol introduced in a previous paper (Parkhi et al., 2015), we used Labeled Faces in the Wild dataset (LFW) (Huang et al., 2007), which includes 13,233 face images with 5,749 identities. The LFW defines 1200 positive and 1200 negative pairs for testing. We used the 2400 test pairs to compute ROC curve and equal error rate (EER). The EER is defined as an error rate at the ROC operating point where the false positive and false negative rates are equal. The EER of the original network is 3.8%.
First, the two fully connected layers were compressed using our algorithm. We set n = 10 and NT = 1000 to learn cx and bx from randomly chosen nNT activations. We tested the cases of kx = 1, 2, 3, 4, and kw = DO/2, DO/4, DO/8, DO/16. Figure 4 reveals an interesting fact that even the fastest and smallest network configuration, kx = 1 and kw = DO/16, had less impact on the EER, in contrast to the previous ImageNet classification task in which the recognition results were corrupted when kx = 1. This indicates that the 4096-dimensional feature space is well preserved regardless of such coarse discretization of both weights and activations.
Next, we also tested to compress convolutional layers. In this experiment, kw and kx were set to DO and 4 which are the the same setting used in Table 4. Table 5 shows positions of compressed layers and EERs. The acceleration rates were almost the same as the results shown in Table 4. This is because architecture of VGG-face is the same as VGG-16 and we used the same parameter for kw and kx. Interestingly, compressing multiple layers from 2nd to 10th still preserves the original EER. As can be seen from this table, our method works very well depending on a certain kind of machine learning task.
6 CONCLUSION
We proposed a network compression model that consists of two components: ternary matrix decomposition and binary activation encoding. Our experiments revealed that the proposed compression model is available not only for multi-class recognition but also for feature embedding. Since our approach is post-processing for a pre-trained model, it is promising that recent networks designed for semantic segmentation, describing images, stereo matching, depth estimation, and much more can also be compressed with our method. For future work, we plan to improve approximation error further by investigating the discrete optimization algorithm.
A BINARY VS. TERNARY
Figure 5 illustrates the reconstruction errors of a 4096×1000 weight matrix of the last fully connected layer in VGG-16 model (Simonyan & Zisserman, 2015). We tested both the binary and ternary constraints on Mw for comparison. The reconstruction error Jw monotonically decreased along with an increase in kw. It was clear that the ternary basis provided better reconstruction than the binary basis.

This paper explores a new quantization method for both the weights and the activations that does not need re-training. In VGG-16 the method reaches compression ratios of 20x and experiences a speed-up of 15x. The paper is very well written and clearly exposes the details of the methodology and the results.
My major criticisms are three-fold: for one, the results are not compared to one of the many other pruning methods that are described in section 1.1, and as such the performance of the method is difficult to judge from the paper alone. Second, there have been several other compression schemes involving pruning, re-training and vector-quantization [e.g. 1, 2, 3] that seem to achieve much higher accuracies, compression ratios and speed-ups. Hence, for the practical application of running such networks on low-power, low-memory devices, other methods seem to be much more suited. The advantage of the given method - other then possibly reducing the time it takes to compress the network - is thus unclear. In particular, taking a pre-trained network as a starting point for a quantized model that is subsequently fine-tuned might not take much longer to process then the method given here (but maybe the authors can quantify this?). Finally, much of the speed-up and memory reduction in the VGG-model seems to arise from the three fully-connected layers, in particular the last one. The speed-up in the convolutional layers is comparably small, making me wonder how well the method would work in all-convolutional networks such as the Inception architecture.
[1] Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,


We consider the ability of deep neural networks to represent data that lies near a low-dimensional manifold in a high-dimensional space. We show that deep networks can efficiently extract the intrinsic, low-dimensional coordinates of such data. Specifically we show that the first two layers of a deep network can exactly embed points lying on a monotonic chain, a special type of piecewise linear manifold, mapping them to a low-dimensional Euclidean space. Remarkably, the network can do this using an almost optimal number of parameters. We also show that this network projects nearby points onto the manifold and then embeds them with little error. Experiments demonstrate that training with stochastic gradient descent can indeed find efficient representations similar to the one presented in this paper.
1 INTRODUCTION
Deep neural networks have achieved state-of-the-art results in a variety of tasks. One possible reason for this remarkable success is that their hierarchical, layered structure may allow them to capture the geometric regularities of commonplace data. We support this hypothesis by exploring ways that networks can handle input data that lie on or near a low-dimenisonal manifold. In many problems, for example face recognition, data lie on or near manifolds that are of much lower dimension than the input space (Turk & Pentland, 1991; Basri & Jacobs, 2003; Lee et al., 2003), and that represent the intrinsic degrees of variation in the data.
We study the ability of deep networks to represent manifold data. We show that the initial layers of networks can approximate data that lies on high-dimensional manifolds using piecewise linear functions, and economically output their coordinates embedded in a low-dimensional Euclidean space. In fact, each new linear segment approximating the manifold can be represented by a single additional hidden unit, leading to a representation of manifold data that in some cases is nearly optimal in the number of parameters of the system. Subsequent layers of a deep network could
build upon these early layers, operating in lower dimensional spaces that more naturally represent the input data. We further show empirical results that suggest that training with stochastic gradient descent can find efficient representations akin to the one suggested in this paper.
We first show how this embedding can be done efficiently for manifolds consisting of monotonic chains of linear segments. We then show how these primitives can be combined to form linear approximations for more complex manifolds. This process is illustrated in Figure 1. We further show that when the data lies sufficiently close to their linear approximation, the error in the embedding will be small. Our constructions will use a feed-forward network with rectified linear unit (RELU) activation. We consider fully connected layers, although the treatment of complex manifolds that are divided into pieces (e.g., of monotonic chains) will be modular, resulting in many zero weights.
2 PRIOR WORK
Realistic learning problems, e.g., in vision and speech processing, involve high dimensional data. Such data is often governed by many fewer variables, producing manifold-like sub-structures in a high dimensional ambient space. A large number of dimensionality reduction techniques, such as principle component analysis and multi-dimensional scaling (Duda et al., 2012), Isomap (Tenenbaum et al., 2000), and local linear embedding (LLE) (Roweis & Saul, 2000), have been introduced. An underlying manifold assumption, which states that different classes lie in separate manifolds, has also guided the design of clustering and semi-supervised learning algorithms (Nadler et al., 2005; Belkin & Niyogi, 2003; Weston et al., 2008; Mobahi et al., 2009).
A number of recent papers examine properties of neural nets in light of this manifold assumption. Brahma et al. (2015) show empirically that the layers of deep networks trained with data that lies on a manifold progressively unfold that data into Euclidean spaces. They do not consider the mechanisms used to perform this unfolding. Rifai et al. (2011) trained a contractive auto-encoder to represent an atlas of manifold charts. Shaham et al. (2015) demonstrate that a 4-layer network can efficiently represent any function on a manifold through a trapezoidal wavelet decomposition. In both, each chart is represented independently, requiring an independent projection for each chart. Likewise, (Chui & Mhaskar, 2016) consider methods by which a neural network can map points on a manifold to a low-dimensional, Euclidean space, although they do not consider the efficiency of this representation in terms of hidden units or weights. We show that for monotonic chains we can reduce the size of the representation to near optimal by exploiting geometric relations between neighboring projection matrices, so an additional chart requires only a single hidden unit.
Another family of networks attempt to learn a “semantic” distance metric for training pairs, often by using a siamese network (Salakhutdinov & Hinton, 2007; Chopra et al., 2005; R. Hadsell & LeCun, 2006; Yi et al., 2014; Huang et al., 2015). These assume that the input space can be mapped nonlinearly by a network to produce the desired distances in a lower dimensional feature space. Giryes et al. (2016) shows that even a feed-forward neural network with random Gaussian weights embeds the input data in an output space while preserving distances between input items.
Another outstanding question is to what extent deep networks can be more efficient than shallow networks with a single hidden layer. Shallow networks are universal approximators (Cybenko, 1989). However, recent work demonstrates that deep networks can be exponentially more efficient in representing certain functions (Bianchini & Scarselli, 2014; Telgarsky, 2015; Eldan & Shamir, 2015; Delalleau & Bengio, 2011; Montufar et al., 2014; Cohen et al., 2015). On the other hand, (Ba & Caruana, 2014) shows empirically that in many practical cases a shallow network can be trained to mimic the behavior of a deep network. Our construction does not produce exponential gains, but does show that the early layers of a network can efficiently reduce the dimensionality of data that feeds into later layers.
3 MONOTONIC CHAINS OF LINEAR SEGMENTS
We construct networks that perform dimensionality reduction on data that lies on or near a manifold. We focus on feed-forward networks with RELU activation, i.e., max(x, 0). Clearly the output of such networks are continuous, piecewise linear functions of their input. It is therefore natural to ask whether they can embed piecewise-linear manifolds in a low-dimensional Euclidean space both ac-
Figure 2: Left: A continuous chain of linear segments (above) that can be flattened to lie in a single low-dimensional linear subspace (bottom). Right: A monotonic chain. Sk denotes the k’th segment in the chain. Hk is a hyperplane bounding the half-space that separates S1, ..., Sk from Sk+1, ..., SK .
curately and efficiently. In this section we construct such efficient networks for a class of manifolds that we call monotonic chains of linear segments, which are defined shortly. These will serve as building blocks for handling more general data that can be decomposed into monotonic chains.
We will consider data lying in a chain of linear segments, denoted C = S1 ∪ ...∪SK . Each segment Sk (1 ≤ k ≤ K) in the chain is a portion of some m-dimensional affine subspace of Rd, and the segments are connected to form a chain (Figure 2). We suppose that every two consecutive segments Sk−1 and Sk intersect, and that the intersection lies in an (m− 1)-dimensional affine subspace. We further assume that these chains can be flattened by isometry so that they may be represented in Rm. Note that any curve on C will be mapped to a curve of the same length in Rm on the flattened chain.
Each unit in the first hidden layer of a neural network will have a response of zero to input points that lie on a hyperplane, defined by its weights and bias term. This hyperplane bounds a half-space in which the output of the unit is positive; when the output is negative, RELU turns the output to zero. We say that a unit is active over the half-space in which its output is positive. There is a close connection between these hyperplanes and the embedding of a manifold, which we begin to develop with the following definition.
Definition: We say that a chain of K linear segments is monotonic (see Figure 2) when there exist a set of hyperplanes such that the k’th hyperplane separates the first k segments from the rest. Denoting the positive half-spaces associated with these hyperplanes as H1, H2, ..., HK−1, then Hk is bounded by a hyperplane that contains the intersection of Sk and Sk+1, and Sk+1, Sk+2, ..., SK ⊂ Hk while S1, S2, ..., Sk ⊂ HCk , where H C k is the complement of Hk. We can consider each halfspace to represent a hidden unit that is active (i.e., non-zero) over a subset of the regions. With a monotonic chain, the set of active units grows monotonically, so that, (Hk+1 ∩ C) ⊆ (Hk ∩ C). We can also define some additional units that are active over all the regions.
Below we show that monotonic chains can be embedded efficiently by networks with two layers of weights. These networks have d units in the input layer, a hidden layer with κ = K + m − 1 units that encodes the structure of the manifold, and an output layer with m units. Denote the weights in the first layer by a κ × d matrix A and further use a bias vector a0 ∈ Rκ. The second layer of weights is captured by a m × κ matrix B. The total number of weights in these two layers is (d + m + 1)(K + m − 1). This network maps a point x ∈ Rd to the embedding space Rm through
u = B[Ax + a0]+ where [.]+ denotes the RELU operation. For now we do not use a bias or RELU in the second level, but those will be used later when we discuss more complex manifolds.
A simple example of a manifold that can be represented efficiently with a neural network occurs when the data lies in a single m-dimensional affine subspace of Rd. Embedding can be done in this case with just one layer, with the matrix A of size m×d containing in its rows a basis parallel to the affine space. One way to extend this example to handle chains is by encoding each linear segment separately. Such encoding will require mK units in addition to units that use RELU to separate each segment from the rest of the segments. A related representation was used, e.g., in (Shaham et al., 2015). Below we show that monotonic chains can be encoded much more efficiently.
We next show how to construct the network (i.e., set the weights in A, a0, and B) to encode monotonic chains. Below we use the notation A(k) to denote the matrix formed by the first k rows of A, a0(k) is the vector containing the first k entries of a0, and B(k) the matrix including the first k columns of B. Therefore B(k)[A(k)x + a0(k)]+ will express the output of the network when only the first k hidden units are used. These will be set to recover the intrinsic coordinates of points in the first k segments in C; RELU ensures that subsequent hidden units do not affect the output for points in these segments.
For the construction we consider the pull-back of the standard basis of Rm onto the chain, producing a geodesic basis to the manifold. Note that to produce a local basis for the intrinsic coordinates of
points on the manifold, we only need a basis for each linear segment. This basis is expressed by a collection of d × m column-orthogonal matrices X(1), X(2), ..., X(K). Each matrix provides an orthogonal basis for one of the segments.
We will construct the network inductively. Suppose k = 1. We set A(1) = X(1) T , B(1) = I , and set a0(1) so that for all x ∈ C all the components of A(1)x + a0(1) are non-negative. Clearly, B(1)A(1) = X(1) T is an orthogonal projection matrix and B(1)A(1)X(1) = I . This shows that the network projects the orthonormal basis for the first segment into I , an orthonormal basis in Rm. Next we will show that B(k)A(k)X(k) = I for all k. This implies that B(k)A(k)x = X(k) T x, so there is no distortion in the projection. This will show that the network extends this basis throughout the monotonic chain in a consistent way.
Suppose we used m + k − 2 units to construct A(k−1), a0(k−1), and B(k−1) for the first k − 1 ≥ 1 segments. (For notational convenience we will next omit the superscript k − 1 for these matrices and vectors, so A = A(k−1), etc.) We will now use those to construct A(k), a0(k), and B(k). We do so by adding a node to the first hidden layer. The weights on the incoming edges to this node will be encoded by appending a row vector aT ∈ Rd to A and a scalar a0 to a0, and the weights on the outgoing edges will be encoded by appending a column vector b ∈ Rm to B. Our aim is to assign values to these vectors and scalar to extend the embedding to Sk.
By induction we assume that any x̃ ∈ S1 ∪ ... ∪ Sk−1 is embedded with no distortion to Rm by
ũ = B[Ax̃ + a0]+,
and that BAX = I . By monotonicity we further assume that Sk−1 ∩ Sk is m − 1 dimensional and there exists a hyperplane H with normal h ∈ Rd that contains this intersection with C − (S1 ∪ ...∪ Sk−1) lying completely on the side of H in the direction of h, while S1 ∪ ...∪ Sk−1 lies on the opposite side of H . We then set a = h and set a0 so that aT x̄+a0 = 0 for any point x̄ ∈ Sk−1∩Sk. (This is well defined since h is orthogonal to Sk−1 ∩ Sk.)
To determine b, we first rotate the bases X(k−1) (referred to as X below) and X(k) by a common, m × m matrix R, i.e., Y = XR and Y (k) = X(k)R so that Y = [w,y2, ...,ym] and Y (k) = [v,y2, ...,ym] with y2, ...,ym providing an orthogonal basis parallel to Sk−1 ∩ Sk. (This is equivalent to rotating the coordinate system in the embedded space and then pulling-back to the manifold.) Note that by the induction assumption BAY RT = I . We next aim to set b so that B(k)A(k)X(k) = I . We note that
B(k)A(k)X(k) = B(k)A(k)Y (k)RT = (BA + baT )Y (k)RT .
We aim to set b so that (BA + baT )Y (k)RT = I = BAY RT . Consider this equality first for the common columns y2, ...,ym of Y and Y (k). These columns are parallel to Sk−1 ∩ Sk, so that aT yj = 0 for 2 ≤ j ≤ m, implying equality for any choice of b. Consider next the left-most column of Y and Y (k), denoted respectively w and v, we get
(BA + baT )v = BAw.
This is satisfied if we set
b = 1
aT v BA(w − v).
We have constructed b so that the segments are embedded with consistent orientations. In Appendix A we show that they are also translated properly by a0, to create a continuous embedding. Note that by construction aT y + a0 ≤ 0 for all y ∈ S1 ∪ ... ∪ Sk−1 so RELU ensures that the embedding of the these segments will not be affected by the additional unit.
Finally, we note that the proposed representation of monotonic chains with a neural network is very efficient and uses only a few parameters beyond the degrees of freedom needed to define such chains. In particular, the definition of a chain requires specifying m basis vectors in Rd for one linear segment (exploiting orthonormality these require m(d − (m + 1)/2) parameters), with each additional segment specified by a 1D direction for the new segment (a unit vector in Rd specified by d − m − 1 parameters) and a direction in the previous segment to be replaced (specified by a unit vector in Rm, i.e. m − 1 parameters). The total number of degrees of freedom of a chain is therefore N = m(d − (m + 1)/2) + (K − 1)(d − 2). This is the number of parameters required to
specify a monotonic chain. Our construction requires N ′ = (K + m + 1)(d + m + 1) parameters. Specifically, note that for any choice of parameters K, d,m > 0, N ≥ (K + m − 1)(d − m − 2). We therefore obtain that
N ′ N ≤
(
1 + 2
K + m − 1
)(
1 + 2m + 3
d − m − 2
)
.
Assuming d,K + m >> 1 we get N ′
N / 1 + 2m d − m .
Since we normally expect that the dimension of the input space will be much greater than the dimension of the manifold, this ratio will be close to 1.
4 ERROR ANALYSIS
We now consider points that do not lie exactly on the monotonic chain, due to noise, or because we are approximating a non-linear manifold with piece-wise linear segments. Let p0 be a point on the segment Sj that is then perturbed by some small noise vector, δ, that is perpendicular to Sj , to produce the point p = p0 + δ. Ideally, the network would represent p using the coordinates of p0. In effect, the network would project all points onto the monotonic chain. If the network embeds p and p0 with coordinates p̂ and p̂0 we define the relative error of the embedding as ‖p̂−p̂0‖ δ . We now analyze this relative error. Our analysis assumes that ‖δ‖ is small enough that p and p0 lie in the same region so that they are both on the same side of all hyperplanes defined by the hidden units.
We note that given sufficient data that lies on the manifold, it is possible to learn local linear projections of the manifold that will embed it with zero relative error. This can be done with traditional manifold learning methods or by neural networks that contain a sufficiently large number of units. Zhang & Zha (2004) provides an error analysis that shows how the error of their approach depends on the noisiness and number of points in the training data, and the magnitude of the difference between the manifold and its linear approximation. Our contribution here is to analyze the error that can occur when a network learns the embedding very efficiently using a small number of units.
In Appendix B we show that in the worst case, the relative error of the embedding can be unbounded. This occurs when the monotonic chain has very high curvature, so that a separating hyperplane has to be nearly parallel to the segment that follows it. In this section we show that for more typical cases, the relative error will be a small constant.
We will consider a class of monotonic chains in which the total curvature between all segments is less than or equal to some angle T , and in each separating hyperplane is not too close to parallel to the next segment. We denote the angle between Sk−1 and Sk as θk−1. (This angle is well defined since Sk−1 and Sk intersect in an m − 1-dimensional affine space.) As before, we will drop the subscript when it is k − 1, and just write θ. Specifically, we define θ so that cos θ = vT w (where v and w are defined as in Sec. 3, as vectors perpendicular to Sk−1 ∩ Sk, and parallel to Sk−1 and Sk, respectively), defining θk similarly for any k. We then express our constraint on the curvature as ∑K−1
k=1 |θk| ≤ T .
Now let c be a constant such that we can bound aT v ≥ 1/c for any k− 1. c is a bound on the cosine of the angle between the normal to a separating hyperplane and a vector in the direction of the next segment. To understand this, recall that a is a unit vector normal to the hyperplane separating Sk−1 and Sk. By saying this bound holds for all k−1, we mean that we are able to choose the hyperplanes that divide the chain into segments so that the angle between the normal to each hyperplane and the following segment is not too big. We next bound the error in terms of c and ‖δ‖.
Let p = p0 + δ be as in the last section. We define the embedding error of p by E(p) =( B(k)A(k) − X(k)T ) p, where X(k) denotes the orthogonal projection to Sk, as in Sec. 3. Noting that, by the construction of our network, B(k)A(k)p0 = X(k)T p0 (since p0 is on Sk) and that X(k)T δ = 0 (due to the orthonormality of X(k)), we obtain E(p) = B(k)A(k)δ. The magnitude of the error therefore is scaled at most by the maximal singular value of B(k)A(k), denoted σk.
To bound σk we note that B(k)A(k) = BA+baT for k ≥ 2 (where, as before, we drop superscripts so that B denotes B(k−1)). Therefore, σk ≤ σk−1+|aT b|, where σk−1 denotes the largest singular
Figure 3: This plot shows the error in flattening the Swiss Roll. Relative error is constant in every segment, starting from zero for each monotonic chain and increasing with each segment. The absolute error (for display purposes this error is normalized by the maximal distance from the Swiss Roll to its linear approximation) behaves similarly, but vanishes at the end points of each segment where the Swiss Roll and its linear approximation coincide.
value of BA. Recall that ‖a‖ = 1 and b = 1 aT v BA(w − v). Note that w − v ≤ θk−1. Therefore, |aT b| ≤ cσk−1θk−1, from which we conclude that σk ≤ σk−1(1 + cθk−1).
Finally, note that B(1)A(1) = X(1)T , implying that σ1 = 1. We therefore obtain σk ≤∏k−1 j=1 (1 + cθj). Note that ∑k−1 j=1 θj ≤ T and so ∏k−1 j=1 (1 + cθj) ≤ (1 + cT k−1 ) k−1. Therefore, σk ≤ ( 1 + cTk−1 )k−1 ≤ ecT . We conclude that ‖E(p0 + δ)‖ ≤ ecT ‖δ‖.
Many segments of many monotonic chains can be divided using hyperplanes in which c is not too big, and may be as low as 1. For such manifolds, when a point is perturbed away from the manifold, its coordinates will not be changed by more than the magnitude of the perturbation times a small constant factor. For example, if T = π/4 and c = 1 then ek ≤ e π 4 ≈ 2.19. Note that rather than beginning at the start of the monotonic chain, we could ”begin” in the middle, and work our way out. That is, provide an orthonormal basis for the middle segment and add hidden units to represent the chain from the central segment toward either ends of the chain. This can reduce the total curvature from the starting point to either end by up to half. We further emphasize that this bound is not tight.
We conclude this section by showing the error obtained in using our construction in the ”Swiss Roll” example. To represent this data we use hidden units and their corresponding hyperplanes to divide the Roll into three monotonic chains (see Section 5 below for further details). We then divide each chain into segments, obtaining a total of 14 segments. Figure 1 shows the points that are input into the network, and the 2D representation that the network outputs. The points are color coded to allow the reader to identify corresponding points. In Figure 3 we further plot the absolute and relative error in embedding every point of the Swiss Roll due to the linear approximation used by the network. One can see that the Swiss Roll is unrolled almost perfectly. In fact, despite the relatively large angular extent of each monotonic chain (the three chains range between 126 to 166.5 degrees each in total curvature), the relative error does not exceed 2.5. (In fact, our bound for this case is very loose, amounting to 18.3 for 166.5◦.) The mean relative error is 0.98, indicating that the magnitude of the error is approximately the same as the distance of points to the approximating monotonic chains.
5 COMBINATIONS OF MONOTONIC CHAINS
To handle non-monotonic chains and more general piecewise linear manifolds that can be flattened we show that we can use a network to divide the manifold into monotonic chains, embed each of these separately, and then stitch these embeddings together. Suppose we wish to flatten a nonmonotonic chain that can be divided into L monotonic chains, M1,M2, ...ML. Let Al, a0l and Bl denote the matrices and bias used to represent the hidden units that flatten Ml, which has Kl segments. We suppose that a set of Jl hyperplanes (that is, a convex polytope) can be found that separate Ml from the other chains. Let Nl denote a matrix in which the rows represent the normals to these hyperplanes, oriented to point away from Ml. We can concatenate these vertically, letting A′l = [Al; Nl]. We next let Υ = −n1m×Jl where 1m×Jl denotes an m × Jl matrix containing all ones and n is a very large constant. Note that Bl has m rows. So we can define B′l = [Bl, Υ], where the matrices are concatenated horizontally.
We now note that if u = B′l[A ′ lx + a0l]+ then when x lies on Ml, u will contain the coordinates of x embedded in Rm, as before. When x lies on a different monotonic chain, u will be a vector with very small negative numbers. Applying RELU will therefore eliminate these numbers.
A′l and B ′ l therefore represent a module consisting of a two layer network that embeds one monotonic chain in Rm while producing zero for other chains. We can then stitch these values together. First,
we must rotate and translate each embedded chain so that each chain picks up where the previous one left off. Let Rl denote the rotation of each chain, and let b0l denote its appropriate translation. Then, for each chain, the appropriate coordinates are produced by
[RlB′l[A ′ lx + a0l]+ + b0l]+.
We can now concatenate these for all chains to produce the final network. We let A, a0 and b0 be the vertical concatenation of all A′l and a0l and b0l respectively, and let B be the block-diagonal concatenation of all RlB′l . The application of [B[Ax + a0]+ + b0]+ to x ∈ Ml will produce a vector with mL entries in which the m(l−1)+1, ...,ml entries give the embedded coordinates of x and the rest of the entries are zero. We can now construct a third layer of the network to then stitch these monotonic chains together. Let C denote a matrix of size m × mL obtained by concatenating horizontally L identity matrices of size m × m. Then the output of the network is:
u = C[B[Ax + a0]+ + b0]+.
Note, for example, that the first element of u is the sum of the first coordinates produced by each module in the first two layers. Each of these modules produces the appropriate coordinates for points in one monotonic chain, while producing 0 for points in all other monotonic chains.
We note that this summation may result in wrong values if there is overlap between the regions (which will generally be of zero measure). This can be rectified by replacing the summation due to C by max pooling, which allows overlap of any size. Together, all three layers will require(∑L
l=1 Jl + m + Kl − 1 )
+ (L + 1)m units. If the network is fully connected, this requires (∑L
l=1 Jl + m + Kl − 1 ) (d + Lm) + Lm2 weights.
Note that the size of this network depends on how many regions are required (L) and how many hyperplanes each region needs to separate it from the rest of the manifold (Ll). In the worst case, this can be quite large. Consider, for example, a 1D manifold that is a polyline that passes through every point with integer coordinates in Rd. To separate any portion of this polyline from the rest will require regions that are not unbounded, and so Ll = O(d) for all l. We expect that many manifolds can be divided appropriately using many fewer hyperplanes. We have shown this for the example of a Swiss rolls (Figure 1).
6 EXPERIMENTS
Up to this point we have theoretically analyzed the representational capacity of a deep network. Our primary result is to show that data lying on a monotonic chain can be efficiently flattened by a network with two hidden layers, using m+k−1 hidden units in the first layer, and m units in the second layer. An important question is whether real networks trained with stochastic gradient descent can uncover such efficient representations. In this section we address that question experimentally.
We do not expect that a trained network will always produce the constructions developed in this paper. First,we note that our constructions provide an upper bound; more efficient representations possible. So we predict that m + k − 1 or fewer hidden units are needed. Second, a trained network may settle in a local minimum, and not produce an efficient embedding, even though one might be possible. To determine whether a particular architecture can produce a good embedding, we train networks with multiple random starting points, and select the solutions that produce very low error.
To determine the number of hidden units needed to create effective embeddings, we generate data on monotonic chains in which we vary the dimension of the manifold, m, and the number of segments,
k. An example in which m = 2 and k = 7 is shown in Figure 5. Note that there is some skew in the chain, so that none of the dimensions can be trivially embedded by a single linear projection. We sample 40,000 points on the manifold. We then train a regressor, with a varying number of hidden units, using the squared difference between the ground truth distance between pairs of embedded points and the distance computed by the network as a loss function. This simulates non-linear metric learning. For each condition, we repeat training 15 times, and report the minimum error in the objective (see Figure 4). We can see that for each curve the error has dropped to an asymptote near zero when h = m + k − 1, just as our theory predicts.
In Figure 5 we show a typical example produced for a 2D manifold with seven segments, shown in a 3D space. Portions of hyperplanes correspond to six hidden units. This solution resembles our constructions in several ways. One hyperplane is active over the entire chain, while the other hyperplanes intersect the manifold at the intersection of consecutive segments. The solution differs from our construction in that some hyperplanes are used to handle two segments of the manifold; it is even more efficient than our construction. And two hyperplanes, at the top, intersect the manifold in the same location. These hidden units have weights with opposite signs, producing positive outputs for different segments. For reasons of space and simplicity, we do not discuss these constructions theoretically, but it is straightforward to show that they can also produce efficient embeddings.
We perform a final experiment to get a sense of whether such embeddings can occur with more realistic data. We generate images of a face with azimuth ranging from 0 to 50 degrees, and with elevation ranging from 0 to 8 degrees. As a loss function, we use an L2 norm between the m output units and the true azimuth and elevation. Because the images have many pixels, and the amount of training data is limited, a fully connected network would overfit the data if we use each pixel as an input dimension. Consequently, we perform PCA before training to reduce the faces to a 3D space, which also allows us to visualize the input and resulting network (see Figure 6). We can see that the data forms an approximately 2D manifold, but that it is much messier than with our previous, synthetic data. The resulting embedding captures the azimuth and elevation reasonably well, but with some noise (eg., it does not form a perfect grid). We can also see that the hyperplanes associated with the first hidden layer of the network also resemble our construction, with individual units periodically intersecting the manifold as it curves.
7 DISCUSSION
We show that deep networks can represent data that lies on a low-dimensional manifold with great efficiency. In particular, when using a monotonic chain to approximate some component of the data, the addition of only a single neural unit can produce a new linear segment to approximate a region of the data. This suggests that deep networks may be very effective devices for such dimensionality reduction. It also may suggest new architectures for deep networks that encourage this type of dimensionality reduction.
We also feel that our work makes a larger point about the nature of deep networks. It has been shown by Montufar et al. (2014) that a deep network can divide the input space into a large number of regions in which the network computes piecewise linear functions. Indeed, the number of regions can be exponential in the number of parameters of the network. While this suggests a source of great power, it also suggests that there are very strong constraints on the set of regions that can be constructed, and the set of functions that can be computed. Our work shows one way in which a single hidden unit can control the variation in the linear function that a network computes in two neighboring regions; it can shape this function to follow a manifold that contains the data.
ACKNOWLEDGEMENTS
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
This research is also based upon work supported by the Israel Binational Science Foundation Grant No. 2010331 and Israel Science Foundation Grants No. 1265/14.
The authors thank Angjoo Kanazawa and Shahar Kovalsky for their helpful comments.
A CONTINUITY OF EMBEDDING
In Section 3 of our paper we defined the weight matrices A(k) and B(k) and the bias vector a0(k) that map an input vector x to its geodesic coordinates on the manifold. We showed that this construction indeed maps points on Sk to their geodesic coordinates, so that this coordinate system is consistent in orientation with the coordinates assigned to the previous segments S1, ..., Sk−1. It is now left to show that the bias a0(k) is chosen properly to create a continuous embedding.
Consider a point x ∈ Sk. Denote by x̄ its projection onto Sk−1 ∩ Sk, so that x = x̄ + βv for a scalar β. Denoting the embedded coordinates of x by u,
u = B(k)(A(k)x + a0 (k)).
We want to verify that as β tends to 0 u will coincide with the embedding of x̄ due to Sk−1, i.e.,
ū = B(Ax̄ + a0).
In our construction, B(k) is obtained from B by appending the column vector b to its right side, and A(k) is obtained from A by appending the row vector aT to its bottom, so that B(k)A(k) = BA + baT . Recall further that a0(k) is obtained from a0 by appending the scalar a0 at its end. We therefore obtain
u = (BA + baT )x + Ba0 + a0b.
Replacing x = x̄ + βv we obtain
u = (BA + baT )x̄ + β(BA + baT )v + Ba0 + a0b.
Since a = h, aT x̄ + ao = 0 and we get
u = B(Ax̄ + a0) + β(BA + ba T )v,
which coincides with ū when β → 0, implying that the embedding is extended continuously to Sk. Note that by construction aT y + a0 ≤ 0 for all y ∈ S1 ∪ ... ∪ Sk−1 so RELU ensures that the embedding of these segments will not be affected by the additional unit.
B WORST-CASE ERROR
In this section we show that the error obtained while embedding noisy points using our construction can in principle be unbounded. As we show below, this happens when we are forced to choose hyperplanes that are almost parallel to the segments they represent. In contrast, Section 4.1 of our paper shows that we can bound the error in many reasonable scenarios.
To show that the error can be unbounded, we consider a simple case in which the piecewise linear manifold consists of three connected 1D line segments, S1, S2 and S3, with 2D vertices respectively of (0, 0) and (N, 0), (N, 0) and (N, ), and (N, ) and (0, ). N is very large, and is very small (see Figure 7). Since three segments compose a 1D manifold, three hidden units defining three hyperplanes, H1, H2 and H3 (lines) will be needed to represent the manifold. In addition, a single output unit will sum the results of these units to produce the geodesic distance from the origin to any point on the three segments.
Using our construction in Section 3 of the paper we get the embedding f(p) = B[Ap + a0]+ with
B =
(
1, 1 q2 ,− 1 r1
(
2 + q1 q2
))
, A = ( 1 0 q1 q2 r1 r2 ) , a0 = ( 0 q3 r3 ) .
Note that the first row of A uses the standard orthogonal projection (x, y) → x; the two other rows of A and a0 separate the three segments with (1) q1, q2 > 0 and q1/q2 ≤ /N and q3 = −q1N set so that the separator H2 goes through (N, 0), and (2) r1 < 0, r2 > 0 and r1/r2 ≥ − /N , and r3 = −r1N − r2 set so that the separator H3 goes through (N, ). It can be easily verified that in this setup points on the first segment (x, 0), 0 ≤ x ≤ N are mapped to x, points (N, y), 0 ≤ y ≤ on the second segment are mapped to N + y, and points (x, ), 0 ≤ x ≤ N on the third segment are mapped to N + + (N − x).
Ideally, we would want p to be embedded to the same point as p0. Let E(p) = f(p) − f(p0). Clearly E(p) = B(k)A(k)δ. It can be readily verified that, under these conditions, when p0 ∈ S1 then E(p) = 0; when p0 ∈ S2 then E(p) = (1 + q1/q2)δ, and when p0 ∈ S3 then E(p) = (1 − (r2/r1)(2 + q2/q1))δ. Therefore, there is no error in embedding p for p0 ∈ S1. The error in embedding p with p0 ∈ S2 is small and bounded (since q1/q2 ≤ /N , assuming is small and N is large), while the error in embedding p when p0 ∈ S3 can be huge since −r2/r1 ≥ N/ . In the next section we show that this can only happen when there is a large angle between a segment and the normal to the previous separating hyperplane.
C CLASSIFICATION
In experiments in the body of this paper we have demonstrated that the theoretical constructions that we analyze can arise when networks are trained to solve regression problems that map points on the manifold to their low-dimensional embeddings. An interesting question is whether similar embeddings may be learned by a network that is trained to classify points that lie on a low-dimensional manifold when it is more efficient to represent the boundaries of these classes in the embedded space than it is in the ambient space. In this Appendix, we describe some very preliminary experiments that address this question.
First we note that the embeddings that arise in solving classification problems may be much less constrained and therefore more complex than those that arise in regression problems. The regression loss function directs the network to learn the known, ground truth coordinates of the embedded manifold. Only an isometric unfolding of the manifold will satisfy this condition. While this isometric embedding will facilitate classification as well, there may be many non-isometric unfoldings that will be equally useful in classification.
As a simple example of this, suppose a monotonic chain contains two classes that are linearly separable, once the chain is isometrically embedded in a low-dimensional space. If instead of an isometric embedding, we allow a related embedding in which each segment of the chain undergoes a different linear transformation that stretches it in the direction of the linear separator, or orthogonal to the separator, the classes will still be linearly separable in the transformed, non-isometric embedding.
As another example, no mapping of the manifold to a low-dimensional space will allow for correct classification if it maps two points from different classes to the same point in the low-dimensional space. However, classification may not be affected if two points from the same class are mapped to the same point. So when points from only one class appear near the boundary between two segments, a network may learn a mapping in which the points from two segments overlap in the low-dimensional space.
It is an open and rather complex problem to determine which mappings of the input to low-dimension may be suitable for classification of a particular set of labeled points. However, we stress that the main point of our paper is to show that when isometric embeddings can be used to solve a problem, a deep network can efficiently represent such embeddings. It is certainly possible that the network can also efficiently find alternate embeddings that are equally useful.
Bearing this in mind, we have designed some simple classification tasks and examined the embeddings that they give rise to in a neural network. We stress that these experiments are quite preliminary, and should be taken as intriguing examples that can help motivate future work.
In our experiments we created monotonic chains with seven segments, similar to those used in our earlier experiments. We generate 20,000 points that lie on each chain. To label these points with classes, we unfolded the chain and intersected it with several lines, varying the number. These lines form an arrangement on the 2D unfolded manifold; we labeled each region of the arrangement, which is a convex polygon, as a separate class. We did this randomly, selecting arrangements in which classes tended to span multiple segments.
We then trained a network to perform classification. After the input layer, the next layer contained between five and eight hidden units. This was followed by a layer containing two hidden units. This was followed by another layer with 10-30 units, and an output layer with a unit for each class. Relu was used between layers, with softmax for the loss function. The layer containing two units essentially represents a two-dimensional embedding of the input. The previous layer could be used to represent the constructions developed in this paper, while the subsequent layer can be used to classify the data in the low-dimensional space. This architecture allows us to easily extract the embedding that the network has learned.
Figure 8 shows a typical example of the results. On the left we plot the input points, color coded to indicate their class. On the right, we plot each point at its embedded location, color coded to indicate to which segment it belongs. The embedding preserves the order and continuity of the segments. In several cases each segment has been approximately transformed by a different linear transformation. In the case of the red and green colored segments on the right, there is some overlap. Looking at the left-hand figure we can see that in this case, points near the boundary between the two segments belong to the same class. So this folding over of the segments in the embedding does not interfere with the network’s ability to correctly classify the points.
In general, this embedding meets our expectations, showing that the monotonic chain can be very efficiently mapped to a low-dimensional space using very few units, in a way that enables accurate classification. It would be interesting in future work to determine the class of mappings that can be instantiated efficiently by a network, and to understand how these relate to different classification problems. It would also be interesting to design classification problems that can only be solved using isometric embeddings, and to determine whether these embeddings can be found by neural networks.
D DEEPER NETWORKS
We also note that the previously developed constructions can be applied recursively, producing a deeper network that progressively approximates data using linear subspaces of decreasing dimension. That is, we may first divide the data into a set of segments that each lie in a low dimensional subspace whose dimension is higher than the intrinsic dimension of the data. Then we may subdivide each segment into a set of subsegments of lower dimension, using a similar construction, and deeper layers of the network. These subsegments may represent the original data, or they be further subdivided by additional layers, until we ultimately produce subsegments that represent the data.
We first illustrate this hierarchical approach with a simple example that requires only one extra layer in the hierarchy. Consider a monotonic chain of K, m2-dimensional linear segments that collectively lie in a m1-dimensional linear subspace, L, of a d-dimensional space, with m2 < m1. We can construct the first hidden layer with m1 units that are active over the entire monotonic chain, so that their gradient directions form an orthonormal basis for L. The output of this layer will contain the coordinates in L of points on the monotonic chain. These can form the input to two layers that then flatten the chain, as described in Section 3.
In Section 3 we had already shown how to flatten the manifold with two layers that take their input directly from the input space. Here we accomplish the same end with an extra layer. However, this construction, while using more layers, may also use fewer parameters. The construction in Section 3 required d(m2 +K−1) parameters. Our new construction will require dm1 +m1(m2 +K−1) parameters. Note that as K increases, the number of parameters used in the first construction increases in proportion to d, while in the second construction the parameters increase only in proportion to m1. Consequently, the second construction can be much more economical when K is large and m1 is small.
In much the same way, we could represent a manifold using a hierarchy of chains. The first layers can map a m1-dimensional chain to a linear m1-dimensional output space. The next layers can select an m2-dimensional chain that lies in this m1-dimensional space, and map it to an m2-dimensional space. This process can repeat indefinitely, but whether it is economical will depend on the structure of the manifold.

The paper presents an analysis of the ability of deep networks with ReLU functions to represent particular types of low-dimensional manifolds. Specifically, the paper focuses on what the authors call "monotonic chains of linear segments", which are essentially sets of intersecting tangent planes. The paper presents a construction that efficiently models such manifolds in a deep net, and presents a basic error analysis of the resulting construction.
While the presented results are novel to the best of my knowledge, they are hardly surprising (1) given what we already know about the representational power of deep networks and (2) given that the study selects a deep network architecture and a data structure that are very "compatible". In particular, I have three main concerns with respect to the results presented in this paper:
(1) In the last decade, there has been quite a bit of work on learning data representations from sets of local tangent planes. Examples that spring to mind are local tangent space analysis of Zhang & Zha (2002), manifold charting by Brand (2002) and alignment of local models by Verbeek, Roweis, and Vlassis (2003). None of this work is referred to in related work, even though it seems highly relevant to the analysis presented here. For instance, it would be interesting to see how these old techniques compare to the deep network trained to produce the embedding of Figure 6. This may provide some insight into the inductive biases the deep net introduces: does it learn better representations that non-parametric techniques because it has better inductive biases, or does it learn worse representations because the loss being optimized is non-convex?
(2) It is difficult to see how the analysis generalizes to more complex data in which local linearity assumptions on the data manifold are vacuous given the sparsity of data in high-dimensional space, or how it generalizes to deep network architectures that are not pure ReLU networks. For instance, most modern networks use a variant of batch normalization; this already appears to break the presented analyses.
(3) The error bound presented in Section 4 appears vacuous for any practical setting, as the upper bound on the error is exponential in the total curvature (a quantity that will be quite large in most practical settings). This is underlined by the analysis of the Swiss roll dataset, of which the authors state that the "bound for this case is very loose". The fact that the bound is already so loose for this arguably very simple manifold makes that the error analysis may tell us very little about the representational power of deep nets.
I would encourage the authors to address issue (1) in the revision of the paper. Issue (2) and (3) may be harder to address, but is essential that they are addressed for the line of work pioneered by this paper to have an impact on our understanding of deep learning.
Minor comments:
- In prior work, the authors only refer to fully supervised siamese network approaches. These approaches differ from that taken by the authors, as their approach is unsupervised. It should be noted that the authors are not the first to study unsupervised representation learners parametrized by deep networks: other important examples are deep autoencoders (Hinton & Salakhutdinov, 2006 and work on denoising autoencoders from Bengio's group) and parametric t-SNE (van der Maaten, 2009).
- What loss do the authors use in their experiments? Using "the difference between the ground truth distance ... and the distance computed by the network" seems odd, because it encourages the network to produce infinitely large distances (to get a loss of minus infinity). Is the difference squared?