LATENT SEQUENCE DECOMPOSITIONS
Sequence-to-sequence models rely on a fixed decomposition of the target sequences into a sequence of tokens that may be words, word-pieces or characters. The choice of these tokens and the decomposition of the target sequences into a sequence of tokens is often static, and independent of the input, output data domains. This can potentially lead to a sub-optimal choice of token dictionaries, as the decomposition is not informed by the particular problem being solved. In this paper we present Latent Sequence Decompositions (LSD), a framework in which the decomposition of sequences into constituent tokens is learnt during the training of the model. The decomposition depends both on the input sequence and on the output sequence. In LSD, during training, the model samples decompositions incrementally, from left to right by locally sampling between valid extensions. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve a WER of 9.6%.
1 INTRODUCTION
Sequence-to-sequence (seq2seq) models (Sutskever et al., 2014; Cho et al., 2014) with attention (Bahdanau et al., 2015) have been successfully applied to many applications including machine translation (Luong et al., 2015; Jean et al., 2015), parsing (Vinyals et al., 2015a), image captioning (Vinyals et al., 2015b; Xu et al., 2015) and Automatic Speech Recognition (ASR) (Chan et al., 2016; Bahdanau et al., 2016a).
Previous work has assumed a fixed deterministic decomposition for each output sequence. The output representation is usually a fixed sequence of words (Sutskever et al., 2014; Cho et al., 2014), phonemes (Chorowski et al., 2015), characters (Chan et al., 2016; Bahdanau et al., 2016a) or even a mixture of characters and words (Luong & Manning, 2016). However, in all these cases, the models are trained towards one fixed decomposition for each output sequence.
We argue against using fixed deterministic decompositions of a sequence that has been defined a priori. Word segmented models (Luong et al., 2015; Jean et al., 2015) often have to deal with large softmax sizes, rare words and Out-of-Vocabulary (OOV) words. Character models (Chan et al., 2016; Bahdanau et al., 2016a) overcome the OOV problem by modelling the smallest output unit, however this typically results in long decoder lengths and computationally expensive inference. And even with mixed (but fixed) character-word models (Luong & Manning, 2016), it is unclear whether such a predefined segmentation is optimal. In all these examples, the output decomposition is only
∗Work done at Google Brain.
a function of the output sequence. This may be acceptable for problems such as translations, but inappropriate for tasks such as speech recognition, where segmentation should also be informed by the characteristics of the inputs, such as audio.
We want our model to have the capacity and flexibility to learn a distribution of sequence decompositions. Additionally, the decomposition should be a sequence of variable length tokens as deemed most probable. For example, language may be more naturally represented as word pieces (Schuster & Nakajima, 2012) rather than individual characters. In many speech and language tasks, it is probably more efficient to model “qu” as one output unit rather than “q” + “u” as separate output units (since in English, “q” is almost always followed by “u”). Word piece models also naturally solve rare word and OOV problems similar to character models.
The output sequence decomposition should be a function of both the input sequence and the output sequence (rather than output sequence alone). For example, in speech, the choice of emitting “ing” as one word piece or as separate tokens of “i” + “n” + “g” should be a function of the current output word as well as the audio signal (i.e., speaking style).
We present the Latent Sequence Decompositions (LSD) framework. LSD does not assume a fixed decomposition for an output sequence, but rather learns to decompose sequences as function of both the input and the output sequence. Each output sequence can be decomposed to a set of latent sequence decompositions using a dictionary of variable length output tokens. The LSD framework produces a distribution over the latent sequence decompositions and marginalizes over them during training. During test inference, we find the best decomposition and output sequence, by using beam search to find the most likely output sequence from the model.
2 LATENT SEQUENCE DECOMPOSITIONS
In this section, we describe LSD more formally. Let x be our input sequence, y be our output sequence and z be a latent sequence decomposition of y. The latent sequence decomposition z consists of a sequence of zi ∈ Z where Z is the constructed token space. Each token zi need not be the same length, but rather in our framework, we expect the tokens to have different lengths. Specifically, Z ⊆ ∪ni=1Ci where C is the set of singleton tokens and n is the length of the largest output token. In ASR , C would typically be the set of English characters, while Z would be word pieces (i.e., n-grams of characters).
To give a concrete example, consider a set of tokens {“a”, “b”, “c”, “at”, “ca”, “cat”}. With this set of tokens, the word “cat” may be represented as the sequence “c”, “a”, “t”, or the sequence “ca”, “t”, or alternatively as the single token “cat”. Since the appropriate decomposition of the word “cat” is not known a priori, the decomposition itself is latent.
Note that the length |za| of a decomposition za need not be the same as the length of the output sequence, |y| (for example “ca”, “t” has a length of 2, whereas the sequence is 3 characters long). Similarly, a different decomposition zb (for example the 3-gram token “cat”) of the same sequence may be of a different length (in this case 1).
Each decomposition, collapses to the target output sequence using a trivial collapsing function y = collapse(z). Clearly, the set of decompositions, {z : collapse(z) = y}, of a sequence, y, using a non-trivial token set, Z , can be combinatorially large. If there was a known, unique, correct segmentation z∗ for a given pair, (x,y), one could simply train the model to output the fixed deterministic decomposition z∗. However, in most problems, we do not know the best possible decomposition z∗; indeed it may be possible that the output can be correctly decomposed into multiple alternative but valid segmentations. For example, in end-to-end ASR we typically use characters as the output unit of choice (Chan et al., 2016; Bahdanau et al., 2016a) but word pieces may be better units as they more closely align with the acoustic entities such as syllables. However, the most appropriate decomposition z∗ for a given is (x,y) pair is often unknown. Given a particular y, the best z∗ could even change depending on the input sequence x (i.e., speaking style).
In LSD, we want to learn a probabilistic segmentation mapping from x → z → y. The model produces a distribution of decompositions, z, given an input sequence x, and the objective is to maximize the log-likelihood of the ground truth sequence y. We can accomplish this by factorizing
and marginalizing over all possible z latent sequence decompositions under our model p(z|x; θ) with parameters θ:
log p(y|x; θ) = log ∑ z p(y, z|x; θ) (1)
= log ∑ z p(y|z,x)p(z|x; θ) (2)
= log ∑ z p(y|z)p(z|x; θ) (3)
where p(y|z) = 1(collapse(z) = y) captures path decompositions z that collapses to y. Due to the exponential number of decompositions of y, exact inference and search is intractable for any nontrivial token set Z and sequence length |y|. We describe a beam search algorithm to do approximate inference decoding in Section 4.
Similarly, computing the exact gradient is intractable. However, we can derive a gradient estimator by differentiating w.r.t. to θ and taking its expectation:
∂ ∂θ log p(y|x; θ) = 1 p(y|x; θ) ∂ ∂θ ∑ z p(y|x, z)p(z|x; θ) (4)
= 1 p(y|x; θ) ∑ z p(y|x, z)∇θp(z|x; θ) (5)
= 1 p(y|x; θ) ∑ z p(y|x, z)p(z|x; θ)∇θ log p(z|x; θ) (6)
= Ez∼p(z|x,y;θ) [∇θ log p(z|x; θ)] (7)
Equation 6 uses the identity ∇θfθ(x) = fθ(x)∇θ log fθ(x) assuming fθ(x) 6= 0 ∀ x. Equation 7 gives us an unbiased estimator of our gradient. It tells us to sample some latent sequence decomposition z ∼ p(z|y,x; θ) under our model’s posterior, where z is constraint to be a valid sequence that collapses to y, i.e. z ∈ {z′ : collapse(z′) = y}. To train the model, we sample z ∼ p(z|y,x; θ) and compute the gradient of ∇θ log p(z|x; θ) using backpropagation. However, sampling z ∼ p(z|y,x; θ) is difficult. Doing this exactly is computationally expensive, because it would require sampling correctly from the posterior – it would be possible to do this using a particle filtering like algorithm, but would require a full forward pass through the output sequence to do this.
Instead, in our implementation we use a heuristic to sample z ∼ p(z|y,x; θ). At each output time step t when producing tokens z1, z2 · · · z(t−1), we sample from zt ∼ p (zt|x,y, z<t, θ) in a left-toright fashion. In other words, we sample valid extensions at each time step t. At the start of the training, this left-to-right sampling procedure is not a good approximation to the posterior, since the next step probabilities at a time step include probabilities of all future paths from that point.
For example, consider the case when the target word is “cat”, and the vocabulary includes all possible characters and the tokens “ca”, and “cat”. At time step 1, when the valid next step options are “c”, “ca”, “cat”, their relative probabilities reflect all possible sequences “c*”, “ca*”, “cat*” respectively, that start from the first time step of the model. These sets of sequences include sequences other than the target sequence “cat”. Thus sampling from the distribution at step 1 is a biased procedure.
However, as training proceeds the model places more and more mass only on the correct hypotheses, and the relative probabilities that the model produces between valid extensions gets closer to the posterior. In practice, we find that the when the model is trained with this method, it quickly collapses to using single character targets, and never escapes from this local minima1. Thus, we follow an -greedy exploration strategy commonly found in reinforcement learning literature (Sutton & Barto, 1998) – we sample zt from a mixture of a uniform distribution over valid next tokens and p (zt|x,y, z<t, θ). The relative probability of using a uniform distribution vs. p (·|x,y, z<t, θ) is varied over training. With this modification the model learns to use the longer n-grams of characters appropriately, as shown in later sections.
1One notable exception was the word piece “qu” (“u” is almost always followed by “q” in English). The model does learn to consistently emit “qu” as one token and never produce “q” + “u” as separate tokens.
3 MODEL
In this work, we model the latent sequence decompositions p(z|x) with an attention-based seq2seq model (Bahdanau et al., 2015). Each output token zi is modelled as a conditional distribution over all previously emitted tokens z<i and the input sequence x using the chain rule:
p(z|x; θ) = ∏ i p(zi|x, z<i) (8)
The input sequence x is processed through an EncodeRNN network. The EncodeRNN function transforms the features x into some higher level representation h. In our experimental implementation EncodeRNN is a stacked Bidirectional LSTM (BLSTM) (Schuster & Paliwal, 1997; Graves et al., 2013) with hierarchical subsampling (Hihi & Bengio, 1996; Koutnik et al., 2014):
h = EncodeRNN(x) (9)
The output sequence z is generated with an attention-based transducer (Bahdanau et al., 2015) one zi token at a time:
si = DecodeRNN([zi−1, ci−1], si−1) (10) ci = AttentionContext(si,h) (11)
p(zi|x, z<i) = TokenDistribution(si, ci) (12)
The DecodeRNN produces a transducer state si as a function of the previously emitted token zi−1, previous attention context ci−1 and previous transducer state si−1. In our implementation, DecodeRNN is a LSTM (Hochreiter & Schmidhuber, 1997) function without peephole connections.
The AttentionContext function generates ci with a content-based MLP attention network (Bahdanau et al., 2015). Energies ei are computed as a function of the encoder features h and current transducer state si. The energies are normalized into an attention distribution αi. The attention context ci is created as a αi weighted linear sum over h:
ei,j = 〈v, tanh(φ(si, hj))〉 (13) αi,j = exp(ei,j)∑ j′ exp(ei,j′) (14)
ci = ∑ j αi,jhj (15)
where φ is linear transform function. TokenDistribution is a MLP function with softmax outputs modelling the distribution p(zi|x, z<i).
4 DECODING
During inference we want to find the most likely word sequence given the input acoustics:
ŷ = argmax y ∑ z log p(y|z)p(z|x) (16)
however this is obviously intractable for any non-trivial token space and sequence lengths. We simply approximate this by decoding for the best word piece sequence ẑ and then collapsing it to its corresponding word sequence ŷ:
ẑ = argmax z
log p(z|x) (17)
ŷ = collapse(ẑ) (18)
We approximate for the best ẑ sequence by doing a left-to-right beam search (Chan et al., 2016).
5 EXPERIMENTS
We experimented with the Wall Street Journal (WSJ) ASR task. We used the standard configuration of train si284 dataset for training, dev93 for validation and eval92 for test evaluation. Our input features were 80 dimensional filterbanks computed every 10ms with delta and delta-delta acceleration normalized with per speaker mean and variance as generated by Kaldi (Povey et al., 2011). The EncodeRNN function is a 3 layer BLSTM with 256 LSTM units per-direction (or 512 total) and 4 = 22 time factor reduction. The DecodeRNN is a 1 layer LSTM with 256 LSTM units. All the weight matrices were initialized with a uniform distribution U(−0.075, 0.075) and bias vectors to 0. Gradient norm clipping of 1 was used, gaussian weight noise N (0, 0.075) and L2 weight decay 1e−5 (Graves, 2011). We used ADAM with the default hyperparameters described in (Kingma & Ba, 2015), however we decayed the learning rate from 1e−3 to 1e−4. We used 8 GPU workers for asynchronous SGD under the TensorFlow framework (Abadi et al., 2015). We monitor the dev93 Word Error Rate (WER) until convergence and report the corresponding eval92 WER. The models took around 5 days to converge.
We created our token vocabularyZ by looking at the n-gram character counts of the training dataset. We explored n ∈ {2, 3, 4, 5} and took the top {256, 512, 1024} tokens based on their count frequencies (since taking the full n-cartesian exponent of the unigrams would result in an intractable number of tokens for n > 2). We found very minor differences in WER based on the vocabulary size, for our n = {2, 3} word piece experiments we used a vocabulary size of 256 while our n = {4, 5} word piece experiments used a vocabulary size of 512. Additionally, we restrict 〈space〉 to be a unigram token and not included in any other word pieces, this forces the decompositions to break on word boundaries.
Table 1 compares the effect of varying the n sized word piece vocabulary. The Latent Sequence Decompositions (LSD) models were trained with the framework described in Section 2 and the (Maximum Extension) MaxExt decomposition is a fixed decomposition. MaxExt is generated in a left-to-right fashion, where at each step the longest word piece extension is selected from the vocabulary. The MaxExt decomposition is not the shortest |z| possible sequence, however it is a deterministic decomposition that can be easily generated in linear time on-the-fly. We decoded these models with simple n-best list beam search without any external dictionary or Language Model (LM).
The baseline model is simply the unigram or character model and achieves 14.76% WER. We find the LSD n = 4 word piece vocabulary model to perform the best at 12.88% WER or yielding a 12.7% relative improvement over the baseline character model. None of our MaxExt models beat our character model baseline, suggesting the maximum extension decomposition to be a poor decomposition choice. However, all our LSD models perform better than the baseline suggesting the LSD framework is able to learn a decomposition better than the baseline character decomposition.
We also look at the distribution of the characters covered based on the word piece lengths during inference across different n sized word piece vocabulary used in training. We define the distribution of the characters covered as the percentage of characters covered by the set of word pieces with the same length across the test set, and we exclude 〈space〉 in this statistic. Figure 1 plots the
distribution of the {1, 2, 3, 4, 5}-ngram word pieces the model decides to use to decompose the sequences. When the model is trained to use the bigram word piece vocabulary, we found the model to prefer bigrams (55% of the characters emitted) over characters (45% of the characters emitted) in the LSD decomposition. This suggest that a character only vocabulary may not be the best vocabulary to learn from. Our best model, LSD with n = 4 word piece vocabulary, covered the word characters with 42.16%, 39.35%, 14.83% and 3.66% of the time using 1, 2, 3, 4 sized word pieces respectively. In the n = 5 word piece vocabulary model, the LSD model uses the n = 5 sized word pieces to cover approximately 2% of the characters. We suspect if we used a larger dataset, we could extend the vocabulary to cover even larger n ≥ 5. The MaxExt model were trained to greedily emit the longest possible word piece, consequently this prior meant the model will prefer to emit long word pieces over characters. While this decomposition results in the shorter |z| length, the WER is slightly worse than the character baseline. This suggest the much shorter decompositions generated by the MaxExt prior may not be best decomposition. This falls onto the principle that the best z∗ decomposition is not only a function of y∗ but as a function of (x,y∗). In the case of ASR, the segmentation is a function of the acoustics as well as the text.
Table 2 compares our WSJ results with other published end-to-end models. The best CTC model achieved 27.3% WER with REINFORCE optimization on WER (Graves & Jaitly, 2014). The previously best reported basic seq2seq model on WSJ WER achieved 18.0% WER (Bahdanau et al., 2016b) with Task Loss Estimation (TLE). Our baseline, also a seq2seq model, achieved 14.8% WER. Main differences between our models is that we did not use convolutional locational-based
priors and we used weight noise during training. The deep CNN model with residual connections, batch normalization and convolutions achieved a WER of 11.8% (Zhang et al., 2017) 2.
Our LSD model using a n = 4 word piece vocabulary achieves a WER of 12.9% or 12.7% relatively better over the baseline seq2seq model. If we combine our LSD model with the CNN (Zhang et al., 2017) model, we achieve a combined WER of 9.6% WER or 35.1% relatively better over the baseline seq2seq model. These numbers are all reported without the use of any language model.
Please see Appendix A for the decompositions generated by our model. The LSD model learns multiple word piece decompositions for the same word sequence.
6 RELATED WORK
Singh et al. (2002); McGraw et al. (2013); Lu et al. (2013) built probabilistic pronunciation models for Hidden Markov Model (HMM) based systems. However, such models are still constraint to the conditional independence and Markovian assumptions of HMM-based systems.
Connectionist Temporal Classification (CTC) (Graves et al., 2006; Graves & Jaitly, 2014) based models assume conditional independence, and can rely on dynamic programming for exact inference. Similarly, Ling et al. (2016) use latent codes to generate text, and also assume conditional independence and leverage on dynamic programming for exact maximum likelihood gradients. Such models can not learn the output language if the language distribution is multimodal. Our seq2seq models makes no such Markovian assumptions and can learn multimodal output distributions. Collobert et al. (2016) and Zweig et al. (2016) developed extensions of CTC where they used some word pieces. However, the word pieces are only used in repeated characters and the decompositions are fixed.
Word piece models with seq2seq have also been recently used in machine translation. Sennrich et al. (2016) used word pieces in rare words, while Wu et al. (2016) used word pieces for all the words, however the decomposition is fixed and defined by heuristics or another model. The decompositions in these models are also only a function of the output sequence, while in LSD the decomposition is a
2 For our CNN architectures, we use and compare to the “(C (3 × 3) / 2) × 2 + NiN” architecture from Table 2 line 4.
function of both the input and output sequence. The LSD framework allows us to learn a distribution of decompositions rather than learning just one decomposition defined by a priori.
Vinyals et al. (2016) used seq2seq to outputs sets, the output sequence is unordered and used fixed length output units; in our decompositions we maintain ordering use variable lengthed output units. Reinforcement learning (i.e., REINFORCE and other task loss estimators) (Sutton & Barto, 1998; Graves & Jaitly, 2014; Ranzato et al., 2016) learn different output sequences can yield different task losses. However, these methods don’t directly learn different decompositions of the same sequence. Future work should incorporate LSD with task loss optimization methods.
7 CONCLUSION
We presented the Latent Sequence Decompositions (LSD) framework. LSD allows us to learn decompositions of sequences that are a function of both the input and output sequence. We presented a biased training algorithm based on sampling valid extensions with an -greedy strategy, and an approximate decoding algorithm. On the Wall Street Journal speech recognition task, the sequenceto-sequence character model baseline achieves 14.8% WER while the LSD model achieves 12.9%. Using a a deep convolutional neural network on the encoder with LSD, we achieve 9.6% WER.
ACKNOWLEDGMENTS
We thank Ashish Agarwal, Philip Bachman, Dzmitry Bahdanau, Eugene Brevdo, Jan Chorowski, Jeff Dean, Chris Dyer, Gilbert Leung, Mohammad Norouzi, Noam Shazeer, Xin Pan, Luke Vilnis, Oriol Vinyals and the Google Brain team for many insightful discussions and technical assistance.
A LEARNING THE DECOMPOSITIONS
We give the top 8 hypothesis generated by a baseline seq2seq character model, a Latent Sequence Decompositions (LSD) word piece model and a Maximum Extension (MaxExt) word piece model. We note that “shamrock’s” is an out-of-vocabulary word while “shamrock” is in-vocabulary. The ground truth is “shamrock’s pretax profit from the sale was one hundred twenty five million dollars a spokeswoman said”. Note how the LSD model generates multiple decompostions for the same word sequence, this does not happen with the MaxExt model.
Ta bl
e 3:
To p
hy po
th es
is co
m pa
rs io
n be
tw ee
n se
q2 se
q ch
ar ac
te rm
od el
,L SD
w or
d pi
ec e
m od
el an
d M
ax E
xt w
or d
pi ec
e m
od el . n H yp ot he si s
L og
Pr ob
R ef
er en ce - sh am
ro ck
’s pr
et ax
pr ofi
tf ro
m th
e sa
le w
as on
e hu
nd re
d tw
en ty
fiv e
m ill
io n
do lla
rs a
sp ok
es w
om an
sa id
-
C ha
ra ct
er se
q2 se q 1 c| h| a| m |r| o| c| k|
’|s ||
p| r|e
|t| a|
x| |p
|r| o|
f|i |t|
|f| r|o
|m ||
t|h |e
|| s|a
|l| e|
|w |a
|s| |o
|n |e
|| h|
u| n|
d| r|e
|d ||
t|w |e
|n |t|
y| |f|
i|v |e
|| m
|i| l|l
|i| o|
n| |d
|o |l|
l|a |r|
s| |a
|| s|p
|o |k
|e |s|
w |o
|m |a
|n ||
s|a |i|
d -1
.3 73
86 8
2 c|
h| a|
m |r|
o| x|
|p |r|
e| t|a
|x ||
p| r|o
|f| i|t
|| f|r
|o |m
|| t|h
|e ||
s|a |l|
e| |w
|a |s|
|o |n
|e ||
h| u|
n| d|
r|e |d
|| t|w
|e |n
|t| y|
|f| i|v
|e ||
m |i|
l|l |i|
o| n|
|d |o
|l| l|a
|r| s|
|a ||
s|p |o
|k |e
|s| w
|o |m
|a |n
|| s|a
|i| d
-2 .2
53 58 1 3 c| h| a| m |r| o| c| k| s| |p |r| e| t|a |x || p| r|o |f| i|t || f|r |o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |a || s|p |o |k |e |s| w |o |m |a |n || s|a |i| d -3 .4 82 71 3 4 c| h| a| m |r| o| c| k| ’|s || p| r|e |t| a| x| |p |r| o| f|i |t| |f| r|o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |o |f| |s| p| o| k| e| s|w |o |m |a |n || s|a |i| d -3 .4 93 95 7 5 c| h| a| m |r| o| d| ’|s || p| r|e |t| a| x| |p |r| o| f|i |t| |f| r|o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |a || s|p |o |k |e |s| w |o |m |a |n || s|a |i| d -3 .8 85 18 5 6 c| h| a| m |r| o| x| |p |r| e| t|a |x || p| r|o |f| i|t || f|r |o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |o |f| |s| p| o| k| e| s|w |o |m |a |n || s|a |i| d -4 .3 73 68 7 6 c| h| a| m |r| o| c| ’|s || p| r|e |t| a| x| |p |r| o| f|i |t| |f| r|o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |a || s|p |o |k |e |s| w |o |m |a |n || s|a |i| d -5 .1 48 48 4 8 c| h| a| m |r| o| c| k| s| |p |r| e| t|a |x || p| r|o |f| i|t || f|r |o |m || t|h |e || s|a |l| e| |w |a |s| |o |n |e || h| u| n| d| r|e |d || t|w |e |n |t| y| |f| i|v |e || m |i| l|l |i| o| n| |d |o |l| l|a |r| s| |o |f| |s| p| o| k| e| s|w |o |m |a |n || s|a |i| d -5 .6 02 79 3 W or d Pi ec e M od el M ax im um E xt en si on 1 sh |a m |ro |c k| ’s || pr e| ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -1 .1 55 20 3 2 sh |a m |ro |x || pr e| ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -3 .0 31 33 0 3 sh |a r|r o| x| |p re |ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -3 .0 74 76 2 4 sh |e || m || ro |x || pr e| ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -3 .8 15 66 2 5 sh |e || m ar |x || pr e| ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -3 .8 80 76 0 6 sh |a r|r o| ck |s| |p re |ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -4 .0 83 27 4 7 sh |e || m || ro |c k| ed || pr e| ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -4 .8 78 02 5 8 sh |e || m || ro |c k| s| |p re |ta |x || pr o| fi| t| |fr om || th e| |sa |le || w as || on e| |h u| nd |re d| |tw |e nt |y || fiv e| |m il| lio n| |d ol l|a rs || a| |sp |o k| es |w o| m an || sa id -5 .1 21 49 0 W or d Pi ec e M od el L at en tS eq ue nc e D ec om po si tio ns 1 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| lio |n || do ll| a| r|s || a| |sp |o k| e| s|w o| m a| n| |sa id -2 8. 11 14 85 2 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| li| o| n| |d ol l|a r|s || a| |sp |o k| e| s|w o| m a| n| |sa id -2 8. 17 28 78 3 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| lio |n || do ll| a| r|s || a| |sp |o k| e| s|w |o m |a |n || sa id -2 8. 45 33 81 4 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| li| o| n| |d ol l|a |r| s| |a || sp |o k| e| s|w |o m |a |n || sa id -2 9. 10 31 84 5 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| lio |n || do ll| a| r|s || a| |sp |o k| e| s|w |o m |a |n || sa |id -2 9. 15 96 60 6 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| lio |n || do ll| a| r|s || a| |sp |o |k |e |s| w |o |m a| n| |sa id -2 9. 16 41 41 7 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| li| o| n| |d ol l|a |r| s| |a || sp |o k| e| s|w |o m |a |n || sa i|d -2 9. 16 93 10 8 sh |a |m |ro |c |k |’s || pr e| ta |x || pr o| fi| t| |fr o| m || t|h |e || sa |l| e| |w as || on |e || hu |n |d r|e |d || t|w e| nt |y || fiv |e || m il| li| o| n| |d ol l|a |r| s| |a || sp |o k| e| s|w |o m |a |n || sa |id -2 9. 80 99 37

This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g.,
I. McGraw, I. Badr, and J. Glass, "Learning lexicons form speech using a pronunciation mixture model," in IEEE Transactions on Audio, Speech, and Language Processing, 2013
L. Lu, A. Ghoshal, S. Renals, "Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition", in Proc. ASRU
R. Singh, B. Raj, and R. Stern, "Automatic generation of subword units for speech recognition systems,"  in IEEE Transactions on Speech and Audio Processing, 2002
It would be interesting to put this work in the context by linking it to some previous works in the HMM framework.
Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper. Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model. Finally, "O(5) days to converge" sounds a bit odd to me.


This paper shows how, by combining prior and supervised representations, one can create architectures that lead to nearly state-of-the-art results on standard benchmarks, which mean they perform as well as a deep network learned from scratch. We use scattering as a generic and fixed initialization of the first layers of a deep network, and learn the remaining layers in a supervised manner. We numerically demonstrate that deep hybrid scattering networks generalize better on small datasets than supervised deep networks. Scattering networks could help current systems to save computation time, while guaranteeing the stability to geometric transformations and noise of the first internal layers. We also show that the learned operators explicitly build invariances to geometrical variabilities, such as local rotation and translation, by analyzing the third layer of our architecture. We demonstrate that it is possible to replace the scattering transform by a standard deep network at the cost of having to learn more parameters and potentially adding instabilities. Finally, we release a new software, ScatWave, using GPUs for fast computations of a scattering network that is integrated in Torch. We evaluate our model on the CIFAR10, CIFAR100 and STL10 datasets.
1 INTRODUCTION
Deep architectures builds generic and low-dimensional representations that lead to state-of-the-art results on tasks such as classification (He et al., 2015), games (Silver et al., 2016), or generative models (Radford et al., 2015). These architectures are designed as cascades of non-linear modules that are fully learned. This paper addresses several questions: is it necessary to learn each module? Can a scattering networks replace the first layers? What are the potential benefits?
Hybrid architectures composed of a supervised representation learned on top of an unsupervised representation (Philbin et al., 2007) have been progressively abandoned for the end-to-end training approach (LeCun et al., 2010). Understanding the nature of the cascade of deep operators is difficult (Szegedy et al., 2013), since they are learned via back-propagation, and not layer-wise. However, the learned features appear to be transferable to other datasets and helpful for classification(Zeiler & Fergus, 2014), which implies that the learned representations have captured generic properties for image classification tasks.
Scattering representations (Mallat, 2012) are predefined and generic representations which only require the learning of a few hyper parameters. They consist of a cascade of wavelet transforms and modulus nonlinearities are have proven to be successful in classification tasks such as textures (Bruna & Mallat, 2013b; Sifre & Mallat, 2013), small digits (Bruna & Mallat, 2013b), sounds (Andén & Mallat, 2014) or complex image datasets with unsupervised representations (Oyallon & Mallat, 2015). Nevertheless, these representations do not adapt to the specific bias of each dataset and there is a huge performance gap between supervised and unsupervised representations(Oyallon & Mallat, 2015).
A convnet is typically a cascade of convolutional layers and nonlinearities, followed by a final average pooling or a sequence of fully connected layers. They lead to state of the art results on CIFAR10 and CIFAR100 (Zagoruyko & Komodakis, 2016). Some related work to ours (Perronnin & Larlus, 2015) proposed to replace the first layers of convolution by a combination of SIFT and Fisher vec-
tors, while learning on top of it a cascade of fully connected layers. This is a hybrid representation in the sense that it combines an unsupervised learned representation and a supervised learned MLP. The numerical results they obtained are competitive on ImageNet with the first AlexNet architecture (Krizhevsky et al., 2012), while saving computations.
Training a state of the art deep network requires a huge amount of labeled data. Several works tried to tackle this difficulty by developing unsupervised algorithm applied to deepnetwork: for instance evaluating on CIFAR10 an unsupervised generative adversarial method (GAN) pretrained on a subset of Imagenet-1K (Radford et al., 2015). In a setting where few annotated data are available, training a deep network is hard and requires a lot of regularization, yet a semisupervised learning algorithm applied to a GAN can improve even more the accuracy on CIFAR10, as in Salimans et al. (2016). Yet, if few data are available, such as in medical imaging, training a deep network from scratch is more complicated: one can only use imagenet pre-trained features (Carneiro et al., 2015).
Section 2 describes our model, which is a cascade of a scattering network and a convnet. We explain how we build our scattering network, describe its stability properties and exhibit our learning pipeline. Section 3 shows that our network provides competitive results on CIFAR10, CIFAR100 and STL10, while having theoretical guarantees for its representations, in both setting with limited data or not. The experiments can be reproduced using ScatWave 1, an implementation of our algorithm in Torch, which we make publicly available. More details about the software are available in the Appendix A.
2 TOWARDS A HYBRID ARCHITECTURE
We construct an architecture that consists of two blocks: the first is based on the scattering transform and involves no learning; the second is a classical convnet. In this section, we describe these architectures and their properties.
2.1 SCATTERING NETWORK
A scattering network belongs to the class of convolutional networks whose filters are predefined as wavelets (Oyallon & Mallat, 2015). The construction of this network has mathematical foundations (Mallat, 2012), meaning it is well understood, relies on few parameters and is stable, in contrast deep networks. Stability properties are discussed in Subsection 2.1.2 and Apprendix B. Besides, most of the parameters of this representation does not need to be adapted to the bias of the dataset (Oyallon & Mallat, 2015), making it a suitable generic representation.
2.1.1 A CASCADE OF WAVELETS AND MODULUS
In this section, we briefly recall the definition of the scattering transform. It is the cascade of wavelet transforms, and modulus nonlinearity which is finally spatially averaged. Since a modulus is nonexpansive, and a wavelet transform is a linear isometry, a scattering transform is also non-expansive. The local averaging of this representation thus builds a local invariance to translation. In this paper, we only consider a second order scattering network, on the group of translations.
Consider a signal x(u), u ∈ R2 and an integer J ∈ N, which is the spatial scale of our scattering transform. Let φJ be an averaging with a spatial window of scale 2J . (for example, a Gaussian averaging) Applying a subsampled averaging AJx(u) = x ? φJ(2Ju) builds an approximate invariant to translations smaller than 2J , but it also results in a loss of high frequencies that are necessary to discriminate signals. We define S0x = AJx as the order 0 scattering.
A solution to avoid this loss is provided by wavelets. A wavelet is an integrable and localized function in the Fourier and space domain, with a 0 average. A family of wavelets is obtained by dilating a complex mother wavelet ψ (for example, a Morlet wavelet) such that ψj,θ(u) = 122j ψ(r−θ u 2j ), where r−θ is the rotation by −θ, and j ≥ 0 is the scale of the wavelet. A given wavelet ψj,θ has thus its energy concentrated at a scale j, in the angular sector θ. Let K ∈ N be an integer representing the number of angles of our operator. A wavelet transform is the convolution of a signal with the family of wavelets introduced above, with an appropriate downsampling, i.e.
1Code can be found here: https://github.com/edouardoyallon/scatwave
W1x(u, θ1, j1) = {x ? ψj1,θ1(2j1u)}j≤J,θ=2π lL ,1≤l≤L. Observe that j and θ have been discretized: the wavelet is chosen to be selective in angle and localized in Fourier, thus the sampling is chosen such that (θ1, j)→ W1x(u, θ1, j1) is regular enough. Besides, the wavelet transform has been spatially oversampled by a factor 1. The wavelet parameters and this discretization were already chosen in (Oyallon & Mallat, 2015), where this representation is shown to be generic, so we have used the same hyper-parameters. In their case, {AJx,W1x} is approximatively an isometry on the set of signals with limited bandwidth, and this implies the energy of the signal is preserved. This operator belongs to the category of multi-resolution analysis operator, each filter being excited by a specific scale and angle, but the output coefficients are not invariant to translation. We can not apply AJ to W1x since it gives a trivial invariant, namely 0.
We build the first order scattering coefficients. Applying a point-wise modulus to W1x, followed by an averaging AJ allows us to build an invariant. If the mother wavelet is analytic, then |W1x| is more regular (Bernstein et al., 2013) which implies that the support in Fourier of |W1x| is more likely to be contained in a lower frequency domain than W1x. Thus, AJ preserves the energy of |W1x|. In this case, it is possible to define S1x = AJ |W1|x, which can also be written as: S1x(u, θ1, j1) = |x ? ψj1,θ1 | ? φJ(2Ju); this is the order 1 scattering. It is consequently invariant to translation up to 2J .
Once more, applying a second wavelet transform W2 = W1 on each channels permits the recovery of the high-frequency loss due to the averaging applied to the first order, leading to S2x = AJ |W2||W1|, which can also be written as S2x(u, θ1, θ2, j1, j2) = ||x ?ψj1,θ1 | ? ψj2,θ2 | ? φJ(2Ju). We only compute increasing paths, i.e. j1 < j2 because non-increasing paths bear no energy (Bruna & Mallat, 2013b). We do not compute higher order scatterings, because their energy has been shown experimentally not to be meaningful (Bruna & Mallat, 2013b).
2.1.2 COVARIANCE AND STABILITY OF THE REPRESENTATION
In this section, we develop mathematical properties that are obtained by wavelets. Covariance with a group of variability permits building a localized invariant via local averaging. The degree of invariance will be decided by a supervised algorithm, in order to be adapted to the bias of the problem of classification. Here, the parameter J corresponds to a trade-off of invariance in translation and discrimination to adjust AJ , and it has to be learned from the data. By construction, as a cascade of convolutions, a scattering network is covariant with translations. Let rθ.x , x(r−θu) be a rotated signal by θ. The representation is still covariant with the rotation in the following sense:
S1(rθ.x)(u, θ1) = S1x(r−θu, θ1 − θ) , rθ.(S1x)(u, θ1)
S2(rθ.x)(u, θ1, θ2) = S2x(r−θu, θ1 − θ, θ2 − θ) , rθ.(S2x)(u, θ1, θ2)
However, for S2x, the natural set of coordinates that gives a rotational invariant for angles is in fact given by:
S̃2x(u, θ1, α) = S2x(u, θ1, θ1 + α)
which naturally leads to:
S̃2(rθ.x)(u, θ1, α) = S̃2x(r−θu, θ1 − θ, α) , rθ.(S2x)(u, θ1, α)
In fact, this representation is covariant with the action of the roto-translation group, i.e. R2 n SO2 (Sifre & Mallat, 2013). In addition, it can be proven that a scattering network linearizes small deformations (Mallat, 2012), which means that a linear operator can build invariants to a subset of deformations. It means also that a deep network could reduce the perturbations due to this variability. Furthermore, this representation is complete, in the sense that it is possible to reconstruct a signal from its scattering coefficients (Bruna & Mallat, 2013a).
It is also important to note that a scattering transform is non-expansive, as a cascade of non-expansive operators, e.g.: ‖Sx− Sy‖ ≤ ‖x− y‖. Thus, this representation is stable to additive noises, which correspond to perturbations studied in Goodfellow et al. (2016); Moosavi-Dezfooli et al. (2015); Szegedy et al. (2013); Goodfellow et al. (2014). By adding a very small quantity to an image, the classification performed by a deep network can be fooled, e.g. image x is well classified, but one can find ‖ ‖ 1 such that x + is not, with the two images being visually indistinguishable.
Apprendix B mathematically quantifies the stability of a hybrid network and proves the unstability potentially rises from the cascaded learned deep network. Indeed, Szegedy et al. (2013) reports that the operators of the different layers of a deep network are not contractive. Corrections were added to fix this by training a deep network on the fooling examples (Goodfellow et al., 2014), but this requires additional computations. With wavelets as an initialization, instabilities cannot occur in the first layers contrary to Szegedy et al. (2013), since the operator is non-expansive.
2.2 ADDING SUPERVISION TO OUR REPRESENTATION: CASCADING A CONVNET
This section introduces our hybrid representation, and explain its interest. We first justify why applying a deep network after scattering is natural. Scattering transforms have yielded excellent numerical results on datasets where the variabilities are completely known, such as MNIST or FERET. In these task, we only encounter the problem of sample variance and handling the variance leads to solving the problem. However, in classification tasks on more complex image datasets, such variabilities are only partially known. Applying the scattering transform on datasets like CIFAR or Caltech leads to nearly state-of-the-art results in the unsupervised case (Oyallon & Mallat, 2015). But there is a huge gap in performance when comparing to supervised representations, which deep networks can fill in.
We now explain why the scattering transform is an appropriate initialization. Recent works (Mallat, 2016; Bruna et al., 2013) have suggested that a deep network could build an approximation of the group of symmetries of a classification task and apply transformations along the orbits of this group. The objective of a supervised classifier is to reduce the variabilities due to those symmetries. To each layer corresponds an approximated Lie group of symmetry, and this approximation is progressive, in the sense that the dimension of this approximation is increasing with depth. For instance, the linear Lie group of symmetry of an image is the translation group, R2. If no non-linearity is applied, it is not possible to discover new linear groups of symmetry for natural images. In the case of a wavelet transform obtained by rotation of a mother wavelet, it is possible to recover a new subgroup of symmetry, the rotation SO2, and the group of symmetry at this layer is the roto-translation group: R2 n SO2. Discovering the next groups of symmetry is however a difficult task; nonetheless, the roto-translation group seems to be a good initialization for the first layers. In this work, we investigate this hypothesis.
We thus build a standard deep convolutional network on top of the scattering transform. Its architecture is represented in Figure 2.2. Our network consists of a cascade of 2C convolutions with spatial kernel size 3 × 3. The C first convolutions use K0 input channels, except for the first layer, while the C next convolutions have K1 output channels, except for the last layer. The number of inputs of the first layer is equal to the size of the scattering features, whereas the last layer consists in an average pooling followed by a linear projection. We used a ReLU non-linearity, and no non-linear pooling is involved in our architecture. Observe that if the scattering is applied up to a scale J, then the signal is at spatial resolution J , i.e. its sampling is 2J , allowing faster computation. In the next section, we discuss the value of those parameters.
Notably, the first layer F1 of this deep convolutional network is structured by its input, the scattering representation. The nature of this operator and the features selected by this supervised algorithm will be discussed in the next sections.
3 EXPERIMENTS
We compare our algorithm to supervised, unsupervised and semi-supervised deep networks and evaluate them on the CIFAR10, CIFAR100 and STL10 datasets. CIFAR10 and CIFAR100 datasets consist of colored images of size 32 × 32, with 50000 images in the training set, and 10000 in the test set. CIFAR10 and CIFAR100 have respectively 10 and 100 classes. SLT10 dataset consists of colored images of size 96 × 96, with only 5000 labeled images in the training set divided equally in 10 classes, and 8000 images in the test set. The unlabeled images of this dataset were not used during our experiments. The three datasets are whitened as preprocessing, following the standard procedure. Our software, ScatWave, is implemented in Torch and is publicly available online 2.
3.1 EXPERIMENTAL RESULTS
3.1.1 METHODOLOGY
During all our experiments, we have trained our architecture with SGD with momentum 0.9 to minimize the standard negative cross-entropy. The batch size is 128. We have applied four types of regularization. First, 0.6 dropout is used after each consecutive two layers. Secondly, we have used 10−4 weight decay. Futhermore, we have used batch normalization techniques which are supposed to lead to a better conditioning of the optimization (Ioffe & Szegedy, 2015). Finally, we have augmented the dataset by using random cropping and flipping. The initial learning rate is 0.25 and we divide it by two after every 30 epochs. The networks are trained during 300 epochs.
We now depict the selected hyperparameters of our architecture for CIFAR and STL10 datasets respectively, which we have kept fixed for all experiments unless specifically stated otherwise. For the CIFAR datasets, using cross-validation, the parameters of invariance are set to J = 2 and the number of angles used is L = 8 for the scattering transform. In this case, the output of the scattering network is a tensor of size N
2J × N 2J × 3(1 + LJ + 12L 2J(J − 1)) = 8× 8× 243, after reshaping. For the deep net architecture, we chose to use C = 10 layers and K0 ∈ {128, 512},K1 = 128 channels. The number of parameters for CIFAR10 is 9 × (243 × K0 + (C − 1)K20 + K0K1 + (C − 1)K21 ) + 10×K1), which is roughly equal to 1.6M and 12M parameters for K0 = 128 and K0 = 512 respectively. For the STL10 dataset, we chosed K0 = K1 = 512 and C = 2: the deep network is shallower to compensate the speed loss due to the use of larger images. In the following, the symbol % corresponds to an absolute percentage of accuracy.
3.1.2 NUMERICAL EXPERIMENTS ON THE ENTIRE DATASET
We report the classification accuracies in Table 3.1.2 and discuss them below. We compare our architecture with the unsupervised scattering architecture (Oyallon & Mallat, 2015). The roto-translation scattering is almost identical to scattering, except that it recombines the channels along the rotation axis by applying a wavelet transform along angles, building more complex geometrical invariants. The classifier of this paper is a RBF SVM kernel, which can be interpreted as a two-layer deep neural network. For the sake of simplicity, we have trained on top of our scattering network a 3 layer fully connected network with size 2048, similarly to (Perronnin & Larlus, 2015). Without data augmentation, the accuracy of the network is respectively 3.7% and 8.9% below the roto-translation scattering on CIFAR10 and CIFAR100. However, applying data augmentation with translation of length less than 22 permits recovering this loss in accuracy, resulting in accuracies of 83.0% and 56.7% (we let the network train for 400 epochs here) respectively on CIFAR10 and CIFAR100. One major difference to our approach is that the system in Oyallon & Mallat (2015) uses a large amount of oversampling. This suggests that in order to learn the appropriate features it is necessary to average the small translation displacement; even if the representation is built to be invariant to translation, its construction relies, for fast computation, on an approximative (but justified) downsampling that leads to non-linear aliasing.
Applying a supervised convnet significantly improves the accuracy of the scattering with 3 fully connected layers, and leads to comparable results with other supervised deep representations: 91.4% on CIFAR10 and 69.5% on CIFAR100. We compare our work with the Highway network (Srivastava et al., 2015), that consists in a deep cascade of 19 linear and non-linear operators, with an extensive
2https://github.com/edouardoyallon/scatwave
Table 1: Accuracy of scattering compared to similar architectures on CIFAR10
Architecture Accuracy Unsupervised architectures Roto-translation scattering 82.3 Scattering (ours) + 3 FC +no data augmentation 78.6 Scattering (ours) + 3 FC 83.0 Supervised hybrid architectures Scattering (ours) + CNN, K = 128 89.4 Scattering (ours) + CNN, K = 512 91.4 Supervised architectures Highway network 92.4 All-CNN 92.8 Wide ResNet 96.2
Table 2: Accuracy of scattering compared to similar architectures on CIFAR100
Architecture Accuracy Unsupervised architectures Roto-translation scattering 56.8 Scattering (ours) + 3 FC + no data augmentation 47.9 Scattering (ours) + 3 FC 56.7 Supervised hybrid architectures Scattering (ours) +CNN, K = 128 64.4 Scattering (ours) +CNN, K = 512 69.5 Supervised architectures Highway network 67.8 All-CNN 66.3 Wide ResNet 81.7
data augmentation. Besides, since our convnet is kept as simple as possible, we also compare our architecture to the All-CNN (Springenberg et al., 2014) work. The latter performs slightly better on CIFAR10, but our hybrid network has a better generalization on CIFAR100. This indicates that supervision is essential, but that a geometric initialization of the first layers of a deep network leads to a discriminative representation as well. It is also interesting to observe that we could not find any architecture that was performing worse on CIFAR10 than ours, but better on CIFAR100. Since the number of samples available per class is lower, this could indicate that learning is easier in this case with a scattering initialization.
A Wide Resnet (Zagoruyko & Komodakis, 2016) outperforms our architecture by 4.8% on CIFAR10 and 12.2% on CIFAR100, but it requires more engineering process to be designed and is deeper. It is important to recall that, contrary to the wide Resnet, there are no instabilities due to geometric transformations (e.g. translations or deformations), and that the two first layers which are responsible for a large fraction of the computation are not learned, resulting in computational savings. Obviously, the scattering layers do not suffer from the vanishing or exploding gradient issues.
3.1.3 NUMERICAL EXPERIMENTS ON A SMALL SUBSET OF THE DATA
Supervised deep networks trained on small datasets easily overfit, for instance in the case of medical imaging where little data are available. Semisupervised algorithms exhibit good performances (Salimans et al., 2016), but it requires a large amount of unlabeled data to work. In this subsection, we show the benefit of using scattering in a framework where those data are not available. We demonstrate that scattering does prevent overfitting on small datasets, while keeping the same architecture and training methodology: this saves time to design an architecture.
For this experiment, we draw several random subsets of CIFAR10 for training our network, and used the same splits for each experiments to train a supervised deep network. Namely, we used a Network in Network (NiN) (Lin et al., 2013), VGG-like Simonyan & Zisserman (2014), Wide ResNet Zagoruyko & Komodakis (2016), which perform better than our network on the full dataset, and we use an implementation available online3. The VGG did not converge in this situation with the given hyperparameters (such as the depth) and for a simple and fair comparison we decided not to adapt them. We however applied a data augmentation to the inputs of the NiN, VGG and ResNet by translation and flipping. Table 3 corresponds to the averaged accuracy over 5 different subsets, with the corresponding standard deviation. We compare the two architectures with a semisupervised model that consists in a GAN (Salimans et al., 2016), that is trained on all the data of CIFAR10 yet only a fraction is labeled. With 4000 and 8000 labeled samples, a wide ResNet with 40 layers outperforms by at least 3% the supervised and unsupervised methods which indicate this architecture suffer from less overfitting than the others. If less than 2000 samples are available, a hybrid network outperforms all the supervised architectures: the difference of accuracy between the hybrid architecture and the others is progressively favorable to the hybrid network when the number
3http://torch.ch/blog/2015/07/30/cifar.html
of samples decreases. Nonetheless, a GAN performs better than a translation scattering with 3 fully connected layers; its performances is almost constant equal to 80%, which shows that this algorithm can adapt itself to the bias of the dataset.
In a second experiment, we apply our hybrid architecture on the STL10 dataset, which is a challenging dataset in the limited sample situation since only 500 samples per class are available. The averaged result is reported in Table 4. A supervised CNN (Swersky et al., 2013) whose hyper parameters have been automatically tuned achieves 70.1% accuracy. Using the unlabeled data improves by at least 5% the accuracy of a CNN. For an Exemplar CNN (Dosovitskiy et al., 2014) or an Unsupervised Discriminative CNN (Huang et al., 2016), the weights of the CNNs are unsupervisedly learned from patches of images. Those techniques add several hyper parameters and require an additional engineering process. Applying a hybrid network is straightforward and outpasses both the supervised and unsupervised state of the art by 7.3% and 0.6% respectively.
3.2 RETRAINING MODULES OF THE ARCHITECTURE
In this section, we show the benefit of using a structured representation. For instance, the first layer of a convnet cascaded on top of a scattering network inherits from the structure of the scattering coefficents. In all the following experiments, we use a converged hybrid network according to the procedure detailed in Subsection 3.1.1.
3.2.1 SIMPLIFYING THE THIRD LAYER OF A DEEPNETWORK
We numerically analyze the nature of the operations performed along angles by the first layer F1 of our deep network on CIFAR10. Let us define as F1x = {F 01 x, F 11 x, F 21 x} the components associated to the order 0,1,2 scattering coefficients respectively. Let 1 ≤ k ≤ K. In this case, F 01 is a convolutional operator that depends on the variables (u, k), F 1 1 depends on (u, θ1, k), and F 21 depends on (u, θ1, α, k). We would like to characterize the smoothness of these operators with respect to the variables, because our representation is covariant to rotations.
To this end, we consider the Fourier transform, for each channels k. We define by F̂ 11 , F̂ 2 1 the Fourier transform of these operators along the variables θ1 and (θ1, α) respectively. In space, we applied a DCT transform restrained to its support which is similar to a Fourier transform, since the support
of the operator is small. Then the operator is expressed in the tensorial Frequency domain. Since those operations are involutive (up to constants and reflexions), one can easily recover the original operator.
Let us demonstrate how to sparsify this operator in its frequency basis. First, it is possible to threshold by the coefficients of the operators in the Fourier domain, i.e. we replaced the operators F̂ 11 , F̂ 21 by 1|F̂ 11 |> F̂ 1 1 and 1|F̂ 21 |> F̂ 2 1 . Before thresholding, all the frequencies were excited. After this operation, approximatively 10% of coefficients are non-zero. We have tested our network without any retraining and observed a negligible loss of accuracy. We proved that this basis permits a sparse approximation of our filters.
Furthermore, we decided to keep only the two first frequencies of F1 along the variable θ1, i.e. we replaced the operators F̂ 11 , F̂ 2 1 by 1|ωθ1 |≤1F̂ 1 1 , 1|ωθ1 |≤1F̂ 2 1 . This results in a loss of accuracy of roughly 5%. We then fix the layer F1, and decide to retrain the next layers, keeping the entire training procedure identical again with an initial learning rate of 1. We obtain a classification accuracy which is 3% below the original architecture. This indicates that the network can almost recover discriminative information from averaged coefficients in angle.
The first experiment shows that in a natural basis it is possible to sparsify the operator. The last experiment indicates that most of the operations performed by the first layer of our network are smooth since they are localized in the Fourier space. This is equivalent to first projecting via an angular averaging each scattering coefficient and it suggests that the system autonomously builds an invariant to geometrical variabilities. While this does not prove that nomore geometrical operators are applied in the next layers, but it does show it is possible to obtain a good accuracy (3% below the original result) with a representation after F1 which is stable to local roto-translation and deformation variabilities, thanks to a roto-translation averaging.
3.2.2 REPLACING THE SCATTERING NETWORK BY A CONVNET
Many theoretical arguments of deep learning rely on the universal approximation theorem (Cybenko, 1989). The flexibility of this deep learning frameworks raises the following question: can we approximate the first scattering layers by a deep network?
In order to explore this question, we consider a 5-layer convnet as a candidate to replace our scattering network on CIFAR10. Its architecture is described on Figure 3.2.2, and it has the same output size as a scattering network. It has two downsampling steps, in order to mimic the behavior of a scattering network. We keep our architecture identical, except that we replace the scattering part by this network. Then we retrain it, keeping the weights of all the other layers constant and equal to the optimal solution found with the scattering in the previous section. Instead of minimizing a loss between the output of a scattering network and this network, we target the best input for the fixed convnet given the classification task.
This architecture can achieve 1% accuracy below the original pipeline, which is convincing. Using a shallower network seems to degrade the performances, but we did not investigate more this question. Besides, the learned network will not have any guarantee of stability properties.
4 CONCLUSION
We proposed a new deep hybrid architecture that involves scattering and convnets and is competitive with existing approaches. We demonstrate its good generalization performances on CIFAR10, CIFAR100, STL10 and subsets of CIFAR10, CIFAR100. We showed that the cascaded convnet learns an invariant to roto-translation, and that it is possible to learn a deep network that mimics the scattering, at the cost to potentially create instabilities. We release also a fast software to compute a scattering transform on GPUs.
This is a preliminary work whose results must be extended to ImageNet. This paper was dedicated to incorporate geometry into deep networks, and we will show how they refine their construction of class invariants in a future work.
ACKNOWLEDGMENTS
The author would like to thank Mathieu Andreux, Eugene Belilovsky, Carmine Cella, Bogdan Cirstea, Michael Eickenberg, Stéphane Mallat and Sergey Zagoruyko, for helpful discussions and support. Also, the author especially thanks Carmine Cella for nice suggestions of experiments. This work is funded by the ERC grant InvariantClass 320959 and via a grant for PhD Students of the Conseil régional d’Ile-de-France (RDM-IdF).
APPENDIX A: IMPLEMENTATION DETAILS OF SCATWAVE
Cascade of multi-resolution computations, such as perfomed in a scattering transform, are delicate. The bottleneck of the scattering on CPU was either speed and memory. It is thus necessary to quickly explain our algorithm: we show that by reorganizing the order of the computation of the algorithm, one can speed them up.
Computing a scattering transform at order 2 requires computing each path ||x ? ψp1 | ? ψp2 | where p1, p2 are the parameters with increasing scales of the filters. Computing each path can be viewed as a computational tree, where the coefficient of the scattering transform before an averaging are the leaf of the tree, and the internal nodes are the modulus of the intermediary wavelet transform. The way the tree is walked affects the computation time. In ScatNet (Andén et al., 2014), the traversal is done by first computing each internal node, storing the results of each internal node. In a second steps, the leafs are computed and stored. In terms of memory, this is not optimal since it requires storing the intermediate computations. Instead, we use an affix traversal of the tree. It reduces at its minimal the memory used, and allow the use of GPUs.
ScatWave is a GPU version of the scattering networks in Torch, that is based on our observation above. ScatNetLight is a MATLAB version on CPUs, which uses as much as possible multithreads. The Table 5 reports the difference in computation time, for identical parameters and output representations (e.g. same sampling, same hyper parameters). The input corresponds to batches of 128 tensor, the two first dimensions being the size of the image, and the third the number of elements in the tensor (e.g. the color in the case of images) . For comparisons, we used a machine with 24 cores and a TiTan GPU. The speed-up is at least of ×15 in all cases, and up to ×70: ScatWave uses the library cuFFT.
APPENDIX B: A NOTE ON THE STABILITY OF A HYBRID DEEP NETWORK
In this section, we recall the notion of additive stability of a deep network and derive some simple properties that shows that instabilities are due to the cascaded deepnetwork, and we demonstrate bounds to quantify them. Instabilities are due to perturbations of an initial sample that a deepnetwork does not reduce correctly: the modification is visually not significant or does not affect the label of the perturbated sample, yet the network incorrectly classifies the image. We consider a (trained) deep network f , with bounded outputs, that are for instance the probabilities output obtained after a sigmoid function. We write label(x) the labels computed by this deepnetwork for an input x. It is possible to define the contraction factor of a deep network f via:
∆(f) = sup label(x) 6=label(y) ‖f(x)− f(y)‖ ‖x− y‖
Observe that label(x) 6= label(y) ⇒ x 6= y. A small value of ∆(f) indicates a better stability. It is possible to introduce a local version of this definition, that depends on the input sample (Moosavi-Dezfooli et al., 2015). It is consistant with the definitions of Goodfellow et al. (2016); Moosavi-Dezfooli et al. (2015); Szegedy et al. (2013); Goodfellow et al. (2014), in the sens that the numerator is bounded, but the denominator might become arbitrary small. Let us now consider a hybrid network, e.g. for an input x, the output is f(Sx) where Sx ∈ Rn is its scattering transform. Let us write S = {Sx, x} the span of a scattering transform, which corresponds to a strict embedding in the standard euclidean space Rn, e.g. S ( Rn. Proposition 1. The following bounds stand:
∆(f ◦ S) ≤ sup label(x̃) 6=label(ỹ)
(x̃,ỹ)∈S2
‖f(x̃)− f(ỹ)‖ ‖x̃− ỹ‖ ≤ ∆(f)
Proof. Let x, y two samples with different estimated labels, then Sx 6= Sy. In this case,
‖f(Sx)− f(Sy)‖ ‖x− y‖ = ‖f(Sx)− f(Sy)‖ ‖Sx− Sy‖ . ‖Sx− Sy‖ ‖x− y‖
Thanks to the non-expansivity property of a Scattering transform,
‖Sx− Sy‖ ‖x− y‖ ≤ 1
Setting x̃ = Sx, ỹ = Sy and taking the supremum ends the demonstration.
One sees that the amplitude of the resulting instabilities depends on the class of f . Furthermore, the inequalities might be strict and the bounds tighter, but there is no reason it does occur. However, it suggests that such instabilities could be removed if additional constraints were added during the training phase of f .
The deformation transformations are as well a class of instabilities. The cited works (Goodfellow et al., 2016; Moosavi-Dezfooli et al., 2015; Szegedy et al., 2013; Goodfellow et al., 2014) do not consider them, however, since the scattering transform linearizes them, one sees it is possible for a deepnetwork to explicitly build such invariance. Actually, the average along rotation and translation variables observed in Subsection 3.2.1 seems to indicate it is likely to occur.

The paper investigates a hybrid network consisting of a scattering network followed by a convolutional network. By using scattering layers, the number of parameters is reduced, and the first layers are guaranteed to be stable to deformations. Experiments show that the hybrid network achieves reasonable performance, and outperforms the network-in-network architecture in the small-data regime.
I have often heard researchers ask why it is necessary to re-learn low level features of convolutional networks every time they are trained. In theory, using fixed features could save parameters and training time. As far as I am aware, this paper is the first to investigate this question. In my view, the results show that using scattering features in the bottom layers does not work as well as learned CNN features. This is not completely obvious a priori, and so the results are interesting, but I disagree with the framing that the hybrid network is superior in terms of generalization.
For the low-data regime, the hybrid network sometimes gives better accuracy than NiN, but this is quite an old architecture and its capacity has not been tuned to the dataset size. For the full dataset, the hybrid network is clearly outperformed by fully learned models. If I understood correctly, the authors have not simply compared identical architectures with and without scattering as the first layers, which further complicates drawing a conclusion.
The authors claim the hybrid network has the theoretical advantage of stability. However, only the first layers of a hybrid network will be stable, while the learned ones can still create instability. Furthermore, if potentially unstable deep networks outperform stable scattering nets and partially-stable hybrid nets, we have to question the importance of stability as defined in the theory of scattering networks.
In conclusion, I think the paper investigates a relevant question, but I am not convinced that the hybrid network really generalizes better than standard deep nets. Faster computation (at test time) could be useful e.g. in low power and mobile devices, but this aspect is not really fleshed out in the paper.
Minor comments:
-section 3.1.2: “learni”


We consider the problem of producing compact architectures for text classification, such that the full model fits in a limited amount of memory. After considering different solutions inspired by the hashing literature, we propose a method built upon product quantization to store word embeddings. While the original technique leads to a loss in accuracy, we adapt this method to circumvent quantization artefacts. Combined with simple approaches specifically adapted to text classification, our approach derived from fastText requires, at test time, only a fraction of the memory compared to the original FastText, without noticeably sacrificing quality in terms of classification accuracy. Our experiments carried out on several benchmarks show that our approach typically requires two orders of magnitude less memory than fastText while being only slightly inferior with respect to accuracy. As a result, it outperforms the state of the art by a good margin in terms of the compromise between memory usage and accuracy.
1 INTRODUCTION
Text classification is an important problem in Natural Language Processing (NLP). Real world usecases include spam filtering or e-mail categorization. It is a core component in more complex systems such as search and ranking. Recently, deep learning techniques based on neural networks have achieved state of the art results in various NLP applications. One of the main successes of deep learning is due to the effectiveness of recurrent networks for language modeling and their application to speech recognition and machine translation (Mikolov, 2012). However, in other cases including several text classification problems, it has been shown that deep networks do not convincingly beat the prior state of the art techniques (Wang & Manning, 2012; Joulin et al., 2016).
In spite of being (typically) orders of magnitude slower to train than traditional techniques based on n-grams, neural networks are often regarded as a promising alternative due to compact model sizes, in particular for character based models. This is important for applications that need to run on systems with limited memory such as smartphones.
This paper specifically addresses the compromise between classification accuracy and the model size. We extend our previous work implemented in the fastText library1. It is based on n-gram features, dimensionality reduction, and a fast approximation of the softmax classifier (Joulin et al., 2016). We show that a few key ingredients, namely feature pruning, quantization, hashing, and retraining, allow us to produce text classification models with tiny size, often less than 100kB when trained on several popular datasets, without noticeably sacrificing accuracy or speed.
We plan to publish the code and scripts required to reproduce our results as an extension of the fastText library, thereby providing strong reproducible baselines for text classifiers that optimize the compromise between the model size and accuracy. We hope that this will help the engineering community to improve existing applications by using more efficient models.
This paper is organized as follows. Section 2 introduces related work, Section 3 describes our text classification model and explains how we drastically reduce the model size. Section 4 shows the effectiveness of our approach in experiments on multiple text classification benchmarks.
1https://github.com/facebookresearch/fastText
2 RELATED WORK
Models for text classification. Text classification is a problem that has its roots in many applications such as web search, information retrieval and document classification (Deerwester et al., 1990; Pang & Lee, 2008). Linear classifiers often obtain state-of-the-art performance while being scalable (Agarwal et al., 2014; Joachims, 1998; Joulin et al., 2016; McCallum & Nigam, 1998). They are particularly interesting when associated with the right features (Wang & Manning, 2012). They usually require storing embeddings for words and n-grams, which makes them memory inefficient.
Compression of language models. Our work is related to compression of statistical language models. Classical approaches include feature pruning based on entropy (Stolcke, 2000) and quantization. Pruning aims to keep only the most important n-grams in the model, leaving out those with probability lower than a specified threshold. Further, the individual n-grams can be compressed by quantizing the probability value, and by storing the n-gram itself more efficiently than as a sequence of characters. Various strategies have been developed, for example using tree structures or hash functions, and are discussed in (Talbot & Brants, 2008).
Compression for similarity estimation and search. There is a large body of literature on how to compress a set of vectors into compact codes, such that the comparison of two codes approximates a target similarity in the original space. The typical use-case of these methods considers an indexed dataset of compressed vectors, and a query for which we want to find the nearest neighbors in the indexed set. One of the most popular is Locality-sensitive hashing (LSH) by Charikar (2002), which is a binarization technique based on random projections that approximates the cosine similarity between two vectors through a monotonous function of the Hamming distance between the two corresponding binary codes. In our paper, LSH refers to this binarization strategy2. Many subsequent works have improved this initial binarization technique, such as spectal hashing (Weiss et al., 2009), or Iterative Quantization (ITQ) (Gong & Lazebnik, 2011), which learns a rotation matrix minimizing the quantization loss of the binarization. We refer the reader to two recent surveys by Wang et al. (2014) and Wang et al. (2015) for an overview of the binary hashing literature.
Beyond these binarization strategies, more general quantization techniques derived from Jegou et al. (2011) offer better trade-offs between memory and the approximation of a distance estimator. The Product Quantization (PQ) method approximates the distances by calculating, in the compressed domain, the distance between their quantized approximations. This method is statistically guaranteed to preserve the Euclidean distance between the vectors within an error bound directly related to the quantization error. The original PQ has been concurrently improved by Ge et al. (2013) and Norouzi & Fleet (2013), who learn an orthogonal transform minimizing the overall quantization loss. In our paper, we will consider the Optimized Product Quantization (OPQ) variant (Ge et al., 2013).
Softmax approximation The aforementioned works approximate either the Euclidean distance or the cosine similarity (both being equivalent in the case of unit-norm vectors). However, in the context of fastText, we are specifically interested in approximating the maximum inner product involved in a softmax layer. Several approaches derived from LSH have been recently proposed to achieve this goal, such as Asymmetric LSH by Shrivastava & Li (2014), subsequently discussed by Neyshabur & Srebro (2015). In our work, since we are not constrained to purely binary codes, we resort a more traditional encoding by employing a magnitude/direction parametrization of our vectors. Therefore we only need to encode/compress an unitary d-dimensional vector, which fits the aforementioned LSH and PQ methods well.
Neural network compression models. Recently, several research efforts have been conducted to compress the parameters of architectures involved in computer vision, namely for state-of-theart Convolutional Neural Networks (CNNs) (Han et al., 2016; Lin et al., 2015). Some use vector quantization (Gong et al., 2014) while others binarize the network (Courbariaux et al., 2016). Denil et al. (2013) show that such classification models are easily compressed because they are overparametrized, which concurs with early observations by LeCun et al. (1990).
2In the literature, LSH refers to multiple distinct strategies related to the Johnson-Lindenstrauss lemma. For instance, LSH sometimes refers to a partitioning technique with random projections allowing for sublinear search via cell probes, see for instance the E2LSH variant of Datar et al. (2004).
Some of these works both aim at reducing the model size and the speed. In our case, since the fastText classifier on which our proposal is built upon is already very efficient, we are primilarly interested in reducing the size of the model while keeping a comparable classification efficiency.
3 PROPOSED APPROACH
3.1 TEXT CLASSIFICATION
In the context of text classification, linear classifiers (Joulin et al., 2016) remain competitive with more sophisticated, deeper models, and are much faster to train. On top of standard tricks commonly used in linear text classification (Agarwal et al., 2014; Wang & Manning, 2012; Weinberger et al., 2009), Joulin et al. (2016) use a low rank constraint to reduce the computation burden while sharing information between different classes. This is especially useful in the case of a large output space, where rare classes may have only a few training examples. In this paper, we focus on a similar model, that is, which minimizes the softmax loss ` over N documents:
N∑ n=1 `(yn, BAxn), (1)
where xn is a bag of one-hot vectors and yn the label of the n-th document. In the case of a large vocabulary and a large output space, the matrices A and B are big and can require gigabytes of memory. Below, we describe how we reduce this memory usage.
3.2 BOTTOM-UP PRODUCT QUANTIZATION
Product quantization is a popular method for compressed-domain approximate nearest neighbor search (Jegou et al., 2011). As a compression technique, it approximates a real-valued vector by finding the closest vector in a pre-defined structured set of centroids, referred to as a codebook. This codebook is not enumerated, since it is extremely large. Instead it is implicitly defined by its structure: a d-dimensional vector x ∈ Rd is approximated as
x̂ = k∑ i=1 qi(x), (2)
where the different subquantizers qi : x 7→ qi(x) are complementary in the sense that their respective centroids lie in distinct orthogonal subspaces, i.e., ∀i 6= j, ∀x, y, 〈qi(x)|qj(y)〉 = 0. In the original PQ, the subspaces are aligned with the natural axis, while OPQ learns a rotation, which amounts to alleviating this constraint and to not depend on the original coordinate system. Another way to see this is to consider that PQ splits a given vector x into k subvectors xi, i = 1 . . . k, each of dimension d/k: x = [x1 . . . xi . . . xk], and quantizes each sub-vector using a distinct k-means quantizer. Each subvector xi is thus mapped to the closest centroid amongst 2b centroids, where b is the number of bits required to store the quantization index of the subquantizer, typically b = 8. The reconstructed vector can take 2kb distinct reproduction values, and is stored in kb bits.
PQ estimates the inner product in the compressed domain as
x>y ≈ x̂>y = k∑
i=1
qi(x i)>yi. (3)
This is a straightforward extension of the square L2 distance estimation of Jegou et al. (2011). In practice, the vector estimate x̂ is trivially reconstructed from the codes, i.e., from the quantization indexes, by concatenating these centroids.
The two parameters involved in PQ, namely the number of subquantizers k and the number of bits b per quantization index, are typically set to k ∈ [2, d/2], and b = 8 to ensure byte-alignment.
Discussion. PQ offers several interesting properties in our context of text classification. Firstly, the training is very fast because the subquantizers have a small number of centroids, i.e., 256 centroids for b = 8. Secondly, at test time it allows the reconstruction of the vectors with almost no
computational and memory overhead. Thirdly, it has been successfully applied in computer vision, offering much better performance than binary codes, which makes it a natural candidate to compress relatively shallow models. As observed by Sánchez & Perronnin (2011), using PQ just before the last layer incurs a very limited loss in accuracy when combined with a support vector machine.
In the context of text classification, the norms of the vectors are widely spread, typically with a ratio of 1000 between the max and the min. Therefore kmeans performs poorly because it optimizes an absolute error objective, so it maps all low-norm vectors to 0. A simple solution is to separate the norm and the angle of the vectors and to quantize them separately. This allows a quantization with no loss of performance, yet requires an extra b bits per vector.
Bottom-up strategy: re-training. The first works aiming at compressing CNN models like the one proposed by (Gong et al., 2014) used the reconstruction from off-the-shelf PQ, i.e., without any re-training. However, as observed in Sablayrolles et al. (2016), when using quantization methods like PQ, it is better to re-train the layers occurring after the quantization, so that the network can re-adjust itself to the quantization. There is a strong argument arguing for this re-training strategy: the square magnitude of vectors is reduced, on average, by the average quantization error for any quantizer satisfying the Lloyd conditions; see Jegou et al. (2011) for details.
This suggests a bottom-up learning strategy where we first quantize the input matrix, then retrain and quantize the output matrix (the input matrix being frozen). Experiments in section 4 show that it is worth adopting this strategy.
Memory savings with PQ. In practice, the bottom-up PQ strategy offers a compression factor of 10 without any noticeable loss of performance. Without re-training, we notice a drop in accuracy between 0.1% and 0.5%, depending on the dataset and setting; see Section 4 and the appendix.
3.3 FURTHER TEXT SPECIFIC TRICKS
The memory usage strongly depends on the size of the vocabulary, which can be large in many text classification tasks. While it is clear that a large part of the vocabulary is useless or redundant, directly reducing the vocabulary to the most frequent words is not satisfactory: most of the frequent words, like “the” or “is” are not discriminative, in contrast to some rare words, e.g., in the context of tag prediction. In this section, we discuss a few heuristics to reduce the space taken by the dictionary. They lead to major memory reduction, in extreme cases by a factor 100. We experimentally show that this drastic reduction is complementary with the PQ compression method, meaning that the combination of both strategies reduces the model size by a factor up to ×1000 for some datasets.
Pruning the vocabulary. Discovering which word or n-gram must be kept to preserve the overall performance is a feature selection problem. While many approaches have been proposed to select groups of variables during training (Bach et al., 2012; Meier et al., 2008), we are interested in selecting a fixed subset of K words and ngrams from a pre-trained model. This can be achieved by selecting the K embeddings that preserve as much of the model as possible, which can be reduced to selecting the K words and ngrams associated with the highest norms.
While this approach offers major memory savings, it has one drawback occurring in some particular cases: some documents may not contained any of the K best features, leading to a significant drop in performance. It is thus important to keep the K best features under the condition that they cover the whole training set. More formally, the problem is to find a subset S in the feature set V that maximizes the sum of their norms ws under the constraint that all the documents in the training set D are covered:
max S⊆V ∑ s∈S ws s.t. |S| ≤ K, P1S ≥ 1D,
where P is a matrix such that Pds = 1 if the s-th feature is in the d-th document, and 0 otherwise. This problem is directly related to set covering problems that are NP-hard (Feige, 1998). Standard greedy approaches require the storing of an inverted index or to do multiple passes over the dataset, which is prohibitive on very large dataset (Chierichetti et al., 2010). This problem can be cast as an instance of online submodular maximization with a rank constraint (Badanidiyuru et al., 2014;
Bateni et al., 2010). In our case, we use a simple online parallelizable greedy approach: For each document, we verify if it is already covered by a retained feature and, if not, we add the feature with the highest norm to our set of retained features. If the number of features is below k, we add the features with the highest norm that have not yet been picked.
Hashing trick & Bloom filter. On small models, the dictionary can take a significant portion of the memory. Instead of saving it, we extend the hashing trick used in Joulin et al. (2016) to both words and n-grams. This strategy is also used in Vowpal Wabbit (Agarwal et al., 2014) in the context of online training. This allows us to save around 1-2Mb with almost no overhead at test time (just the cost of computing the hashing function).
Pruning the vocabulary while using the hashing trick requires keeping a list of the indices of the K remaining buckets. At test time, a binary search over the list of indices is required. It has a complexity of O(log(K)) and a memory overhead of a few hundreds of kilobytes. Using Bloom filters instead reduces the complexityO(1) at test time and saves a few hundred kilobytes. However, in practice, it degrades performance.
4 EXPERIMENTS
This section evaluates the quality of our model compression pipeline and compare it to other compression methods on different text classification problems, and to other compact text classifiers.
Evaluation protocol and datasets. Our experimental pipeline is as follows: we train a model using fastText with the default setting unless specified otherwise. That is 2M buckets, a learning rate of 0.1 and 10 training epochs. The dimensionality d of the embeddings is set to powers of 2 to avoid border effects that could make the interpretation of the results more difficult. As baselines, we use Locality-Sensitive Hashing (LSH) (Charikar, 2002), PQ (Jegou et al., 2011) and OPQ (Ge et al., 2013) (the non-parametric variant). Note that we use an improved version of LSH where random orthogonal matrices are used instead of random matrix projection Jégou et al. (2008). In a first series of experiments, we use the 8 datasets and evaluation protocol of Zhang et al. (2015). These datasets contain few million documents and have at most 10 classes. We also explore the limit of quantization on a dataset with an extremely large output space, that is a tag dataset extracted from the YFCC100M collection (Thomee et al., 2016)3, referred to as FlickrTag in the rest of this paper.
4.1 SMALL DATASETS
Compression techniques. We compare three popular methods used for similarity estimation with compact codes: LSH, PQ and OPQ on the datasets released by Zhang et al. (2015). Figure 1 shows the accuracy as a function of the number of bytes used per embedding, which corresponds to the number k of subvectors in the case of PQ and OPQ. See more results in the appendix. As discussed in Section 2, LSH reproduces the cosine similarity and is therefore not adapted to un-normalized data. Therefore we only report results with normalization. Once normalized, PQ and OPQ are almost lossless even when using only k = 4 subquantizers per embedding (equivalently, bytes). We observe in practice that using k = d/2, i.e., half of the components of the embeddings, works well in practice. In the rest of the paper and if not stated otherwise, we focus on this setting. The difference between the normalized versions of PQ and OPQ is limited and depends on the dataset. Therefore we adopt the normalized PQ (NPQ) for the rest of this study, since it is faster to train.
3Data available at https://research.facebook.com/research/fasttext/
Pruning. Figure 2 shows the performance of our model with different sizes. We fix k = d/2 and use different pruning thresholds. NPQ offers a compression rate of×10 compared to the full model. As the pruning becomes more agressive, the overall compression can increase up up to ×1, 000 with little drop of performance and no additional overhead at test time. In fact, using a smaller dictionary makes the model faster at test time. We also compare with character-level Convolutional Neural Networks (CNN) (Zhang et al., 2015; Xiao & Cho, 2016). They are attractive models for text classification because they achieve similar performance with less memory usage than linear models (Xiao & Cho, 2016). Even though fastText with the default setting uses more memory, NPQ is already on par with CNNs’ memory usage. Note that CNNs are not quantized, and it would be worth seeing how much they can be quantized with no drop of performance. Such a study is beyond the scope of this paper. Our pruning is based on the norm of the embeddings according to the guidelines of Section 3.3. Table 1 compares the ranking obtained with norms to the ranking obtained using entropy, which is commonly used in unsupervised settings Stolcke (2000).
Extreme compression. Finally, in Table 2, we explore the limit of quantized model by looking at the performance obtained for models under 64KiB. Surprisingly, even at 64KiB and 32KiB, the drop of performance is only around 0.8% and 1.7% despite a compression rate of ×1, 000− 4, 000.
4.2 LARGE DATASET: FLICKRTAG
In this section, we explore the limit of compression algorithms on very large datasets. Similar to Joulin et al. (2016), we consider a hashtag prediction dataset containing 312, 116 labels. We set the minimum count for words at 10, leading to a dictionary of 1, 427, 667 words. We take 10M buckets for n-grams and a hierarchical softmax. We refer to this dataset as FlickrTag.
Output encoding. We are interested in understanding how the performance degrades if the classifier is also quantized (i.e., the matrix B in Eq. 1) and when the pruning is at the limit of the minimum number of features required to cover the full dataset.
Table 3 shows that quantizing both the “input” matrix (i.e., A in Eq. 1) and the “output” matrix (i.e., B) does not degrade the performance compared to the full model. We use embeddings with d = 256 dimensions and use k = d/2 subquantizers. We do not use any text specific tricks, which leads to a compression factor of 8. Note that even if the output matrix is not retrained over the embeddings, the performance is only 0.2% away from the full model. As shown in the Appendix, using less subquantizers significantly decreases the performance for a small memory gain.
Pruning. Table 4 shows how the performance evolves with pruning. We measure this effect on top of a fully quantized model. The full model misses 11.6% of the test set because of missing words (some documents are either only composed of hashtags or have only rare words). There are 312, 116 labels and thus it seems reasonable to keep embeddings in the order of the million. A naive pruning with 1M features misses about 30−40% of the test set, leading to a significant drop of performance. On the other hand, even though the max-coverage pruning approach was set on the train set, it does not suffer from any coverage loss on the test set. This leads to a smaller drop of performance. If the pruning is too aggressive, however, the coverage decreases significantly.
5 FUTURE WORK
It may be possible to obtain further reduction of the model size in the future. One idea is to condition the size of the vectors (both for the input features and the labels) based on their frequency (Chen et al., 2015; Grave et al., 2016). For example, it is probably not worth representing the rare labels by full 256-dimensional vectors in the case of the FlickrTag dataset. Thus, conditioning the vector size on the frequency and norm seems like an interesting direction to explore in the future.
We may also consider combining the entropy and norm pruning criteria: instead of keeping the features in the model based just on the frequency or the norm, we can use both to keep a good set of features. This could help to keep features that are both frequent and discriminative, and thereby to reduce the coverage problem that we have observed.
Additionally, instead of pruning out the less useful features, we can decompose them into smaller units (Mikolov et al., 2012). For example, this can be achieved by splitting every non-discriminative word into a sequence of character trigrams. This could help in cases where training and test examples are very short (for example just a single word).
6 CONCLUSION
In this paper, we have presented several simple techniques to reduce, by several orders of magnitude, the memory complexity of certain text classifiers without sacrificing accuracy nor speed. This is achieved by applying discriminative pruning which aims to keep only important features in the trained model, and by performing quantization of the weight matrices and hashing of the dictionary.
We will publish the code as an extension of the fastText library. We hope that our work will serve as a baseline to the research community, where there is an increasing interest for comparing the performance of various deep learning text classifiers for a given number of parameters. Overall, compared to recent work based on convolutional neural networks, fastText.zip is often more accurate, while requiring several orders of magnitude less time to train on common CPUs, and incurring a fraction of the memory complexity.

This paper describes how to approximate the FastText approach such that its memory footprint is reduced by several orders of magnitude, while preserving its classification accuracy. The original FastText approach was based on a linear classifier on top of bag-of-words embeddings. This type of method is extremely fast to train and test, but the model size can be quite large.
This paper focuses on approximating the original approach with lossy compression techniques. Namely, the embeddings and classifier matrices A and B are compressed with Product Quantization, and an aggressive dictionary pruning is carried out. Experiments on various datasets (either with small or large number of classes) are conducted to tune the parameters and demonstrate the effectiveness of the approach. With a negligible loss in classification accuracy, an important reduction in term of model size (memory footprint) can be achieved, in the order of 100~1000 folds compared to the original size.
The paper is well written overall. The goal is clearly defined and well carried out, as well as the experiments. Different options for compressing the model data are evaluated and compared (e.g. PQ vs LSH), which is also interesting. Nevertheless the paper does not propose by itself any novel idea for text classification. It just focuses on adapting existing lossy compression techniques, which is not necessarily a problem. Specifically, it introduces:
  - a straightforward variant of PQ for unnormalized vectors,
  - dictionary pruning is cast as a set covering problem (which is NP-hard), but a greedy approach is shown to yield excellent results nonetheless,
  - hashing tricks and bloom filter are simply borrowed from previous papers.
These techniques are quite generic and could as well be used in other works.
Here are some minor problems with the paper:
  - it is not made clear how the full model size is computed. What is exactly in the model? Which proportion of the full size do the A and B matrices, the dictionary, and the rest, account for? It is hard to follow where is the size bottleneck, which also seems to depend on the target application (i.e. small or large number of test classes). It would have been nice to provide a formula to calculate the total model size as a function of all parameters (k,b for PQ and K for dictionary, number of classes).

  - some parts lack clarity. For instance, the greedy approach to prune the dictionary is exposed in less than 4 lines (top of page 5), though it is far from being straightforward. Likewise, it is not clear why the binary search used for the hashing trick would introduce an overhead of a few hundreds of KB.

Overall this looks like a solid work, but with potentially limited impact research-wise.