Encoder-decoder models have been widely used to solve sequence to sequence prediction tasks. However current approaches suffer from two shortcomings. First, the encoders compute a representation of each word taking into account only the history of the words it has read so far, yielding suboptimal representations. Second, current models utilize large vocabularies in order to minimize the problem of unknown words, resulting in slow decoding times and large storage costs. In this paper we address both shortcomings. Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word. Furthermore, we propose a simple copy mechanism that is able to exploit very small vocabularies and handle out-of-vocabulary words. We demonstrate the effectiveness of our approach on the Gigaword dataset and DUC competition outperforming the state-of-the-art.
1 INTRODUCTION
Encoder-decoder models have been widely used in sequence to sequence tasks such as machine translation (Cho et al. (2014); Sutskever et al. (2014)). They consist of an encoder which represents the whole input sequence with a single feature vector. The decoder then takes this representation and generates the desired output sequence. The most successful models are LSTM and GRU as they are much easier to train than vanilla RNNs.
In this paper we are interested in summarization where the input sequence is a sentence/paragraph and the output is a summary of the text. Several encoding-decoding approaches have been proposed (Rush et al. (2015); Hu et al. (2015); Chopra et al. (2016)). Despite their success, it is commonly believed that the intermediate feature vectors are limited as they are created by only looking at previous words. This is particularly detrimental when dealing with large input sequences. Bi-directorial RNNs (Schuster & Paliwal (1997); Bahdanau et al. (2014)) try to address this problem by computing two different representations resulting of reading the input sequence left-to-right and right-to-left. The final vectors are computed by concatenating the two representations. However, the word representations are computed with limited scope.
The decoder employed in all these methods outputs at each time step a distribution over a fixed vocabulary. In practice, this introduces problems with rare words (e.g., proper nouns) which are out of vocabulary. To alleviate this problem, one could potentially increase the size of the decoder vocabulary, but decoding becomes computationally much harder, as one has to compute the soft-max over all possible words. Gulcehre et al. (2016), Nallapati et al. (2016) and Gu et al. (2016) proposed to use a copy mechanism that dynamically copy the words from the input sequence while decoding. However, they lack the ability to extract proper embeddings of out-of-vocabulary words from the input context. Bahdanau et al. (2014) proposed to use an attention mechanism to emphasize specific parts of the input sentence when generating each word. However the encoder problem still remains in this approach.
In this work, we propose two simple mechanisms to deal with both encoder and decoder problems. We borrowed intuition from human readers which read the text multiple times before generating summaries. We thus propose a ‘Read-Again’ model that first reads the input sequence before committing to a representation of each word. The first read representation then biases the second read
representation and thus allows the intermediate hidden vectors to capture the meaning appropriate for the input text. We show that this idea can be applied to both LSTM and GRU models. Our second contribution is a copy mechanism which allows us to use much smaller vocabulary sizes resulting in much faster decoding and much smaller storage space. Our copy mechanism also allows us to construct a better representation of out-of-vocabulary words. We demonstrate the effectiveness of our approach in the challenging Gigaword dataset and DUC competition showing state-of-the-art performance.
2 RELATED WORK
2.1 SUMMARIZATION
In the past few years, there has been a lot of work on extractive summarization, where a summary is created by composing words or sentences from the source text. Notable examples are Neto et al. (2002), Erkan & Radev (2004), Wong et al. (2008), Filippova & Altun (2013) and Colmenares et al. (2015). As a consequence of their extractive nature the summary is restricted to words (sentences) in the source text.
Abstractive summarization, on the contrary, aims at generating consistent summaries based on understanding the input text. Although there has been much less work on abstractive methods, they can in principle produce much richer summaries. Abstractive summarization is standardized by the DUC2003 and DUC2004 competitions (Over et al. (2007)). Some of the prominent approaches on this task includes Banko et al. (2000), Zajic et al. (2004), Cohn & Lapata (2008) and Woodsend et al. (2010). Among them, the TOPIARY system (Zajic et al. (2004)) performs the best in the competitions amongst non neural net based methods.
Very recently, the success of deep neural networks in many natural language processing tasks (Collobert et al. (2011)) has inspired new work in abstractive summarization . Rush et al. (2015) propose a neural attention model with a convolutional encoder to solve this task. Hu et al. (2015) build a large dataset for Chinese text summarization and propose to feed all hidden states from the encoder into the decoder. More recently, Chopra et al. (2016) extended Rush et al. (2015)’s work with an RNN decoder, and Nallapati et al. (2016) proposed an RNN encoder-decoder architecture for summarization. Both techniques are currently the state-of-the-art on the DUC competition. However, the encoders exploited in these methods lack the ability to encode each word condition on the whole text, as an RNN encodes a word into a hidden vector by taking into account only the words up to that time step.
In contrast, in this work we propose a ‘Read-Again’ encoder-decoder architecture, which enables the encoder to understand each input word after reading the whole sentence. Our encoder first reads the text, and the results from the first read help represent the text in the second pass over the source text. Our second contribution is a simple copy mechanism that allows us to significantly reduce the decoder vocabulary size resulting in much faster inference times. Furthermore our copy mechanism allows us to handle out-of-vocabulary words in a principled manner. Finally our experiments show state-of-the-art performance on the DUC competition.
2.2 NEURAL MACHINE TRANSLATION
Our work is also closely related to recent work on neural machine translation, where neural encoderdecoder models have shown promising results (Kalchbrenner & Blunsom (2013); Cho et al. (2014); Sutskever et al. (2014)). Bahdanau et al. (2014) further developed an attention mechanism in the decoder in order to pay attention to a specific part of the input at every generating time-step. Our approach also exploits an attention mechanism during decoding.
2.3 OUT-OF-VOCABULARY AND COPY MECHANISM
Dealing with Out-Of-Vocabulary words (OOVs) is an important issue in sequence to sequence approaches as we cannot enumerate all possible words and learn their embeddings since they might not be part of our training set. Luong et al. (2014) address this issue by annotating words on the source, and aligning OOVs in the target with those source words. Recently, Vinyals et al. (2015)
propose Pointer Networks, which calculate a probability distribution over the input sequence instead of predicting a token from a pre-defined dictionary. Cheng & Lapata (2016) develop a neural-based extractive summarization model, which predicts the targets from the input sequences. Gulcehre et al. (2016); Nallapati et al. (2016) use explicit gating to decide adaptively wether to generate a target word from the fixed-size dictionary or from the input sequence. Gu et al. (2016) use a implicit implicit gating operation instead of the explicit gating. This is similar to our decoder. However, our decoder can also extract different OOVs’ embedding accordingly from the input text instead of using a single <UNK> embedding to represent all OOVs. This further enhances the model’s ability to handle OOVs.
3 THE READ AGAIN MODEL
Text summarization can be formulated as a sequence to sequence prediction task, where the input is a longer text and the output is a summary of that text. In this paper we develop an encoder-decoder approach to summarization. The encoder is used to represent the input text with a set of continuous vectors, and the decoder is used to generate a summary word by word.
In the following, we first introduce our ‘Read-Again’ model for encoding sentences. The idea behind our approach is very intuitive and is inspired by how humans do this task. When we create summaries, we first read the text and then we do a second read where we pay special attention to the words that are relevant to generate the summary. Our ‘Read-Again’ model implements this idea by reading the input text twice and using the information acquired from the first read to bias the second read. This idea can be seamlessly plugged into LSTM and GRU models. Our second contribution is a copy mechanism used in the decoder. It allows us to reduce the decoder vocabulary size dramatically and can be used to extract a better embedding for OOVs. Fig. 1(a) gives an overview of our model.
3.1 ENCODER
We first review the typical encoder used in machine translation (e.g., Sutskever et al. (2014); Bahdanau et al. (2014)). Let x = {x1, x2, · · · , xn} be the input sequence of words. An encoder sequentially reads each word and creates the hidden representation hi by exploting a recurrent neural network (RNN)
hi = RNN(xi, hi−1), (1)
where xi is the word embedding of xi. The hidden vectors h = {h1, h2, · · · , hn} are then treated as the feature representations for the whole input sentence and can be used by another RNN to decode and generate a target sentence. Although RNNs have been shown to be useful in modeling sequences, one of the major drawback is that hi depends only on past information i.e., {x1, · · · , xi}. However, it is hard (even for humans) to have a proper representation of a word without reading the whole input sentence.
Following this intuition, we propose our ‘Read-Again’ model where the encoder reads the input sentence twice. In particular, the first read is used to bias the second more attentive read. We apply
this idea to two popular RNN architectures, i.e. GRU and LSTM, resulting in better encodings of the input text. Note that although other alternatives, such as bidirectional RNN exist, the hidden states from the forward RNN lack direct interactions with the backward RNN, and thus forward/backward hidden states still cannot utilize the whole sequence. Besides, although we only use our model in a uni-directional manner, it can also be easily adapted to the bidirectional case. We now describe the two variants of our model.
3.1.1 GRU READ-AGAIN
We read the input sentence {x1, x2, · · · , xn} for the first-time using a standard GRU
h1i = GRU 1(xi, h 1 i−1), (2)
where the function GRU1 is defined as,
zi = σ(Wz[xi, h 1 i−1]) (3)
ri = σ(Wr[xi, h 1 i−1])
h̃1i = tanh(Wh[xi, ri h1i−1]) h1i = (1− zi) h1i−1 + zi h̃1i
It consists of two gatings zi, ri, controlling whether the current hidden state h1i should be directly copied from h1i−1 or should pass through a more complex path h̃ 1 i .
Given the sentence feature vector h1n, we then compute an importance weight vector αi of each word for the second reading. We put the importance weight αi on the skip-connections as shown in Fig. 2(a) to bias the two information flows: If the current word xi has a very small weight αi, then the second read hidden state h2i will mostly take the information directly from the previous state h2i−1, ignoring the influence of the current word. If αi is close to 1 then it will be similar to a standard GRU, which is only influenced from the current word. Thus the second reading has the following update rule
h2i = (1− αi) h2i−1 + αi GRU2(xi, h2i−1), (4)
where means element-wise product. We compute the importance weights by attending h1i with h1n as follows
αi = tanh(Weh 1 i + Ueh 1 n + Vexi), (5)
where We, Ue, Ve are learnable parameters. Note that αi is a vector representing the importance of each dimension in the word embedding. Empirically, we find that using a vector is better than a scalar gating. We hypothesize that this is because different dimensions represent different semantic meanings, and a scalar gating mechanism lacks the ability to capture the variances among these dimensions.
Combining this with the standard GRU update rule
GRU2(xi, h2i−1) = (1− zi) h2i−1 + zi h̃2i , we can simplify the updating rule Eq. (4) to get
h2i = (1− αi zi) h2i−1 + (αi zi) h̃2i (6)
This equations shows that our ‘read-again’ model on GRU is equivalent to replace the GRU cell with a more general gating mechanism that also depends on the feature representation of the whole sentence computed from the first reading pass. We argue that adding this global information could help direct the information flow for the forward pass resulting in a better encoder.
3.1.2 LSTM READ-AGAIN
We now apply the ‘Read-Again’ idea to the LSTM architecture as shown in Fig. 2(b). Our first reading is performed by an LSTM1 defined as
fi = σ(Wf [xi, hi−1]) (7) ii = σ(Wi[xi, hi−1])
oi = σ(Wo[xi, hi−1])
C̃i = tanh(WC [xi, hi−1])
Ci = ft Ci−1 + ii C̃i hi = oi tanh(Ci)
Different from the GRU architecture, LSTM calculates the hidden state by applying a non-linear activation function to the cell state Ci, instead of a linear combination of two paths used in the GRU. Thus for our second read, instead of using skip-connections, we make the gating functions explicitly depend on the whole sentence vector computed from the first reading pass. We argue that this helps the encoding of the second reading LSTM2, as all gating and updating increments are also conditioned on the whole sequence feature vector (h1i , h 1 n). Thus
h2i = LSTM 2([xi, h 1 i , h 1 n], h 2 i−1), (8)
3.1.3 READING MULTIPLE SENTENCES
In this section we extend our ‘Read-Again’ model to the case where the input sequence has more than one sentence. Towards this goal, we propose to use a hierarchical representation, where each sentence has its own feature vector from the first reading pass. We then combine them into a single vector to bias the second reading pass. We illustrate this in the context of two input sentences, but it is easy to generalize to more sentences. Let {x1, x2, · · · , xn} and {x′1, · · · , x′m} be the two
input sentences. The first RNN reads these two sentences independently to get two sentence feature vectors h1n and h ′1 m respectively.
Here we investigate two different ways to handle multiple sentences. Our first option is to simply concatenate the two feature vectors to bias our second reading pass:
h2i = RNN 2([xi, h 1 i , h 1 n, h ′1 m], h 2 i−1) (9)
h′2i = RNN 2([x′i, h ′1 i , h 1 n, h ′1 m], h ′2 i−1)
where h20 and h ′2 0 are initialized as zero vectors. Feeding h 1 n, h ′1 m into the second RNN provides more global information explicitly and helps acquire long term dependencies.
The second option we explored is shown in Fig. 3. In particular, we use a non-linear transformation to get a single feature vector hglobal from both sentence feature vectors:
hglobal = tanh(Wrh 1 n + Urh ′1 m + vr) (10)
The second reading pass is then
h̃2i = RNN 2([xi, h 1 i , h 1 n, hglobal], h 2 i−1) (11)
h̃′2i = RNN 2([x′i, h ′1 i , h ′1 m, hglobal], h ′2 i−1)
Note that this is more easily scalable to more sentences. In our experiments both approaches perform similarly.
3.2 DECODER WITH COPY MECHANISM
In this paper we argue that only a small number of common words are needed for generating a summary in addition to the words that are present in the source text. We can consider this as a hybrid approach which combines extractive and abstractive summarization. This has two benefits: first it allow us to use a very small vocabulary size, speeding up inference. Furthermore, we can create summaries which contain OOVs if they are present in the source text.
Our decoder reads the vector representations of the input text using an attention mechanism, and generates the target summary word by word. We use an LSTM as our decoder, with a fixed-size vocabulary dictionary Y and learnable word embeddings Y ∈ R|Y |×dim. At time-step t the LSTM generates a summary word yt by first computing the current hidden state st from the previous hidden state st−1, previous summary word yt−1 and current context vector ct
st = LSTM([yt−1, ct], st−1), (12) where the context vector ct is computed with an attention mechanism on the encoder hidden states:
ct = n∑ i=1 βith 2 i . (13)
The attention score βit at time-step t on the i-th word is computed via a soft-max over oit, where oit = att(st−1, h 2 i ) = v T a tanh(Wast−1 + Uah 2 i ), (14) with va, Wa, Ua learnable parameters.
A typical way to treat OOVs is to encode them with a single shared embedding. However, different OOVs can have very different meanings, and thus using a single embedding for all OOVs will confuse the model. This is particularly detrimental when using small vocabulary sizes. Here we address this issue by deriving the representations of OOVs from their corresponding context in the input text. Towards this goal, we change the update rule of yt−1. In particular, if yt−1 belongs to a word that is in our decoder vocabulary we take its representation from the word embedding, otherwise if it appears in the input sentence as xi we use
yt−1 = pi = tanh(Wch 2 i + bc) (15)
where Wc and bc are learnable parameters. Since h2i encodes useful context information of the source word xi, pi can be interpreted as the semantics of this word extracted from the input sentence. Furthermore, if yt−1 does not appear in the input text, nor in Y , then we represent yt−1 using the <UNK> embedding.
Given the current decoder’s hidden state st, we can generate the target summary word yt. As shown in Fig. 1(b), at each time step during decoding, the decoder outputs a distribution over generating words from Y , as well as over copying a specific word xi from the source sentence.
3.3 LEARNING
We jointly learn our encoder and decoder by maximizing the likelihood of decoding the correct word at each time step. We refer the reader to the experimental evaluation for more details.
4 EXPERIMENTAL EVALALUATION
In this section, we show results of abstractive summarization on Gigaword (Graff & Cieri (2003); Napoles et al. (2012)) and DUC2004 (Over et al. (2007)) datasets. Our model can learn a meaningful re-reading weight distribution for each word in the input text, putting more emphasis on important verb and nous, while ignoring common words such as prepositions. As for the decoder, we demonstrate that our copy mechanism can successfully reduce the typical vocabulary size by a factor 5 while achieving much better performance than the state-of-the-art, and by a factor of 30 while maintaining the same level of performance. In addition, we provide an analysis and examples of which words are copied during decoding.
Dataset and Evaluation Metric: We use the Gigaword corpus to train and evaluate our models. Gigaword is a news corpus where the title is employed as a proxy for the summary of the article. We follow the same pre-processing steps of Rush et al. (2015), which include filtering, PTB tokenization, lower-casing, replacing digit characters with #, replacing low-frequency words with UNK and extracting the first sentence in each article. This results in a training set of 3.8M articles, a validation set and a test set each containing 400K articles. The average sentence length is 31.3 words for the source, and 8.3 words for the summaries. Following the standard protocol we evaluate ROUGE score on 2000 random samples from the test set. As for evaluation metric, we use full-length F1 score on Rouge-1, Rouge-2 and Rouge-L, following Chopra et al. (2016) and Nallapati et al. (2016), since these metrics are less bias to the outputs’ length than full-length recall scores.
Implemetation Details: We implement our model in Tensorflow and conduct all experiments on a NVIDIA Titan X GPU. Our models converged after 2-3 days of training, depending on model size. Our RNN cells in all models have 1 layer, 512-dimensional hidden states, and 512-dimensional word embeddings. We use dropout rate of 0.2 in all activation layers. All parameters, except the biases are initialized uniformly with a range of √ 3/d, where d is the dimension of the hidden state (Sussillo & Abbott (2014)). The biases are initialized to 0.1. We use plain SGD to train the model with gradient clipped at 10. We start with an initial learning rate of 2, and halve it every epoch after first 5 epochs. Our max epoch for training is 10. We use a mini-batch size of 64, which is shuffled during training.
4.1 QUANTITATIVE EVALUATION
Results on Gigaword: We compare the performances of different architectures and report ROUGE scores in Table 1. Our baselines include the ABS model of Rush et al. (2015) with its proposed
vocabulary size as well as an attention encoder-decoder model with uni-directional GRU encoder. We allow the decoder to generate variable length summaries. As shown in Table 1 our Read-Again models outperform the baselines on all ROUGE scores, when using both 15K and 69K sized vocabularies. We also observe that adding the copy mechanism further helps to improve performance: Even though the decoder vocabulary size of our approach with copy (15K) is much smaller than ABS (69K) and GRU (69K), it achieves a higher ROUGE score. Besides, our Multiple-Sentences model achieves the best performance.
Evaluation on DUC2004: DUC 2004 (Over et al. (2007)) is a commonly used benchmark on summarization task consisting of 500 news articles. Each article is paired with 4 different humangenerated reference summaries, capped at 75 characters. This dataset is evaluation-only. Similar to Rush et al. (2015), we train our neural model on the Gigaword training set, and show the models’ performances on DUC2004. Following the convention, we also use ROUGE limited-length recall as our evaluation metric, and set the capping length to 75 characters. We generate summaries with 15 words using beam-size of 10. As shown in Table 2, our method outperforms all previous methods on Rouge-1 and Rouge-L, and is comparable on Rouge-2. Furthermore, our model only uses 15k decoder vocabulary, while previous methods use 69k or 200k.
Importance Weight Visualization: As we described in the section before, αi is a high-dimension vector representing the importance of each word xi. While the importance of a word is different over each dimension, by averaging we can still look at general trends of which word is more relevant. Fig. 4 depicts sample sentences with the importance weight αi over input words. Words such as the, a, ’s, have small αi, while words such as aeronautics, resettled, impediments, which carry more information have higher values. This shows that our read-again technique indeed extracts useful information from the first reading to help bias the second reading results.
the sh utt le sp ac e
do wn tou ch ed co lum bia at su nd ay ken ne dythe ce nte r ca na ve ral ca pein sp ac e flo rid a,
ae ron
au tic s na tio na l the sp ac e, ad mi nis tra tio n an d sa id . <e os >
indonesia has moved #.# million people and resettled them in #,### villages in a national transmigration scheme over the
past ## years , presi ent suharto said here monday .
ind on
es ia #.#ha s an d
mi llio
n
mo ve
d
res ett
led
pe op
le
vill ag
es the m in
tra ns
mi gra
tio n
na tio
na la #.# ##
sc he
me pa st ov er the he re##
pre sid
en t .
<e os
> ye ars ,
mo nd
ay
su ha
rto sa idin
indo esia 's state- wned domestic carrier merp ti nusantara airlines will lease ## aircraft from the united states , the official
antara news agency reported here thursday .
ind on
es ia sta teow ne d‘s me rpa ti ca rrie r do me sti c air lin es nu sa nta ra lea sewil l air cra ft un ite dthefro m## the sta tes
thu rsd
ay
offi cia
l
he re
ag en
cy, ne ws an tar a
rep ort
ed . <e os > tar iffs ba rrie rsan d
im pe
dim en
ts rem ainoth er to se rio us an d tra de in reg ion
as ia-
pa cifi
c
the
bu sin
es s “
de sp
ite
pro gre
ss
su bs
tan tia
l the las t” rep ortov er se ve n . <e os >
tariffs and other barriers remain serious impediments to trade and business in the asia-pacific region despite `` substantial ''
progress over the last seven years , a leading economic council said in a report .
ye ars , a lea din g
ec on
om ic
co un
cilsa id in a
Figure 4: Weight Visualization. Black indicates high weight
4.2 EVALUATION OF COPY MECHANISM
Table 3 shows the effect on our model of decreasing the decoder vocabulary size. We can see that when using the copy mechanism, we are able to reduce the decoder vocabulary size from 69K to 2K, with only 2-3 points drop on ROUGE score. This contrasts the models that do not use the copy mechanism. Equipped with a copy mechanism, our model is able to generate OOVs as summary
words, and thus maintains its expressive ability even with a small decoder vocabulary size. We also observe from Table 4 that the copy mechanism help us to decrease the encoder vocabulary size as well. The model without copy suffers from severe OOV problem when encoder size is small, since a single shared <UNK> embedding cannot depict many different OOVs. This makes it difficult for the encoder to understand the input text. Meanwhile, our copy model can extract an OOV’s meaning accordingly from its context in the input text, and thus it is sufficient to learn and store only the high-frequency words embeddings using our model, which in turn save the storage. We also notice that shrinking the encoder vocabulary to 15k achieves better result. One possible reason is that long tail words can not learn efficient embeddings during training, and representing them with extracted embedding from our model performs better.
Table 5 shows the decoding time as a function of vocabulary size. As computing the soft-max is usually the bottleneck for decoding, reducing vocabulary size dramatically reduces the decoding time from 0.38 second per sentence to 0.08 second.
Table 6 provides some examples of visualization of the copy mechanism. Note that we are able to copy key words from source sentences to improve the summary. From these examples we can see that our model is able to copy different types of rare words, such as special entities’ names in case 1 and 2, rare nouns in case 3 and 4, adjectives in case 5 and 6, and even rare verbs in the last example. Note that in the third example, when the copy model’s decoder uses the embedding of headmaster as its first input, which is extracted from the source sentence, it generates the same following sentence as the no-copy model. This probably means that the extracted embedding of headmaster is closely related to the learned embedding of teacher.
5 CONCLUSION
In this paper we have proposed two simple mechanisms to alleviate the problems of current encoderdecoder models. Our first contribution is a ‘Read-Again’ model which does not form a representation of the input word until the whole sentence is read. Our second contribution is a copy mechanism that can handle out-of-vocabulary words in a principled manner allowing us to reduce the decoder vocabulary size and significantly speed up inference. We have demonstrated the effectiveness of our approach in the context of summarization and shown state-of-the-art performance. In the future, we plan to tackle summarization problems with large input text. We also plan to exploit our findings in other tasks such as machine translation.

Summary: This paper proposes a read-again attention-based representation of the document with the copy mechanism for the summarization task. The model reads each sentence in the input document twice and creates a hierarchical representation of it instead of a bidirectional RNN. During the decoding, it uses the representation of the document obtained via the read-again mechanism and points the words that are OOV in the source document. The model does abstractive summarization. The authors show improvements on DUC 2004 dataset and provide an analysis of their model with different configurations.
Contributions:
The main contribution of this paper is the read-again attention mechanism where the model reads the same sentence twice and obtains a better representation of the document.
Writing:
The text of this paper needs more work. There are several typos and the explanations of the model/architecture are not really clear, some parts of the paper feel somewhat bloated.
Pros:
- The proposed model is a simple extension to the model to the model proposed in [2] for summarization.
- The results are better than the baselines.
Cons:
- The improvements are not that large.
- Justifications are not strong enough.
- The paper needs a better writeup. Several parts of the text are not using a clear/precise language and the paper needs a better reorganization. Some parts of the text is somewhat informal.
- The paper is very application oriented.
Question:
- How does the training speed when compared to the regular LSTM?
Some Criticisms:
A similar approach to the read again mechanism which is proposed in this paper has already been explored in [1] in the context of algorithmic learning and I wouldn’t consider the application of that on the summarization task a significant contribution.  The justification behind the read-again mechanism proposed in this paper is very weak. It is not really clear why additional gating alpha_i is needed for the read again stage.
As authors also suggest, pointer mechanism for the unknown/rare words [2] and it is adopted for the read-again attention mechanism. However, in the paper, it is not clear where the real is the gain coming from, whether from “read-again” mechanism or the use of “pointing”.
The paper is very application focused, the contributions of the paper in terms of ML point of view is very weak.
It is possible to try this read-again mechanism on more tasks other than summarization, such as NMT, in order to see whether if those improvements are
The writing of this paper needs more work. In general, it is not very well-written.
Minor comments:
Some of the corrections that I would recommend fixing,
On page 4: “… better than a single value … ” —> “… scalar gating …”
On page 4: “… single value lacks the ability to model the variances among these dimensions.” —> “… scalar gating couldn’t capture the ….”
On page 6: “ … where h_0^2 and h_0^'2 are initial zero vectors … “ —> “… h_0^2 and h_0^'2 are initialized to a zero vector in the beginning of each sequence …"
There are some inconsistencies for example parts of the paper refer to Tab. 1 and some parts of the paper refer to Table 2.
Better naming of the models in Table 1 is needed.
The location of Table 1 is a bit off.
[1] Zaremba, Wojciech, and Ilya Sutskever. "Reinforcement learning neural Turing machines." arXiv preprint arXiv:1505.00521 362 (2015).
[2] Gulcehre, Caglar, et al. "Pointing the Unknown Words." arXiv preprint arXiv:1603.08148 (2016).


DOMAIN-INVARIANT REPRESENTATION LEARNING
The learning of domain-invariant representations in the context of domain adaptation with neural networks is considered. We propose a new regularization method that minimizes the domain-specific latent feature representations directly in the hidden activation space. Although some standard distribution matching approaches exist that can be interpreted as the matching of weighted sums of moments, e.g. Maximum Mean Discrepancy, an explicit order-wise matching of higher order moments has not been considered before. We propose to match the higher order central moments of probability distributions by means of order-wise moment differences. Our model does not require computationally expensive distance and kernel matrix computations. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, called Central Moment Discrepancy (CMD). We prove that CMD is a metric on the set of probability distributions on a compact interval. We further prove that convergence of probability distributions on compact intervals w. r. t. the new metric implies convergence in distribution of the respective random variables. We test our approach on two different benchmark data sets for object recognition (Office) and sentiment analysis of product reviews (Amazon reviews). CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with Maximum Mean Discrepancy, Variational Fair Autoencoders and Domain Adversarial Neural Networks on Amazon reviews. In addition, a post-hoc parameter sensitivity analysis shows that the new approach is stable w. r. t. parameter changes in a certain interval. The source code of the experiments is publicly available1.
1 INTRODUCTION
The collection and preprocessing of large amounts of data for new domains is often time consuming and expensive. This in turn limits the application of state-of-the-art methods like deep neural network architectures, that require large amounts of data. However, often data from related domains can be used to improve the prediction model in the new domain. This paper addresses the particularly important and challenging domain-invariant representation learning task of unsupervised domain adaptation (Glorot et al., 2011; Li et al., 2014; Pan et al., 2011; Ganin et al., 2016). In unsupervised domain adaptation, the training data consists of labeled data from the source domain(s) and unlabeled data from the target domain. In practice, this setting is quite common, as in many applications
∗http://www.flll.jku.at †http://www.scch.at 1https://github.com/wzell/cmd
the collection of input data is cheap, but the collection of labels is expensive. Typical examples include image analysis tasks and sentiment analysis, where labels have to be collected manually.
Recent research shows that domain adaptation approaches work particularly well with (deep) neural networks, which produce outstanding results on some domain adaptation data sets (Ganin et al., 2016; Sun & Saenko, 2016; Li et al., 2016; Aljundi et al., 2015; Long et al., 2015; Li et al., 2015; Zhuang et al., 2015; Louizos et al., 2016). The most successful methods have in common that they encourage similarity between the latent network representations w. r. t. the different domains. This similarity is often enforced by minimizing a certain distance between the networks’ domainspecific hidden activations. Three outstanding approaches for the choice of the distance function are the Proxy A-distance (Ben-David et al., 2010), the Kullback-Leibler (KL) divergence Kullback & Leibler (1951), applied to the mean of the activations (Zhuang et al., 2015), and the Maximum Mean Discrepancy (Gretton et al., 2006, MMD).
Two of them, the MMD and the KL-divergence approach, can be viewed as the matching of statistical moments. The KL-divergence approach is based on mean (first raw moment) matching. Using the Taylor expansion of the Gaussian kernel, most MMD-based approaches can be viewed as minimizing a certain distance between weighted sums of all raw moments (Li et al., 2015).
The interpretation of the KL-divergence approaches and MMD-based approaches as moment matching procedures motivate us to match the higher order moments of the domain-specific activation distributions directly in the hidden activation space. The matching of the higher order moments is performed explicitly for each moment order and each hidden coordinate. Compared to KL-divergencebased approaches, which only match the first moment, our approach also matches higher order moments. In comparison to MMD-based approaches, our method explicitly matches the moments for each order, and it does not require any computationally expensive distance- and kernel matrix computations.
The proposed distribution matching method induces a metric between probability distributions. This is possible since distributions on compact intervals have an equivalent representation by means of their moment sequences. We utilize central moments due to their translation invariance and natural geometric interpretation. We call the new metric Central Moment Discrepancy (CMD).
The contributions of this paper are as follows:
• We propose to match the domain-specific hidden representations by explicitly minimizing differences of higher order central moments for each moment order. We utilize the equivalent representation of probability distributions by moment sequences to define a new distance function, which we call Central Moment Discrepancy (CMD).
• Probability theoretic analysis is used to prove that CMD is a metric on the set of probability distributions on a compact interval.
• We additionally prove that convergence of probability distributions on compact intervals w. r. t. to the new metric implies convergence in distribution of the respective random variables. This means that minimizing the CMD metric between probability distributions leads to convergence of the cumulative distribution functions of the random variables.
• In contrast to MMD-based approaches our method does not require computationally expensive kernel matrix computations.
• We achieve a new state-of-the-art performance on most domain adaptation tasks of Office and outperform networks trained with MMD, variational fair autoencoders and domain adversarial neural networks on Amazon reviews.
• A parameter sensitivity analysis shows that CMD is insensitive to parameter changes within a certain interval. Consequently, no additional hyper-parameter search has to be performed.
2 HIDDEN ACTIVATION MATCHING
We consider the unsupervised domain adaptation setting (Glorot et al., 2011; Li et al., 2014; Pan et al., 2011; Ganin et al., 2016) with an input space X and a label space Y . Two distributions over X × Y are given: the labeled source domain DS and the unlabeled target domain DT . Two corresponding samples are given: the source sample S = (XS , YS) = {(xi, yi)}ni=1 i.i.d.∼ (DS)n and
the target sample T = XT = {xi}mi=1 i.i.d.∼ (DT )m. The goal of the unsupervised domain adaptation setting is to build a classifier f : X → Y with a low target risk RT (f) = Pr (x,y)∼DT (f(x) 6= y), while no information about the labels in DT is given.
We focus our studies on neural network classifiers fθ : X → Y with parameters θ ∈ Θ, the input space X = RI with input dimension I , and the label space Y = [0, 1]|C| with the cardinality |C| of the set of classes C. We further assume a network output fθ(x) ∈ [0, 1]|C| of an example x ∈ RI to be normalized by the softmax-function σ : R|C| → [0, 1]|C| with σ(z)j = e
zj∑|C| k=1 e zk for
z = {z1, . . . , z|C|}. We focus on bounded activation functions gH : R→ [a, b]N for the hidden layer H with N hidden nodes, e.g. the hyperbolic tangent or the sigmoid function. Unbounded activation functions, e.g. rectified linear units or exponential linear units, can be used if the output is clipped or normalized to be bounded. Using the loss function l : Θ × X × Y → R, e.g. cross-entropy l(θ, x, y) = − ∑ i∈C yi log(fθ(x)i), and the sample set (X,Y ) ⊂ RI × [0, 1]|C|, we define the objective function as min θ∈Θ E(l(θ,X, Y )) (1)
where E denotes the empirical expectation, i.e. E(l(θ,X, Y )) = 1|(X,Y )| ∑
(x,y)∈(X,Y ) l(θ, x, y). Let us denote the source hidden activations by AH(θ,XS) = gH(θTHAH′(θ,XS)) ⊂ [a, b]N and the target hidden activations by AH(θ,XT ) = gH(θTHAH′(θ,XT )) ⊂ [a, b]N for the hidden layer H with N hidden nodes and parameter θH , and the hidden layer H ′ before H .
One fundamental assumption of most unsupervised domain adaptation networks is that the source risk RS(f) is a good indicator for the target risk RT (f), when the domain-specific latent space representations are similar (Ganin et al., 2016). This similarity can be enforced by matching the distributions of the hidden activations AH(θ,XS) and AH(θ,XT ) of higher layers H . Recent stateof-the-art approaches define a domain regularizer d : ([a, b]N )n × ([a, b]N )m → [0,∞), which gives a measure for the domain discrepancy in the activation space [a, b]N . The domain regularizer is added to the objective by means of an additional weighting parameter λ.
min θ∈Θ
E(l(θ,XS , YS)) + λ · d(AH(θ,XS), AH(θ,XT )) (2)
Fig. 1 shows a sketch of the described architecture and fig. 2 shows the hidden activations of a simple neural network optimized by eq. (1) (left) and eq. (2) (right). It can be seen that similar activation distributions are obtained when being optimized on the basis of the domain regularized objective.
3 RELATED WORK
Recently, several measures d for objective (2) have been proposed. One approach is the Proxy Adistance, given by d̂A = 2(1 − 2 ), where is the generalization error on the problem of discriminating between source and target samples (Ben-David et al., 2010). Ganin et al. (2016) compute the
value with a neural network classifier that is simultaneously trained with the original network by means of a gradient reversal layer. They call their approach domain-adversarial neural networks. Unfortunately, a new classifier has to be trained in this approach including the need of new parameters, additional computation times and validation procedures.
Another approach is to make use of the MMD (Gretton et al., 2006) as domain regularizer. MMD(X,Y )2 = E(K(X,X))− 2E(K(X,Y )) + E(K(Y, Y )) (3)
where E(K(X,Y )) = 1|X|·|Y | ∑ k∈K(X,Y ) k is the empirical expectation of the kernel products k between all examples in X and Y stored by the kernel matrix K(X,Y ). A suitable choice of the kernel seems to be the Gaussian kernel e−β‖x−y‖ 2
(Louizos et al., 2016; Li et al., 2015; Tzeng et al., 2014). This approach has two major drawbacks: (a) the need of tuning an additional kernel parameter β, and (b) the need of the kernel matrix computation K(X,Y ) (computational complexity O(n2 + nm+m2)), which becomes inefficient (resource-intensive) in case of large data sets. Concerning (a), the tuning of β is sophisticated since no target samples are available in the domain adaptation setting. Suitable tuning procedures are transfer learning specific cross-validation methods (Zhong et al., 2010). More general methods that don’t utilize source labels include heuristics that are based on kernel space properties (Sriperumbudur et al., 2009; Gretton et al., 2012), combinations of multiple kernels (Li et al., 2015), and kernel choices that maximize the MMD test power (Sutherland et al., 2016). The drawback (b) of the kernel matrix computation can be handled by approximating the MMD (Zhao & Meng, 2015), or by using linear time estimators (Gretton et al., 2012). In this work we focus on the quadratic-time MMD with the Gaussian kernel (Gretton et al., 2012; Tzeng et al., 2014) and transfer learning specific cross-validation for parameter tuning (Zhong et al., 2010; Ganin et al., 2016).
The two approaches MMD and the Proxy A-distance have in common that they do not minimize the domain discrepancy explicitly in the hidden activation space. In contrast, the authors in Zhuang et al. (2015) do so by minimizing a modified version of the Kullback-Leibler divergence of the mean activations (MKL). That is, for samples X,Y ⊂ RN ,
MKL(X,Y ) = N∑ i=1 E(X)i log E(X)i E(Y )i + E(Y )i log E(Y )i E(X)i
(4)
with E(X)i being the ith coordinate of the empirical expectation E(X) = 1|X| ∑ x∈X x. This approach is fast to compute and has an explicit interpretation in the activation space. Our empirical observations (section Experiments) show that minimizing the distance between only the first moment (mean) of the activation distributions can be improved by also minimizing the distance between higher order moments.
As noted in the introduction, our approach is motivated by the fact that the MMD and the KLdivergence approach can be seen as the matching of statistical moments of the hidden activations AH(θ,XS) and AH(θ,XT ). In particular, MMD-based approaches that use the Gaussian kernel are equivalent to minimizing a certain distance between weighted sums of all moments of the hidden activation distributions (Li et al., 2015).
We propose to minimize differences of higher order central moments of the activations AH(θ,XS) and AH(θ,XT ). The difference minimization is performed explicitly for each moment order. Our
approach utilizes the equivalent representation of probability distributions in terms of its moment series. We further utilize central moments due to their translation invariance and natural geometric interpretation. Our approach contrasts with other moment-based approaches, as they either match only the first moment (MKL) or they don’t explicitly match the moments for each order (MMD). As a result, our approach improves over MMD-based approaches in terms of computational complexity with O (N(n+m)) for CMD and O ( N(n2 + nm+m2) ) for MMD. In contrast to MKL-based approaches more accurate distribution matching characteristics are obtained. In addition, CMD achieves a new state-of-the-art performance on most domain adaptation tasks of Office and outperforms networks trained with MMD, variational fair autoencoders and domain adversarial neural networks on Amazon reviews.
4 CENTRAL MOMENT DISCREPANCY (CMD)
In this section we first propose a new distance function CMD on probability distributions on compact intervals. The definition is extended by two theorems that identify CMD as a metric and analyze a convergence property. The final domain regularizer is then defined as an empirical estimate of CMD. The proofs of the theorems are given in the appendix.
Definition 1 (CMD metric). Let X = (X1, . . . , Xn) and Y = (Y1, . . . , Yn) be bounded random vectors independent and identically distributed from two probability distributions p and q on the compact interval [a, b]N . The central moment discrepancy metric (CMD) is defined by
CMD(p, q) = 1
|b− a| ‖E(X)− E(Y )‖2 + ∞∑ k=2
1
|b− a|k ‖ck(X)− ck(Y )‖2 (5)
where E(X) is the expectation of X , and
ck(X) = ( E ( N∏ i=1 (Xi − E(Xi))ri ))
r1+...+rN=k r1,...,rn≥0
is the central moment vector of order k.
The first order central moments are zero, the second order central moments are related to variance, and the third and fourth order central moments are related to the skewness and the kurtosis of probability distributions. It is easy to see that CMD(p, q) ≥ 0, CMD(p, q) = CMD(q, p), CMD(p, q) ≤ CMD(p, r) + CMD(r, q) and p = q ⇒ CMD(p, q) = 0. The following theorem shows the remaining property for CMD to be a metric on the set of probability distributions on a compact interval.
Theorem 1. Let p and q be two probability distributions on a compact interval and let CMD be defined as in (5), then
CMD(p, q) = 0 ⇒ p = q
Our approach is to minimize the discrepancy between the domain-specific hidden activation distributions by minimizing the CMD. Thus, in the optimization procedure, we increasingly expect to see the domain-specific cumulative distribution functions approach each other. This characteristic can be expressed by the concept of convergence in distribution and it is shown in the following theorem.
Theorem 2. Let pn and p be probability distributions on a compact interval and let CMD be defined as in (5), then
CMD(pn, p)→ 0 ⇒ pn d−→ p
where d−→ denotes convergence in distribution.
We define the final central moment discrepancy regularizer as an empirical estimate of the CMD metric. Only the central moments that correspond to the marginal distributions are computed. The number of central moments is limited by a new parameter K and the expectation is sampled by the empirical expectation.
Definition 2 (CMD regularizer). Let X and Y be bounded random samples with respective probability distributions p and q on the interval [a, b]N . The central moment discrepancy regularizer CMDK is defined as an empirical estimate of the CMD metric, by
CMDK(X,Y ) = 1
|b− a| ‖E(X)−E(Y )‖2 + K∑ k=2
1
|b− a|k ‖Ck(X)− Ck(Y )‖2 (6)
where E(X) = 1|X| ∑ x∈X x is the empirical expectation vector computed on the sample X and Ck(X) = E((x−E(X))k) is the vector of all kth order sample central moments of the coordinates of X .
This definition includes three approximation steps: (a) the computation of only marginal central moments, (b) the bound on the order of central moment terms via parameter K, and (c) the sampling of the probability distributions by the replacement of the expected value with the empirical expectation.
Applying approximation (a) and assuming independent marginal distributions, a zero CMD distance value still implies equal joint distributions (thm. 1) but convergence in distribution (thm. 2) applies only to the marginals. In the case of dependent marginal distributions, zero CMD distance implies equal marginals and convergence in CMD implies convergence in distribution of the marginals. However, the matching properties for the joint distributions are not obtained with dependent marginals and approximation (a). The computational complexity is reduced to be linear w. r. t. the number of samples.
Concerning (b), proposition 1 shows that the marginal distribution specific CMD terms have an upper bound that is strictly decreasing with increasing moment order. This bound is convergent to zero. That is, higher CMD terms can contribute less to the overall distance value. This observation is experimentally strengthened in subsection Parameter Sensitivity. Proposition 1. Let X and Y be bounded random vectors with respective probability distributions p and q on the compact interval [a, b]N . Then
1
|b− a|k ‖ck(X)− ck(Y )‖2 ≤ 2
√ N
( 1
k + 1
( k
k + 1
)k + 1
21+k
) (7)
where ck(X) = E((X − E(X))k) is the vector of all kth order sample central moments of the marginal distributions of p.
Concerning approximation (c), the joint application of the weak law of large numbers (Billingsley, 2008) with the continuous mapping theorem (Billingsley, 2013) proves that this approximation creates a consistent estimate.
We would like to underline that the training of neural networks with eq. (2) and the CMD regularizer in eq. (6) can be easily realized by gradient descent algorithms. The gradients of the CMD regularizer are simple aggregations of derivatives of the standard functions gH , xk and ‖.‖2.
5 EXPERIMENTS
Our experimental evaluations are based on two benchmark datasets for domain adaptation, Amazon reviews and Office, described in subsection Datasets. The experimental setup is discussed in subsection Experimental Setup and our classification accuracy results are discussed in subsection Results. Subsection Parameter Sensitivity analysis the accuracy sensitivity w. r. t. parameter changes of K for CMD and β for MMD.
5.1 DATASETS
Amazon reviews: For our first experiment we use the Amazon reviews data set with the same preprocessing as used by Chen et al. (2012); Ganin et al. (2016); Louizos et al. (2016). The data set contains product reviews of four different product categories: books, DVDs, kitchen appliances and electronics. Reviews are encoded in 5000 dimensional feature vectors of bag-of-words unigrams and bigrams with binary labels: 0 if the product is ranked by 1 − 3 stars and 1 if the product is ranked
by 4 or 5 stars. From the four categories we obtain twelve domain adaptation tasks (each category serves once as source category and once as target category).
Office: The second experiment is based on the computer vision classification data set from Saenko et al. (2010) with images from three distinct domains: amazon (A), webcam (W) and dslr (D). This data set is a de facto standard for domain adaptation algorithms in computer vision. Amazon, the largest domain, is a composition of 2817 images and its corresponding 31 classes. Following previous works we assess the performance of our method across all six possible transfer tasks.
5.2 EXPERIMENTAL SETUP
Amazon Reviews:
For the Amazon reviews experiment, we use the same data splits as previous works for every task. Thus we have 2000 labeled source examples and 2000 unlabeled target examples for training, and between 3000 and 6000 examples for testing.
We use a similar architecture as Ganin et al. (2016) with one dense hidden layer with 50 hidden nodes, sigmoid activation functions and softmax output function. Three neural networks are trained by means of eq. (2): (a) a base model without domain regularization (λ = 0), (b) with the MMD as domain regularizer and (c) with CMD as domain regularizer. These models are additionally compared with the state-of-the-art models VFAE (Louizos et al., 2016) and DANN (Ganin et al., 2016). The models (a),(b) and (c) are trained with similar setup as in Louizos et al. (2016) and Ganin et al. (2016).
For the CMD regularizer, the λ parameter of eq. (2) is set to 1, i.e. the weighting parameter λ is neglected. The parameter K is heuristically set to five, as the first five moments capture rich geometric information about the shape of a distribution and K = 5 is small enough to be computationally efficient. However, the experiments in subsection Parameter Sensitivity show that similar results are obtained for K ≥ 3. For the MMD regularizer we use the Gaussian kernel with parameter β. We performed a hyperparameter search for β and λ, which has to be performed in an unsupervised way (no labels in the target domain). We use a variant of the reverse cross-validation approach proposed by Zhong et al. (2010), in which we initialize the model weights of the reverse classifier by the weights of the first learned classifier (see Ganin et al. (2016) for details). Thereby, the parameter λ is tuned on 10 values between 0.1 and 500 on a logarithmic scale. The parameter β is tuned on 10 values between 0.01 and 10 on a logarithmic scale. Without this parameter search, no competitive prediction accuracy results could be obtained.
Since we have to deal with sparse data, we rely on the Adagrad optimizer (Duchi et al., 2011). For all evaluations, the default parametrization is used as implemented in Keras (Chollet, 2015). All evaluations are repeated 10 times based on different shuffles of the data, and the mean accuracies and standard deviations are analyzed.
Office: Since the office dataset is rather small with only 2817 images in its largest domain, we use the latent representations of the convolution neural network VGG16 of Simonyan & Zisserman (2014). In particular we train a classifier with one hidden layer, 256 hidden nodes and sigmoid activation function on top of the output of the first dense layer in the network. We again train one base model without domain regularization and a CMD regularized version with K = 5 and λ = 1.
We follow the standard training protocol for this data set and use all available source and target examples during training. Using this ”fully-transductive” protocol, we compare our method with other state-of-the-art approaches including DLID (Chopra et al., 2013), DDC (Tzeng et al., 2014), DAN (Long et al., 2015), Deep CORAL (Sun & Saenko, 2016), and DANN (Ganin et al., 2016), based on fine-tuning of the baseline model AlexNet (Krizhevsky et al., 2012). We further compare our method to LSSA (Aljundi et al., 2015), CORAL (Sun et al., 2016), and AdaBN (Li et al., 2016), based on the fine-tuning of InceptionBN (Ioffe & Szegedy, 2015).
As an alternative to Adagrad for non-sparse data, we use the Adadelta optimizer from Zeiler (2012). Again, the default parametrization from Keras is used. We handle unbalances between source and target sample by randomly down-sampling (up-sampling) the source sample. In addition, we ensure a sub-sampled source batch that is balanced w. r. t. the class labels.
Since all hyper-parameters are set a-priori, no hyper-parameter search has to be performed.
All experiments are repeated 10 times with randomly shuffled data sets and random initializations.
5.3 RESULTS
Amazon Reviews: Table 1 shows the classification accuracies of four models: The Source Only model is the non domain regularized neural network trained with objective (1), and serves as a base model for the domain adaptation improvements. The models MMD and CMD are trained with the same architecture and objective (2) with d as the domain regularizer MMD and CMD, respectively. VFAE refers to the Variational Fair Autoencoder of Louizos et al. (2016), including a slightly modified version of the MMD regularizer for faster computations, and DANN refers to the domainadversarial neural networks model of Ganin et al. (2016). The last two columns are taken directly from these publications.
As one can observe in table 1, our accuracy of the CMD-based model is the highest in 9 out of 12 domain adaptation tasks, whereas on the remaining 3 it is the second best method. However, the difference in accuracy compared to the best method is smaller than the standard deviation over all data shuffles.
Office: Table 2 shows the classification accuracy of different models trained on the Office dataset. Note that some of the methods (LSSA, CORAL and AdaBN) are evaluated based on the InceptionBN model, which shows higher accuracy than the base model (VGG16) of our method in most tasks. However, our method outperforms related state-of-the-art methods on all except two tasks, on which it performs similar. We improve the previous state-of-the-art method AdaBN (Li et al., 2016) by more than 3.2% in average accuracy.
5.4 PARAMETER SENSITIVITY
The first sensitivity experiment aims at providing evidence regarding the accuracy sensitivity of the CMD regularizer w. r. t. parameter changes of K. That is, the contribution of higher terms in the CMD regularizer are analyzed. The claim is that the accuracy of CMD-based networks does not depend strongly on the choice of K in a range around its default value 5.
In fig. 3 on the upper left we analyze the classification accuracy of a CMD-based network trained on all tasks of the Amazon reviews experiment. We perform a grid search for the two regularization hyper-parameters λ andK. We empirically choose a representative stable region for each parameter, [0.3, 3] for λ and {1, . . . , 7} for K. Since we want to analyze the sensitivity w. r. t. K, we averaged over the λ-dimension, resulting in one accuracy value per K for each of the 12 tasks. Each accuracy is transformed into an accuracy ratio value by dividing it with the accuracy of K = 5. Thus, for each K and task we get one value representing the ratio between the obtained accuracy (for this K and task) and the accuracy of K = 5. The results are shown in fig. 3 (upper left). The accuracy
ratios between K = 5 and K ∈ {3, 4, 6, 7} are lower than 0.5%, which underpins the claim that the accuracy of CMD-based networks does not depend strongly on the choice of K in a range around its default value 5. For K = 1 and K = 2 higher ratio values are obtained. In addition, for these two values many tasks show worse accuracy than obtained by K ∈ {3, 4, 5, 6, 7}. From this we additionally conclude that higher values of K are preferable to K = 1 and K = 2.
The same experimental procedure is performed with MMD regularization wighted by λ ∈ [5, 45] and Gaussian kernel parameter β ∈ [0.3, 1.7]. We calculate the ratio values w. r. t. the accuracy of β = 1.2, since this value of β shows the highest mean accuracy of all tasks. Fig. 3 (upper right) shows the results. It can be seen that the accuracy of the MMD network is more sensitive to parameter changes than the CMD regularized version. Note that the problem of finding the best settings for the parameter β of the Gaussian kernel is a well known problem (Hsu et al., 2003).
The default number of hidden nodes in all our experiments is 256 because of the high classification accuracy of the networks without domain regularization (Source Only) on the source domains. The question arises if the accuracy of the CMD is lower for higher numbers of hidden nodes. That is, if the accuracy ratio between the accuracy, of the CMD regularized networks compared to the accuracy of the Source Only models, decreases with increasing hidden activation dimension. In order to answer this question we calculate these ratio values for each task of the Amazon reviews data set for different number of hidden nodes (128, 256, 384, . . . , 1664). For higher numbers of hidden nodes our Source Only models don’t converge with the optimization settings under consideration. For the parameters λ and K we use our default setting λ = 1 and K = 5. Fig. 3 on the lower left shows the ratio values (vertical axis) for every number of hidden nodes (horizontal axis) and every task (colored lines). It can be seen that the accuracy improvement of the CMD domain regularizer varies between 4% and 6%. However, no accuracy ratio decrease can be observed.
Please note that we use a default setting for K and λ. Thus, fig. 3 shows that our default setting (λ = 1,K = 5) can be used independently of the number of hidden nodes. This is an additional result.
The same procedure is performed with the MMD weighted by parameter λ = 9 and β = 1.2 as these values show the highest classification accuracy for 256 hidden nodes. Fig. 3 on the lower right shows that the accuracy improvement using the MMD decreases with increasing number of hidden nodes for this parameter setting. That is, for accurate performance of the MMD, additional parameter tuning procedures for λ and β need to be performed. Note that the problem of finding the best setting for the parameter β of the Gaussian kernel is a well known problem (Hsu et al., 2003).
6 CONCLUSION AND OUTLOOK
In this paper we proposed the central moment discrepancy (CMD) for domain-invariant representation learning, a distance function between probability distributions. Similar to other state-of-the-art approaches (MMD, KL-divergence, Proxy A-distance), the CMD function can be used to minimize the domain discrepancy of latent feature representations. This is achieved by order-wise differences
of central moments. By using probability theoretic analysis, we proved that CMD is a metric and that convergence in CMD implies convergence in distribution for probability distributions on compact intervals. Our method yields state-of-the-art performance on most tasks of the Office benchmark data set and outperforms Gaussian kernel based MMD, VFAE and DANN on most tasks of the Amazon reviews benchmark data set. These results are achieved with the default parameter setting of K = 5. In addition, we experimentally underpinned the claim that the classification accuracy is not sensitive to the particular choice of K for K ≥ 3. Therefore, no computationally expensive hyper-parameter selection is required.
In our experimental analysis we compared our approach to different other state-of-the-art distribution matching methods like the Maximum Mean Discrepancy (MMD) based on the Gaussian kernel using a quadratic time estimate. In the future we want to extend our experimental analysis to other MMD approaches including other kernels, parameter selection procedures and linear time estimators. In addition, we plan to use the CMD for training generative models and to further investigate the approximation quality of the proposed empirical estimate.
A THEOREM PROOFS
Theorem 1. Let p and q be two probability distributions on a compact interval and let CMD be defined as in (5), then
CMD(p, q) = 0 ⇒ p = q
Proof. Let X and Y be two random vectors that have probability distributions p and q, respectively. Let X̂ = X − E(X) and Ŷ = Y − E(Y ) be the mean centered random variables. From CMD(p, q) = 0 it follows that all moments of the bounded random variables X̂ and Ŷ are equal. Therefore, the joint moment generating functions of X̂ and Ŷ are equal. Using the property that p
and q have compact support, we obtain the equality of the joint distribution functions of X̂ and Ŷ . Since E(X) = E(Y ), it follows that X = Y .
Theorem 2. Let pn and p be probability distributions on a compact interval and let CMD be defined as in (5), then
CMD(pn, p)→ 0 ⇒ pn d−→ p
where d−→ denotes convergence in distribution.
Proof. Let Xn and X be random vectors that have probability distributions pn and p respectively. Let X̂ = X − E(X) and X̂n = Xn − E(Xn) be the mean centered random variables. From CMD(Xn, X) → 0 it follows that the moments of X̂n converge to the moments of X̂ . Therefore, the joint moment generating functions of X̂n converge to the joint moment generating function of X̂ , which implies convergence in distribution of the mean centered random variables. Using E(Xn)→ E(X) we obtain pn d−→ p.
Proposition 1. Let X and Y be bounded random vectors with respective probability distributions p and q on the compact interval [a, b]N . Then
1
|b− a|k ‖ck(X)− ck(Y )‖2 ≤ 2
√ N
( 1
k + 1
( k
k + 1
)k + 1
21+k
) (8)
where ck(X) = E((X − E(X))k) is the vector of all kth order sample central moments of the marginal distributions of p.
Proof. Let X ([a, b]) be the set of all random variables with values in [a, b]. Then it follows that 1
|b− a|k ‖ck(X)− ck(Y )‖2 = ∥∥∥∥ ck(X)|b− a|k − ck(Y )|b− a|k ∥∥∥∥
2 ≤ ∥∥∥∥ ck(X)|b− a|k ∥∥∥∥ 2 + ∥∥∥∥ ck(Y )|b− a|k ∥∥∥∥ 2
= ∥∥∥∥∥E (( X − E(X) |b− a| )k)∥∥∥∥∥ 2 + ∥∥∥∥∥E (( Y − E(Y ) |b− a| )k)∥∥∥∥∥ 2
≤ ∥∥∥∥∥E (∣∣∣∣X − E(X)b− a ∣∣∣∣k )∥∥∥∥∥
2
+ ∥∥∥∥∥E (∣∣∣∣Y − E(Y )b− a ∣∣∣∣k )∥∥∥∥∥
2
≤ 2 √ N sup
X∈X ([a,b]) E (∣∣∣∣X − E(X)b− a ∣∣∣∣k )
The latter term refers to the absolute central moment of order k, for which the smallest upper bound is known (Egozcue et al., 2012):
1
|b− a|k ‖ck(X)− ck(Y )‖2 ≤ 2
√ N sup
x∈[0,1] x(1− x)k + (1− x)xk
Egozcue et al. (2012) also give a more explicit bound:
1
|b− a|k ‖ck(X)− ck(Y )‖2 ≤ 2
√ N
( 1
k + 1
( k
k + 1
)k + 1
21+k
)
ACKNOWLEDGEMENTS
The research reported in this paper has been supported by the Austrian Ministry for Transport, Innovation and Technology, the Federal Ministry of Science, Research and Economy, and the Province of Upper Austria in the frame of the COMET center SCCH.
We would like to thank Bernhard Moser and Florian Sobieczky for fruitful discussions on metric spaces.

Variational auto-encoders, adversarial networks, and kernel scoring rules like MMD have recently gained popularity as methods for learning directed generative models and for other applications like domain adaptation. This paper gives an additional method along the scoring rules direction that uses the matching of central moments to match two probability distributions. The technique is simple, and in the case of domain adaptation, highly effective.
CMD seems like a very nice and straightforward solution to the domain adaptation problem. The method is computationally straightforward to implement, and seems quite stable with respect to the tuning parameters when compared to MMD. I was skeptical reading through this, especially given the fact that you only use K=5 in your experiments, but the results seem quite good. The natural question that I have now is: how will this method do in training generative models? This is beyond the scope of this paper, but it’s the lowest hanging fruit.
Below I give more detailed feedback.
One way to speed up MMD is to use a random Fourier basis as was done in “Fastmmd: Ensemble of circular discrepancy for efficient two-sample test” by Zhao and Meng, 2015. There are also linear time estimators, e.g., in “A Kernel Two-Sample Test“ by Gretton et al., 2012. I don’t think you need to compare against these approaches since you compare to the full MMD, but they should be cited.
The paper “Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy” by Sutherland et al. submitted to ICLR 2017 as well, discusses techniques for optimizing the kernel used in MMD and is worth citing in section 3.
How limiting is the assumption that the distribution has independent marginals?
The sample complexity of MMD depends heavily on the dimensionality of the input space - do you have any intuitions about the sample complexity of CMD? It seems like it's relatively insensitive based on the results in Figure 4, but I would be surprised if this were the case with 10,000 hidden units. I mainly ask this because with generative models, the output space can be quite high-dimensional.
I’m concerned that the central moments won’t be numerically stable at higher orders when backpropagating. This doesn’t seem to be a problem in the experimental results, but perhaps the authors could comment a bit on this? I’m referring to the fact that ck(X) can be very large for k >= 3. Proposition 1 alleviates my concerns that the overall objective is unstable, I’m referring specifically to the individual terms within.
Figure 3 is rather cluttered, and aside from the mouse class it’s not clear to me from the visualization that the CMD regularizer is actually helping. It would be useful to remove some of the classes for the purpose of visualization.
I would like some clarification about the natural geometric interpretations of K=5. Do you mean that the moments up to K=5 have been well-studied? Do you have any references for this? Why does K >= 6 not have a natural geometric interpretation?
Figure 4 should have a legend


We provide an algorithm for polynomial feature expansion that both operates on and produces a compressed sparse row matrix without any densification. For a vector of dimension D, density d, and degree k the algorithm has time complexity O(dD) where k is the polynomial-feature order; this is an improvement by a factor d over the standard method.
1 INTRODUCTION
Polynomial feature expansion has long been used in statistics to approximate nonlinear functions Gergonne (1974); Smith (1918). The compressed sparse row (CSR) matrix format is a widelyused data structure to hold design matrices for statistics and machine learning applications. However, polynomial expansions are typically not performed directly on sparse CSR matrices, nor on any sparse matrix format for that matter, without intermediate densification steps. This densification not only adds extra overhead, but wastefully computes combinations of features that have a product of zero, which are then discarded during conversion into a sparse format.
We provide an algorithm that allows CSR matrices to be the input of a polynomial feature expansion without any densification. The algorithm leverages the CSR format to only compute products of features that result in nonzero values. This exploits the sparsity of the data to achieve an improved time complexity of O(dkDk) on each vector of the matrix where k is the degree of the expansion, D is the dimensionality, and d is the density. The standard algorithm has time complexity O(Dk). Since 0 ≤ d ≤ 1, our algorithm is a significant improvement. While the algorithm we describe uses CSR matrices, it could be modified to operate on other sparse formats.
2 PRELIMINARIES
Matrices are denoted by uppercase bold letters thus: A. The ithe row of A is written ai. All vectors are written in bold, and a, with no subscript, is a vector.
A compressed sparse row (CSR) matrix representation of an r-row matrix A consists of three vectors: c, d, and p and a single number: the number of columns of A. The vectors c and d contain the same number of elements, and hold the column indices and data values, respectively, of all nonzero elements of A. The vector p has r entries. The values in p index both c and d. The ith entry pi of p tells where the data describing nonzero columns of ai are within the other two vectors: cpi:pi+1 contain the column indices of those entries; dpi:pi+1 contain the entries themselves. Since only nonzero elements of each row are held, the overall number of columns of A must also be stored, since it cannot be derived from the other data.
Scalars, vectors, and matrices are often referenced with the superscript k. This is not to be interpreted as an exponent, but to indicate that it is the analogous aspect of that which procedes it, but in its polynomial expansion form. For example, c2 is the vector that holds columns for nonzero values in A’s quadratic feature expansion CSR representation.
For simplicity in the presentation, we work with polynomial expansions of degree 2, but continue to use the exponent k to show how the ideas apply in the general case. ∗Now at Google †The authors contributed equally important and fundamental aspects of this work.
We do provide an algorithm for third degree expansions, and derive the big-O time complexity of the general case.
We have also developed an algorithm for second and third degree interaction features (combinations without repetition), which can be found in the implementation.
3 MOTIVATION
In this section, we present a strawman algorithm for computing polynomial feature expansions on dense matrices. We then modify the algorithm slightly to operate on a CSR matrix, in order to expose its infeasibility in that context. We then show how the algorithm would be feasible with an added component, which we then derive in the following section.
3.1 DENSE EXPANSION ALGORITHM
A natural way to calculate polynomial features for a matrix A is to walk down its rows and, for each row, take products of all k-combinations of elements. To determine in which column of Aki products of elements in Ai belong, a simple counter can be set to zero for each row of A and incremented efter each polynomial feature is generated. This counter gives the column of Aki into which each expansion feature belongs.
SECOND ORDER (k = 2) DENSE POLYNOMIAL EXPANSION ALGORITHM(A) 1 N = row count of A 2 D = column count of A 3 Ak = empty N × ( D 2 ) matrix 4 for i = 0 to N − 1 5 cp = 0 6 for j1 = 0 to D − 1 7 for j2 = j1 to D − 1 8 Akicp = Aij1 ·Aij2 9 cp = cp + 1
3.2 IMPERFECT CSR EXPANSION ALGORITHM
Now consider how this algorithm might be modified to accept a CSR matrix. Instead of walking directly down rows of A, we will walk down sections of c and d partitioned by p, and instead of inserting polynomial features into Ak, we will insert column numbers into ck and data elements into dk.
INCOMPLETE SECOND ORDER (k = 2) CSR POLYNOMIAL EXPANSION ALGORITHM(A) 1 N = row count of A 2 pk = vector of size N + 1 3 pk0 = 0 4 nnzk = 0 5 for i = 0 to N − 1 6 istart = pi 7 istop = pi+1 8 ci = cistart:istop 9 nnzki = (|ci| 2
) 10 nnzk = nnzk + nnzki 11 pki+1 = p k i + nnz k i
// Build up the elements of pk, ck, and dk
12 pk = vector of size N + 1 13 ck = vector of size nnzk 14 dk = vector of size nnzk 15 n = 0 16 for i = 0 to N − 1 17 istart = pi 18 istop = pi+1 19 ci = cistart:istop 20 di = distart:istop 21 for c1 = 0 to |ci| − 1 22 for c2 = c1 to |ci| − 1 23 dkn = dc0 · dc1 24 ckn =? 25 n = n+ 1
The crux of the problem is at line 24. Given the arbitrary columns involved in a polynomial feature of Ai, we need to determine the corresponding column of Aki . We cannot simply reset a counter for each row as we did in the dense algorithm, because only columns corresponding to nonzero values are stored. Any time a column that would have held a zero value is implicitly skipped, the counter would err.
To develop a general algorithm, we require a mapping from columns of A to a column of Ak. If there are D columns of A and ( D k ) columns of Ak, this can be accomplished by a bijective mapping of the following form:
(j0, j1, . . . , jk−1) pj0j1...ik−1 ∈ {0, 1, . . . , ( D
k
) − 1} (1)
such that 0 ≤ j0 ≤ j1 ≤ · · · ≤ jk−1 < D where (j0, j1, . . . , jk−1) are elements of c and pj0j1...ik−1 is an element of ck.
4 CONSTRUCTION OF MAPPING
Within this section, i, j, and k denote column indices. For the second degree case, we seek a map from matrix indices (i, j) (with 0 ≤ i < j < D ) to numbers f(i, j) with 0 ≤ f(i, j) < D(D−1)2 , one that follows the pattern indicated by x 0 1 3x x 2 4x x x 5
x x x x  (2) where the entry in row i, column j, displays the value f(i, j). We let T2(n) = 12n(n + 1) be the nth triangular number; then in Equation 2, column j (for j > 0) contains entries with T2(j − 1) ≤
e < T2(j); the entry in the ith row is just i + T2(j − 1). Thus we have f(i, j) = i + T2(j − 1) = 1 2 (2i + j
2 − j). For instance, in column j = 2 in our example (the third column), the entry in row i = 1 is i+ T2(j − 1) = 1 + 1 = 2. With one-based indexing in both the domain and codomain, the formula above becomes f1(i, j) = 1 2 (2i+ j
2 − 3j + 2). For polynomial features, we seek a similar map g, one that also handles the case i = j. In this case, a similar analysis yields g(i, j) = i+ T2(j) = 12 (2i+ j 2 + j + 1).
To handle three-way interactions, we need to map triples of indices in a 3-index array to a flat list, and similarly for higher-order interactions. For this, we’ll need the tetrahedral numbers T3(n) =∑n
i=1 T2(n) = 1 6 (n 3 + 3n2 + 2n).
For three indices, i, j, k, with 0 ≤ i < j < k < D, we have a similar recurrence. Calling the mapping h, we have
h(i, j, k) = i+ T2(j − 1) + T3(k − 2); (3) if we define T1(i) = i, then this has the very regular form
h(i, j, k) = T1(i) + T2(j − 1) + T3(k − 2); (4) and from this the generalization to higher dimensions is straightforward. The formulas for “higher triangular numbers”, i.e., those defined by
Tk(n) = n∑ i=1 Tk−1(n) (5)
for k > 1 can be determined inductively.
The explicit formula for 3-way interactions, with zero-based indexing, is
h(i, j, k) = 1 + (i− 1) + (j − 1)j 2 + (6)
(k − 2)3 + 3(k − 2)2 + 2(k − 2) 6 . (7)
5 FINAL CSR EXPANSION ALGORITHM
With the mapping from columns of A to a column of Ak, we can now write the final form of the innermost loop of the algorithm from 3.2. Let the mapping for k = 2 be denoted h2. Then the innermost loop becomes:
for c2 = c1 to |ci| − 1 j0 = cc0 j1 = cc1 cp = h
2(j0, j1) dkn = dc0 · dc1 ckn = cp n = n+ 1
The algorithm can be generalized to higher degrees by simply adding more nested loops, using higher order mappings, modifying the output dimensionality, and adjusting the counting of nonzero polynomial features in line 9.
6 TIME COMPLEXITY
6.1 ANALYTICAL
Calculating k-degree polynomial features via our method for a vector of dimensionality D and density d requires ( dD k ) (with repetition) products. The complexity of the algorithm, for fixed k
dD, is therefore
O
(( dD + k − 1
k
)) = O ( (dD + k − 1)! k!(dD − 1)! ) (8)
= O
( (dD + k − 1)(dD + k − 2) . . . (dD)
k!
) (9)
= O ((dD + k − 1)(dD + k − 2) . . . (dD)) for k dD (10) = O ( dkDk ) (11)
6.2 EMPIRICAL
To demonstrate how our algorithm scales with the density of a matrix, we compare it to the traditional polynomial expansion algorithm in the popular machine library scikit-learn Pedregosa et al. (2011) in the task of generating second degree polynomial expansions. Matrices of size 100× 5000 were randomly generated with densities of 0.2, 0.4, 0.6, 0.8, and 1.0. Thirty matrices of each density were randomly generated, and the mean times (gray) of each algorithm were plotted. The red or blue width around the mean marks the third standard deviation from the mean. The time to densify the input to the standard algorithm was not counted.
The standard algorithm’s runtime stays constant no matter the density of the matrix. This is because it does not avoid products that result in zero, but simply multiplies all second order combinations of features. Our algorithm scales quadratically with respect to the density. If the task were third degree expansions rather than second, the plot would show cubic scaling.
The fact that our algorithm is approximately 6.5 times faster than the scikit-learn algorithm on 100× 5000 matrices that are entirely dense is likely a language implementation difference. What matters is that the time of our algorithm increases quadratically with respect to the density in accordance with the big-O analysis.
7 CONCLUSION
We have developed an algorithm for performing polynomial feature expansions on CSR matrices that scales polynomially with respect to the density of the matrix. The areas within machine learning that this work touches are not en vogue, but they are workhorses of industry, and every improvement in core representations has an impact across a broad range of applications.

This paper proposes an algorithm for polynomial feature expansion on CSR matrices, which reduces the time complexity of the standard method by a factor d^k where d is the density of the sparse matrix. The main contribution of this work is not significant enough. The experiments are incomplete and not convincing.
The background of the problem is not sufficiently introduced. There are only two references in the introduction part (overall only three papers are cited), which are from decades ago. Many more relevant papers should be cited from the recent literature.
The experiment part is very weak. This paper claims that the time complexity of their algorithm is O(d^k D^k), which is an improvement over standard method O(D^k) by a factor d^k. But in the experiments, when d=1, there is still a large gap (~14s vs. ~90s) between the proposed method and the standard one. The authors explain this as "likely a language implementation", which is not convincing. To fairly compare the two methods, of course you need to implement both in the same programming language and run experiments in the same environment. For higher degree feature expansion, there is no empirical experiments to show the advantage of the proposed method.
Some minor problems are listed below.
1) In Section 2, the notation "p_i:p_i+1" is not clearly defined.
2) In Section 3.1, typo: "efter" - "after"
3) All the algorithms in this paper are not titled. The input and output is not clearly listed.
4) In Figure 1, the meaning of the colored area is not described. Is it standard deviation or some quantile of the running time? How many runs of each algorithm are used to generate the ribbons? Many details of the experimental settings are missing.