summaries: The paper presents an alternative view on the training procedure for the VQ-VAE. The authors have noticed that there is a close connection between the original training algorithm and the well-known EM algorithm. Then, they proposed to use the soft EM algorithm. In the experiments the authors showed that the soft EM allows to obtain significantly better results than the standard learning procedure on both image and text datasets.

summaries: This paper discusses VQ-VAE for learning discrete latent variables, and its application to NMT with a non-autoregressive decoder to reduce latency (obtained by producing a number of latent variables that is much smaller than the number of target words, and then producing all target words in parallel conditioned on the latent variables and the source text). The authors show the connection between the existing EMA technique for learning the discrete latent states and hard EM, and introduce a Monte-Carlo EM algorithm as a new learning technique. They show strong empirical results on EN-DE NMT with a latent Transformer (Kaiser et al. (2018)).

abstractText: Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable models has remained challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete autoencoder with EM and combining it with sequence level knowledge distillation alows us to develop a nonautoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

authors:
Leon Bottou
Yoshua Bengio

title: Convergence properties of the k-means algorithms

venue: In Advances in neural information processing systems,

authors:
Kyunghyun Cho
Bart van Merrienboer
Caglar Gulcehre
Fethi Bougares
Holger Schwenk
Yoshua Bengio

title: Learning phrase representations using RNN encoder-decoder for statistical machine translation

authors:
Arthur P Dempster
Nan M Laird
Donald B Rubin

title: Maximum likelihood from incomplete data via the em algorithm

venue: Journal of the royal statistical society. Series B (methodological),

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

authors:
Will Grathwohl
Dami Choi
Yuhuai Wu
Geoff Roeder
David Duvenaud

title: Backpropagation through the void: Optimizing control variates for black-box gradient estimation

venue: arXiv preprint arXiv:1711.00123,

authors:
Jiatao Gu
James Bradbury
Caiming Xiong
Victor O.K. Li
Richard Socher

title: Nonautoregressive neural machine translation

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparameterization with gumbel-softmax

authors:
Łukasz Kaiser
Samy Bengio

title: Discrete autoencoders for sequence models

authors:
Łukasz Kaiser
Aurko Roy
Ashish Vaswani
Niki Pamar
Samy Bengio
Jakob Uszkoreit
Noam Shazeer

title: Fast decoding in sequence models using discrete latent variables

venue: arXiv preprint arXiv:1803.03382,

authors:
Nal Kalchbrenner
Aaron van den Oord
Karen Simonyan
Ivo Danihelka
Oriol Vinyals
Alex Graves
Koray Kavukcuoglu

title: Video pixel networks

venue: arXiv preprint arXiv:1610.00527,

authors:
Yoon Kim
Alexander Rush

title: Sequence-level knowledge distillation

authors:
D. Kingma
M. Welling

title: Auto-encoding variational Bayes

venue: In ICLR,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Tim Salimans
Rafal Jozefowicz
Xi Chen
Ilya Sutskever
Max Welling

title: Improved variational inference with inverse autoregressive flow

venue: In Advances in Neural Information Processing Systems,

authors:
Guillaume Lample
Myle Ott
Alexis Conneau
Ludovic Denoyer
Marc’Aurelio Ranzato

title: Phrase-based & neural unsupervised machine translation

venue: arXiv preprint arXiv:1804.07755,

authors:
Jason Lee
Elman Mansimov
Kyunghyun Cho

title: Deterministic non-autoregressive neural sequence modeling by iterative refinement

venue: arXiv preprint arXiv:1802.06901,

authors:
Percy Liang
Dan Klein

title: Online em for unsupervised models. In Proceedings of human language technologies: The 2009 annual conference of the North American chapter of the association for computational linguistics, pp. 611–619

venue: Association for Computational Linguistics,

authors:
Ilya Loshchilov
Frank Hutter

title: Fixing weight decay regularization in adam

venue: arXiv preprint arXiv:1711.05101,

title: Some methods for classification and analysis of multivariate observations

venue: In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability,

authors:
Chris J. Maddison
Andriy Mnih
Yee Whye Teh

title: The concrete distribution: A continuous relaxation of discrete random variables

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

venue: In NIPS workshop on deep learning and unsupervised feature learning,

authors:
Mohammad Norouzi
David J Fleet

title: Cartesian k-means

venue: In Computer Vision and Pattern Recognition (CVPR),

authors:
Gabriel Pereyra
George Tucker
Jan Chorowski
Łukasz Kaiser
Geoffrey Hinton

title: Regularizing neural networks by penalizing confident output distributions

venue: arXiv preprint arXiv:1701.06548,

authors:
Scott Reed
Aäron van den Oord
Nal Kalchbrenner
Sergio Gómez Colmenarejo
Ziyu Wang
Dan Belov
Nando de Freitas

title: Parallel multiscale autoregressive density estimation

venue: arXiv preprint arXiv:1703.03664,

authors:
Tim Salimans
Andrej Karpathy
Xi Chen
Diederik P Kingma

title: Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications

venue: arXiv preprint arXiv:1701.05517,

authors:
Masa-Aki Sato
Shin Ishii

title: On-line em algorithm for the normalized gaussian network

venue: Neural computation,

title: Web-scale k-means clustering

venue: In Proceedings of the 19th international conference on World wide web,

authors:
Ilya Sutskever
Oriol Vinyals
Quoc V. Le

title: Sequence to sequence learning with neural networks

venue: In Advances in Neural Information Processing Systems, pp. 3104–3112,

authors:
Lucas Theis
Wenzhe Shi
Andrew Cunningham
Ferenc Huszár

title: Lossy image compression with compressive autoencoders

venue: arXiv preprint arXiv:1703.00395,

authors:
George Tucker
Andriy Mnih
Chris J Maddison
John Lawson
Jascha Sohl-Dickstein

title: Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models

authors:
Aaron Van Den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Aaron van den Oord
Nal Kalchbrenner
Lasse Espeholt
Oriol Vinyals
Alex Graves

title: Conditional image generation with pixelcnn decoders

authors:
Aäron van den Oord
Oriol Vinyals
Koray Kavukcuoglu

title: Neural discrete representation learning

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need. CoRR, 2017

authors:
Oriol Vinyals
Alexander Toshev
Samy Bengio
Dumitru Erhan

title: Show and tell: A neural image caption generator

authors:
Greg CG Wei
Martin A Tanner

title: A monte carlo implementation of the em algorithm and the poor man’s data augmentation algorithms

venue: Journal of the American statistical Association,

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

venue: In Reinforcement Learning,

title: 2017) for image reconstruction. We use the notation x to denote the input image, with the output of the encoder ze(x) ∈ R being used to perform nearest neighbor search to select the (sequence of) discrete latent

title: 2017) and EM on the CIFAR-10 data set, modeling the joint probability P

authors:
VQ-VAE (van den Oord et al
EM

title: The autoencoder is similar to our CIFAR-10

title: The y-axis denotes the teacher-forced BLEU score on the test set, which is used only for evaluation while training. Notice that the VQ-VAE run collapsed (green curve), while the EM runs (yellow and blue curves) exhibit more stability

venue: Emergence of EOS/PAD latent:

heading: 1 Introduction

text:
Unsupervised learning of meaningful representations is a fundamental problem in machine learning since obtaining labeled data can often be very expensive. Continuous representations have largely been the workhorse of unsupervised deep learning models of images (Goodfellow et al., 2014; van den Oord et al., 2016; Kingma et al., 2016; Salimans et al., 2017; Parmar et al., 2018), audio (Van Den Oord et al., 2016; Reed et al., 2017), and video (Kalchbrenner et al., 2016). However, it is often the case that datasets are more naturally modeled as a sequence of discrete symbols rather than continuous ones. For example, language and speech are inherently discrete in nature and images are often concisely described by language, see e.g., Vinyals et al. (2015). Improved discrete latent variable models could also prove useful for learning novel data compression algorithms (Theis et al., 2017), while having far more interpretable representations of the data.
We build on Vector Quantized Variational Autoencoder (VQ-VAE) (van den Oord et al., 2017), a recently proposed training technique for learning discrete latent variables. The method uses a learned code-book combined with nearest neighbor search to train the discrete latent variable model. The nearest neighbor search is performed between the encoder output and the embedding of the latent code using the `2 distance metric. VQ-VAE adopts the standard latent variable model generative process, first sampling latent codes from a prior, P (z), which are then consumed by the decoder to generate data from P (x | z). In van den Oord et al. (2017), the authors use both uniform and autoregressive priors for P (z). The resulting discrete autoencoder obtains impressive results on unconditional image, speech, and video generation. In particular, on image generation, VQ-VAE was shown to perform almost on par with continuous VAEs on datasets such as CIFAR-10 (van den Oord et al., 2017). An extension of this method to conditional supervised generation, out-performs continuous autoencoders on WMT English-German translation task (Kaiser et al., 2018).
The work of Kaiser et al. (2018) introduced the Latent Transformer, which set a new stateof-the-art in non-autoregressive Neural Machine Translation. However, additional training heuristics, namely, exponential moving averages (EMA) of cluster assignment counts, and
product quantization (Norouzi & Fleet, 2013) were essential to achieve competitive results with VQ-VAE. In this work, we show that tuning for the code-book size can significantly outperform the results presented in Kaiser et al. (2018). We also exploit VQ-VAE’s connection with the expectation maximization (EM) algorithm (Dempster et al., 1977), yielding additional improvements. With both improvements, we achieve a BLEU score of 22.4 on English to German translation, outperforming Kaiser et al. (2018) by 2.6 BLEU. Knowledge distillation (Hinton et al., 2015; Kim & Rush, 2016) provides significant gains with our best models and EM, achieving 26.7 BLEU, which almost matches the autoregressive transformer model with no beam search at 27.0 BLEU, while being 3.3× faster. Our contributions can be summarized as follows:
1. We show that VQ-VAE from van den Oord et al. (2017) can outperform previous state-of-the-art without product quantization.
2. Inspired by the EM algorithm, we introduce a new training algorithm for training discrete variational autoencoders, that outperforms the previous best result with discrete latent autoencoders for neural machine translation.
3. Using EM training, and combining it sequence level knowledge distillation (Hinton et al., 2015; Kim & Rush, 2016), allows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.
4. On the larger English-French dataset, we show that denoising discrete autoencoders gives us a significant improvement (1.0 BLEU) on top of our non-autoregressive baseline (see Section D).

heading: 2 VQ-VAE and the Hard EM Algorithm

text: The connection between K-means, and hard EM, or the Viterbi EM algorithm is well known (Bottou & Bengio, 1995), where the former can be seen a special case of hard-EM style algorithm with a mixture-of-Gaussians model with identity covariance and uniform prior over cluster probabilities. In the following sections we briefly explain the VQ-VAE discrete autoencoder for completeness and it’s connection to classical EM.

heading: 2.1 VQ-VAE discretization algorithm

text:
VQ-VAE models the joint distribution PΘ(x, z) where Θ are the model parameters, x is the data point and z is the sequence of discrete latent variables or codes. Each position in the encoded sequence has its own set of latent codes. Given a data point, the discrete latent code in each position is selected independently using the encoder output. For simplicity, we describe the procedure for selecting the discrete latent code (zi) in one position given the data point (xi). The encoder output ze(xi) ∈ RD is passed through a discretization bottleneck using a nearest-neighbor lookup on embedding vectors e ∈ RK×D. Here K is the number of latent codes (in a particular position of the discrete latent sequence) in the model. More specifically, the discrete latent variable assignment is given by,
zi = arg min j∈[K] ‖ze(xi)− ej‖2 (1)
The selected latent variable’s embedding is passed as input to the decoder,
zq(xi) = ezi
The model is trained to minimize:
L = lr + β ‖ze(xi)− sg (zq(xi))‖2 , (2) where lr is the reconstruction loss of the decoder given zq(x) (e.g., the cross entropy loss), and, sg (.) is the stop gradient operator defined as follows:
sg (x) = { x forward pass 0 backward pass
To train the embedding vectors e ∈ RK×D, van den Oord et al. (2017) proposed using a gradient based loss function
‖sg (ze(xi))− zq(xi)‖2 , (3)
and also suggested an alternate technique of training the embeddings: by maintaining an exponential moving average (EMA) of all the encoder hidden states that get assigned to it. It was observed in Kaiser et al. (2018) that the EMA update for training the code-book embedding, results in more stable training than using gradient-based methods. We analyze this in more detail in Section 5.1.1.
Specifically, an exponential moving average is maintained over the following two quantities: 1) the embeddings ej for every j ∈ [1, . . . ,K] and, 2) the count cj measuring the number of encoder hidden states that have ej as it’s nearest neighbor. The counts are updated in a mini-batch of targets as:
cj ← λcj + (1− λ) ∑ i 1 [zq(xi) = ej ] , (4)
with the embedding ej being subsequently updated as:
ej ← λej + (1− λ) ∑ i 1 [zq(xi) = ej ] ze(xi) cj , (5)
where 1[.] is the indicator function and λ is a decay parameter which we set to 0.999 in our experiments. This amounts to doing stochastic gradient in the space of both code-book embeddings and cluster assignments. These techniques have also been successfully used in minibatch K-means (Sculley, 2010) and online EM (Liang & Klein, 2009; Sato & Ishii, 2000).
The generative process for our latent variable NMT model, P (y, z | x), begins by autoregressively sampling a sequence of discrete latent codes from a model conditioned on the input x,
P (z | x) = |z|∏ i=1 P ( zi | z1,...,(i−1), x ) , (6)
which we refer to as the Latent Predictor model (Kaiser et al., 2018). The decoder then consumes this sequence of discrete latent variables to generate the target y all at once, where
P (y | z, x) = |y|∏ j=1 P (yj | z, x). (7)
The autoregressive learned prior prior is fit on the discrete latent variables produced by the autoencoder. Our goal is to learn a sequence of latents, that is much shorter than the targets, |z| |y|, thereby speeding up decoding significantly with no loss in accuracy. The architecture of the encoder, the decoder, and the latent predictor model are described in further detail in Section 5.
2.2 Hard EM and the K-means algorithm
In this section we briefly recall the hard Expectation maximization (EM) algorithm (Dempster et al., 1977). Given a set of data points (x1, . . . , xN ), the hard EM algorithm approximately solves the following optimization problem:
Θ∗ = arg max Θ max z1,...,zN PΘ(x1, . . . , xN , z1, . . . , zN ), (8)
Hard EM performs coordinate descent over the following two coordinates: the model parameters Θ, and the hidden variables z1, . . . , zN . In other words, hard EM consists of repeating the following two steps until convergence:
1. E step: (z1, . . . , zN )← arg maxz1,...,zN PΘ(x1, . . . , xN , z1, . . . , zN ),
2. M step: Θ← arg maxΘ PΘ(x1, . . . , xN , z1, . . . , zN )
A special case of the hard EM algorithm is K-means clustering (MacQueen et al., 1967; Bottou & Bengio, 1995) where the likelihood is modelled by a Gaussian with identity covariance matrix. Here, the means of the K Gaussians are the parameters to be estimated,
Θ = 〈µ1, . . . , µK〉, µk ∈ RD.
With a uniform prior over the hidden variables (PΘ(zi) = 1K ), the marginal is given by PΘ(xi | zi) = N (µzi , I)(xi). In this case, equation (8) is equivalent to:
( µ1, . . . , µK )∗ = arg max
µ1,...,µK min z1,...,zN N∑ i=1 ‖µzi − xi‖22 (9)
Note that optimizing equation (9) is NP-hard, however one can find a local optima by applying coordinate descent until convergence:
1. E step: Cluster assignment is given by,
zi ← arg min j∈[K] ∥∥µj − xi∥∥22 , (10) 2. M step: The means of the clusters are updated as,
cj ← N∑ i=1 1[zi = j]; µ j ← 1 cj N∑ i=1 1[zi = j]xi. (11)
We can now easily see the connections between the training updates of VQ-VAE and K-means clustering. The encoder output ze(x) ∈ RD corresponds to the data point while the discrete latent variables corresponds to clusters. Given this, Equation 1 is equivalent to the E-step (Equation 10) and the EMA updates in Equation 4 and Equation 5 converge to the M-step (Equation 11) in the limit. The M-step in K-means overwrites the old values while the EMA updates interpolate between the old values and the M step update.

heading: 3 VQ-VAE training with EM

text: In this section, we investigate a new training strategy for VQ-VAE using the EM algorithm.

heading: 3.1 Expectation Maximization

text:
First, we briefly describe the EM algorithm. While the hard EM procedure selects one cluster or latent variable assignment for a data point, here the data point is assigned to a mixture of clusters. Now, the optimization objective is given by,
Θ∗ = arg max Θ PΘ(x1, . . . , xN )
= arg max Θ ∑ z1,...,zN PΘ(x1, . . . , xN , z1, . . . , zN )
Coordinate descent algorithm is again used to approximately solve the above optimization algorithm. The E and M step are given by:
1. E step:
ρ(zi)← PΘ(zi | xi), (12)
2. M step:
Θ← arg max Θ Ezi∼ρ[logPΘ(xi, zi)] (13)

heading: 3.2 Vector Quantized Autoencoders trained with EM

text:
Now, we describe vector quantized autoencoders training using the EM algorithm. As discussed in the previous section, the encoder output ze(x) ∈ RD corresponds to the data point while the discrete latent variables corresponds to clusters. The E step instead of hard assignment now produces a probability distribution over the set of discrete latent variables (Equation 12). Following VQ-VAE, we continue to assume a uniform prior over clusters, since we observe that training the cluster priors seemed to cause the cluster assignments to collapse to only a few clusters. The probability distribution is modeled as a Gaussian with identity covariance matrix,
PΘ(zi | ze(xi)) ∝ e−‖ezi−ze(xi)‖ 2 2
As an alternative to computing the full expectation term in the M step (Equation 13) we perform Monte-Carlo Expectation Maximization (Wei & Tanner, 1990) by drawing m samples z1i , · · · , zmi ∼ Multinomial ( −‖e1 − ze(xi)‖22 , . . . ,−‖eK − ze(xi)‖ 2 2 ) , where
Multinomial(l1, . . . , lK) refers to the K-way multinomial distribution with logits l1, . . . , lK . This results in a less diffuse target for the autoregressive prior. Thus, the E step can be finally written as:
E step: z1i , . . . , z m i ← Multinomial ( −‖e1 − ze(xi)‖22 , . . . ,−‖eK − ze(xi)‖ 2 2 ) The model parameters Θ are then updated to maximize this Monte-Carlo estimate in the M step given by
M step: cj ← 1
m N∑ i=1 m∑ l=1 1 [ zli = j ] ; ej ← 1 mcj N∑ i=1 m∑ l=1 1 [ zli = j ] ze(xi).
Instead of exactly following the above M step update, we use the EMA version of this update similar to the one described in Section 2.1.
When sending the embedding of the discrete latent to the decoder, instead of sending the posterior mode, argmaxzP (z | x), similar to hard EM and K-means, we send the average of the embeddings of the sampled latents:
zq(xi) = 1
m m∑ l=1 ezli . (14)
Since m latent code embeddings are sent to the decoder in the forward pass, all of them are updated in the backward pass for a single training example. In hard EM training, only one of them is updated during training. Sending averaged embeddings also results in more stable training using the EM algorithm compared to VQ-VAE as shown in Section 5.
To train the latent predictor model (Section 2.1) in this case, we use an approach similar to label smoothing (Pereyra et al., 2017): the latent predictor model is trained to minimize the cross entropy loss with the labels being the average of the one-hot labels of z1i , . . . , zmi .

heading: 4 Other Related Work

text:
Variational autoencoders were first introduced by Kingma & Welling (2014) for training continuous representations; unfortunately, training them for discrete latent variable models has proved challenging. One promising approach has been to use various gradient estimators for discrete latent variable models, starting with the REINFORCE estimator of Williams (1992), an unbiased, high-variance gradient estimator. Subsequent work on improving the variance of the REINFORCE estimator are REBAR (Tucker et al., 2017) and RELAX (Grathwohl et al., 2017). An alternate approach towards gradient estimators is to use continuous relaxations of categorical distributions, for e.g., the Gumbel-Softmax reparametrization trick (Jang et al.,
2016; Maddison et al., 2016). These methods provide biased but low variance gradients for training.
Machine translation using deep neural networks have been shown to achieve impressive results (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014; Vaswani et al., 2017). The state-of-the-art models in Neural Machine Translation are all auto-regressive, which means that during decoding, the model consumes all previously generated tokens to predict the next one. Recently, there have been multiple efforts to speed-up machine translation decoding. Gu et al. (2017) attempts to address this issue by using the Transformer model (Vaswani et al., 2017) together with the REINFORCE algorithm (Williams, 1992), to model the fertilities of words. The main drawback of the approach of Gu et al. (2017) is the need for extensive fine-tuning to make policy gradients work, as well as the non-generic nature of the solution. Lee et al. (2018) propose a non-autoregressive model using iterative refinement. Here, instead of decoding the target sentence in one-shot, the output is successively refined to produce the final output. While the output is produced in parallel at each step, the refinement steps happen sequentially.

heading: 5 Experiments

text: In this section we report our experiments with VQ-VAE and EM on the English-German translation task, with the aim of improving the decoding speed of autoregressive translation models. Our model and generative process follows the architecture proposed in Kaiser et al. (2018) and is depicted in Figure 1. For all our experiments, we use the Adam (Kingma & Ba, 2014) optimizer and decay the learning rate exponentially after initial warm-up steps. Unless otherwise stated, the dimension of the hidden states of the encoder and the decoder is 512, see Table 4 for a comparison of models with lower dimension. For all configurations we select the optimal hyperparameters by using WMT’13 English-German as the validation set and reporting the BLEU score on the WMT’14 English-German test set.

heading: 5.1 Machine Translation

text:
In Neural Machine Translation with latent variables, we model P (y, z | x), where y and x are the target and source sentence respectively. Our model architecture, depicted in Figure 1, is similar to the one in Kaiser et al. (2018). The encoder function is a series of strided convolutional layers with residual convolutional layers in between and takes target sentence y
as input. The source sentence x is converted to a sequence of hidden states through multiple causal self-attention layers. In Kaiser et al. (2018), the encoder of the autoencoder attends additionally to this sequence of continuous representation of the source sentence. We use VQ-VAE as the discretization algorithm. The decoders, applied after the bottleneck layer uses transposed convolution layers whose continuous output is fed to a transformer decoder with causal attention, which generates the output.
The results are summarized in Table 1. Our implementation of VQ-VAE achieves a significantly better BLEU score and faster decoding speed compared to Kaiser et al. (2018). We found that tuning the code-book size (number of clusters) for using 212 discrete latents achieves the best accuracy which is 16 times smaller as compared to the code-book size in Kaiser et al. (2018). Additionally, we see a large improvement in the performance of the model by using sequence-level distillation (Hinton et al., 2015; Kim & Rush, 2016), as has been observed previously in non-autoregressive models (Gu et al., 2017; Lee et al., 2018). Our teacher model is a base Transformer (Vaswani et al., 2017) that achieves a BLEU score of 28.1 and 27.0 on the WMT’14 test set using beam search decoding and greedy decoding respectively. The distilled data is decoded from the base Transformer using a beam size of 4. Our VQ-VAE model trained with soft EM and distillation, achieves a BLEU score of 26.7, without noisy parallel decoding (Gu et al., 2017). This perforamce is 1.4 bleu points lower than an autoregressive model decoded with a beam size of 4, while being 4.1× faster. Importantly, we nearly match the same autoregressive model with beam size 1 (greedy decoding), with a 3.3× speedup. The length of the sequence of discrete latent variables is shorter than that of target sentence y. Specifically, at each compression step of the encoder we reduce its length by half. We denote by nc, the compression factor for the latents, i.e. the number of steps for which we do this compression. In almost all our experiments, we use nc = 3 reducing the length by 8. We can decrease the decoding time further by increasing the number of compression steps. As shown in Table 1, by setting nc to 4, the decoding time drops to 58 milliseconds achieving 25.4 BLEU while a NAT model (Gu et al., 2017) with similar decoding speed achieves only 18.7 BLEU. Note that, all NAT models also train with sequence level knowledge distillation from an autoregressive teacher.

heading: 5.1.1 Analysis

text:
Attention to Source Sentence Encoder: While the encoder of the discrete autoencoder in Kaiser et al. (2018) attends to the output of the encoder of the source sentence, we find that to be unnecessary, with both models achieving the same BLEU score with 212 latents. Removing this attention step results in more stable training (see Figure 3) and is the main reason why VQ-VAE works in our setup (see Table 1) without the use of Product Quantization (DVQ) (Kaiser et al., 2018). Note that the decoder of the discrete autoencoder in both Kaiser et al. (2018) and our work does not attend to the source sentence.
Size of Discrete Latent Variable code-book: Table 2 shows the BLEU score for different code-book sizes for models trained using VQ-VAE without distillation. While Kaiser et al. (2018) use 216 as their code-book size, we find that 212 gives the best performance.
Number of samples in Monte-Carlo EM update: While training with EM, we perform a Monte-Carlo update with a small number of samples (Section 3.2). Table 3 shows the impact of number of samples on the final BLEU score.
VQ-VAE vs Other Discretization Techniques: We compare the Gumbel-Softmax of (Jang et al., 2016; Maddison et al., 2016) and the improved semantic hashing discretization technique proposed in Kaiser et al. (2018) to VQ-VAE. When trained with sequence level knowledge distillation, the model using Gumbel-Softmax reached 23.2 BLEU, the model using improved semantic hashing reached 24.1 BLEU, and the model using VQ-VAE reached 26.4 BLEU on WMT’14 English-German.
* Speedup reported for these items are compared to the decode time of 408 ms for an autoregressive Transformer from Gu et al. (2017).

heading: 6 Conclusion

text: We investigate an alternate training technique for VQ-VAE inspired by its connection to the EM algorithm. Training the discrete autoencoder with EM and combining it with sequence level knowledge distillation, allows us to develop a non-autoregressive machine translation model whose accuracy almost matches a greedy autoregressive baseline, while being 3.3 times faster at inference. While sequence distillation is very important for training our best model, we find that the improvements from EM on harder tasks is quite significant. We hope that our results will inspire further research on using vector quantization for fast decoding of autoregressive sequence models.

heading: B Ablation Tables

heading: C Additional Analysis

text:
Gradient based update vs EMA update of code-book: The original VQ-VAE paper (van den Oord et al., 2017) proposed a gradient based update rule for learning the code-book where the code-book entries are trained by minimizing ‖sg (ze(x))− zq(x)‖2. However, it was found in Kaiser et al. (2018) that the EMA update worked better than this gradient based loss. Note that if the gradient based loss was minimized using SGD then the update rule for the embeddings is
ej ← (1− η)ej + η (∑
i 1 [zq(xi) = ej ] ze(xi)∑ i 1 [zq(xi) = ej ]
) , (15)
for a learning rate η. This is quite similar to the EMA update rule of Equation 5, with the only difference being that the latter also maintains an EMA over the counts cj . When using SGD with momentum or Adam, the update rule becomes quite different however, since we now take the moving average of the gradient term itself, before subtracting it from current value of the embedding ej . This is similar to the issue of using weight decay with Adam, where using the `2 penalty in the loss function results in worse performance (Loshchilov & Hutter, 2017).
Model Size: The effect of model size on BLEU score for models trained with EM and distillation is shown in Table 4.
Robustness of EM to Hyperparameters: While EM training gives a small performance improvement, we find that it also leads to more robust training for machine translation.
Our experiments on image reconstruction on SVHN (Netzer et al., 2011) in section A also highlight the robustness of EM training. The training approach from van den Oord et al. (2017) exhibits high variance on reconstruction quality, while EM is much more stable, resulting in good reconstructions in almost all training runs.
Emergence of EOS/PAD latent: We observe that all the latent sentences for a specific experiment with VQ-VAE or EM end with a fixed latent indicating the end of the sequence. Since we always fix the length of the latent sentence to be 2nc times smaller than the true sentence, the model learns to pad the remainder of the latent sequence with this special code (see Table 5 for examples). Note that one can speed up decoding even further by stopping the Latent Predictor (LP) model as soon as it outputs this special code.
Denoising autoencoder: We also use word dropout with a dropout rate of 0.3 and word permutation with a shuffle rate of 0.5 as in Lample et al. (2018). On the WMT EnglishGerman we did not notice any improvement from using these regularization techniques, but on the larger WMT English-French dataset, we observe that using a denoising autoencoder significantly improves performance with a gain of 1.0 BLEU on VQ-VAE and 0.9 BLEU over EM (see Table 6).
Additional analysis on latents: In order to compute correlations between the discrete latents and n-grams in the original text, we computed Point-wise Mutual Information (PMI) and tf-idf scores where the latents are treated as documents. However, we were unable to see any semantic patterns that stood out in this analysis.

heading: D Preliminary Results on English French

text: In this section we report preliminary results on the WMT English-French dataset without using knowledge distillation from an autoregressive teacher (Hinton et al., 2015; Kim & Rush, 2016). We use a Transformer base model from Vaswani et al. (2017). Our best non-autoregressive base model trained on non-distilled targets gets 30.0 BLEU compared to the autoregressive base model with the same choice of hyperparameters, which gets 33.3 BLEU (see Table 6). As in the case of English-German, we anticipate that using knowledge distillation Hinton et al. (2015) will likely close this gap.

title: Towards a better understanding of Vector Quantized Autoencoders


summaries: This paper presents a multiview framework for sentence representation in NLP tasks. Authors propose two architectures, one using a generative objective, while the other uses a discriminative objective. Both combine a recurrent based encoding function and a linear model. Large experiments have been conducted on several NLP tasks and datasets, showing improvement of the introduced frameworks compared to baselines.

summaries: This paper is about a multi-view framework for learning sentence representations. Two objective functions (a generative one and a discriminative one) are proposed that make use of two encoders, one of them is based on an RNN and the other on a linear projection of averaged word embeddings. Each of these objective functions has a multi-view framework where their respective objective functions are in part based on making sure their is some relationship between the two different encoders. This multi-view framework is shown to be helpful over having independent encoders in their ablation study.

abstractText: Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.

authors:
Eneko Agirre
Daniel M. Cer
Mona T. Diab
Aitor

title: Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity

authors:
Eneko Agirre
Daniel M. Cer
Mona T. Diab
Aitor Gonzalez-Agirre
Weiwei Guo

title: sem 2013 shared task: Semantic textual similarity

authors:
Eneko Agirre
Carmen Banea
Claire Cardie
Daniel M. Cer
Mona T. Diab
Aitor Gonzalez-Agirre
Weiwei Guo
Rada Mihalcea
German Rigau
Janyce Wiebe

title: Semeval-2014 task 10: Multilingual semantic textual similarity

authors:
Eneko Agirre
Carmen Banea
Claire Cardie
Daniel M. Cer
Mona T. Diab
Aitor Gonzalez-Agirre
Weiwei Guo
Iñigo Lopez-Gazpio
Montse Maritxalar
Rada Mihalcea
German Rigau
Larraitz Uria
Janyce Wiebe

title: Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability

authors:
Eneko Agirre
Carmen Banea
Daniel M. Cer
Mona T. Diab
Aitor Gonzalez-Agirre
Rada Mihalcea
German Rigau
Janyce Wiebe

title: Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation

authors:
Sanjeev Arora
Yuanzhi Li
Yingyu Liang
Tengyu Ma
Andrej Risteski

title: A latent variable model approach to pmi-based word

venue: embeddings. TACL,

authors:
Sanjeev Arora
Yingyu Liang
Tengyu Ma

title: A simple but tough-to-beat baseline for sentence embeddings

venue: In International Conference on Learning Representations,

authors:
Piotr Bojanowski
Edouard Grave
Armand Joulin
Tomas Mikolov

title: Enriching word vectors with subword information

venue: TACL,

authors:
Samuel R. Bowman
Gabor Angeli
Christopher Potts
Christopher D. Manning

title: A large annotated corpus for learning natural language inference

venue: In EMNLP,

title: Laterality functional asymmetry in the intact

venue: brain. Elsevier,

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural networks on sequence modeling

venue: arXiv preprint arXiv:1412.3555,

authors:
Moustapha Cissé
Piotr Bojanowski
Edouard Grave
Yann Dauphin
Nicolas Usunier

title: Parseval networks: Improving robustness to adversarial examples

venue: In ICML,

authors:
Alexis Conneau
Douwe Kiela

title: Senteval: An evaluation toolkit for universal sentence representations

venue: In LREC,

authors:
Alexis Conneau
Douwe Kiela
Holger Schwenk
Loı̈c Barrault
Antoine Bordes

title: Supervised learning of universal sentence representations from natural language inference data

authors:
Seana Coulson
Cyma van Petten

title: A special role for the right hemisphere in metaphor comprehension? erp evidence from hemifield presentation

venue: Brain research,

authors:
Seana Coulson
Kara D. Federmeier
Cyma van Petten
Marta Kutas

title: Right hemisphere sensitivity to word- and sentence-level context: evidence from event-related brain potentials

venue: Journal of experimental psychology. Learning, memory, and cognition,

title: Learning classification with unlabeled data

venue: In NIPS, pp

authors:
Paramveer S. Dhillon
Dean P. Foster
Lyle H. Ungar

title: Multi-view learning of word embeddings via cca

venue: In NIPS,

authors:
William B. Dolan
Chris Quirk
Chris Brockett

title: Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources

venue: In COLING,

title: A synopsis of linguistic theory

authors:
Zhe Gan
Yunchen Pu
Ricardo Henao
Chunyuan Li
Xiaodong He
Lawrence Carin

title: Learning generic sentence representations using convolutional neural networks

authors:
Lushan Han
Abhay L Kashyap
Tim Finin
James Mayfield
Jonathan Weese

title: Umbc ebiquitycore: semantic textual similarity systems

venue: In Second Joint Conference on Lexical and Computational Semantics (* SEM),

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

venue: IEEE International Conference on Computer Vision (ICCV),

authors:
Felix Hill
Kyunghyun Cho
Anna Korhonen

title: Learning distributed representations of sentences from unlabelled data

venue: In HLT-NAACL,

authors:
Minqing Hu
Bing Liu

title: Mining and summarizing customer reviews

venue: In KDD,

authors:
Yacine Jernite
Samuel R. Bowman
David Sontag

title: Discourse-based objectives for fast unsupervised sentence representation learning

authors:
Diederik Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Jamie Ryan Kiros
Yukun Zhu
Ruslan Salakhutdinov
Richard S. Zemel
Raquel Urtasun
Antonio Torralba
Sanja Fidler

title: Skip-thought vectors

authors:
Quoc V. Le
Tomas Mikolov

title: Distributed representations of sentences and documents

authors:
Omer Levy
Yoav Goldberg
Ido Dagan

title: Improving distributional similarity with lessons learned from word embeddings

authors:
Jiwei Li
Eduard H. Hovy

title: A model of coherence based on distributed sentence representation

authors:
Xin Li
Dan Roth

title: Learning question classifiers

authors:
Lajanugen Logeswaran
Honglak Lee

title: An efficient framework for learning sentence representations

venue: In ICLR,

authors:
Marco Marelli
Stefano Menini
Marco Baroni
Luisa Bentivogli
Raffaella Bernardi
Roberto Zamparelli

title: A sick cure for the evaluation of compositional distributional semantic models

authors:
Julian J. McAuley
Christopher Targett
Qinfeng Shi
Anton van den Hengel

title: Image-based recommendations on styles and substitutes

venue: In SIGIR,

authors:
Bryan McCann
James Bradbury
Caiming Xiong
Richard Socher

title: Learned in translation: Contextualized word vectors

authors:
Tomas Mikolov
Ilya Sutskever
Kai Chen
Gregory S. Corrado
Jeffrey Dean

title: Distributed representations of words and phrases and their compositionality

authors:
Tomas Mikolov
Edouard Grave
Piotr Bojanowski
Christian Puhrsch
Armand Joulin

title: Advances in pre-training distributed word

venue: representations. CoRR,

authors:
RV Mises
Hilda Pollaczek-Geiringer

title: Praktische verfahren der gleichungsauflösung

venue: ZAMMJournal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik,

authors:
Allen Nie
Erin D. Bennett
Noah D. Goodman

title: Dissent: Sentence representation learning from explicit discourse

venue: relations. CoRR,

authors:
Bo Pang
Lillian Lee

title: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts

venue: In ACL,

title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

authors:
Razvan Pascanu
Tomas Mikolov
Yoshua Bengio

title: On the difficulty of training recurrent neural networks

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

venue: NIPS-W,

authors:
Jeffrey Pennington
Richard Socher
Christopher D. Manning

title: Glove: Global vectors for word representation

authors:
Dinghan Shen
Guoyin Wang
Wenlin Wang
Martin Renqiang Min
Qinliang Su
Yizhe Zhang
Chunyuan Li
Ricardo Henao
Lawrence Carin

title: Baseline needs more love: On simple wordembedding-based models and associated pooling mechanisms

authors:
Richard Socher
Alex Perelygin
Jean Wu
Jason Chuang
Christopher D. Manning
Andrew Ng
Christopher Potts

title: Recursive deep models for semantic compositionality over a sentiment treebank

authors:
Sandeep Subramanian
Adam Trischler
Yoshua Bengio
Christopher J Pal

title: Learning general purpose distributed sentence representations via large scale multi-task learning

venue: arXiv preprint arXiv:1804.00079,

authors:
Kai Sheng Tai
Richard Socher
Christopher D. Manning

title: Improved semantic representations from tree-structured long short-term memory networks

authors:
Shuai Tang
Hailin Jin
Chen Fang
Zhaowen Wang
Virginia R. de Sa

title: Speeding up contextbased sentence representation learning with non-autoregressive convolutional decoding

venue: In Rep4NLP@ACL,

authors:
Peter D. Turney
Patrick Pantel

title: From frequency to meaning: Vector space models of semantics

venue: J. Artif. Intell. Res.,

authors:
Weiran Wang
Raman Arora
Karen Livescu
Jeff A. Bilmes

title: On deep multi-view representation learning

authors:
Janyce Wiebe
Theresa Wilson
Claire Cardie

title: Annotating expressions of opinions and emotions in language

venue: Language Resources and Evaluation,

authors:
John Wieting
Kevin Gimpel

title: Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations

authors:
John Wieting
Mohit Bansal
Kevin Gimpel
Karen Livescu

title: From paraphrase database to compositional paraphrase model and back

authors:
Adina Williams
Nikita Nangia
Samuel R. Bowman

title: A broad-coverage challenge corpus for sentence understanding through inference

authors:
Zichao Yang
Diyi Yang
Chris Dyer
Xiaodong He
Alexander J. Smola
Eduard H. Hovy

title: Hierarchical attention networks for document classification

authors:
Han Zhao
Zhengdong Lu
Pascal Poupart

title: Self-adaptive hierarchical sentence model

venue: In IJCAI,

authors:
Yukun Zhu
Ryan Kiros
Richard S. Zemel
Ruslan Salakhutdinov
Raquel Urtasun
Antonio Torralba
Sanja Fidler

title: Aligning books and movies: Towards story-like visual explanations by watching movies and reading

heading: 1 INTRODUCTION

text:
Multi-view learning methods provide the ability to extract information from different views of the data and enable self-supervised learning of useful features for future prediction when annotated data is not available (de Sa, 1993). Minimising the disagreement among multiple views helps the model to learn rich feature representations of the data and, also after learning, the ensemble of the feature vectors from multiple views can provide an even stronger generalisation ability.
Distributional hypothesis (Harris, 1954) noted that words that occur in similar contexts tend to have similar meaning (Turney & Pantel, 2010), and distributional similarity (Firth, 1957) consolidated this idea by stating that the meaning of a word can be determined by the company it has. The hypothesis has been widely used in machine learning community to learn vector representations of human languages. Models built upon distributional similarity don’t explicitly require humanannotated training data; the supervision comes from the semantic continuity of the language data.
Large quantities of annotated data are usually hard and costly to obtain, thus it is important to study unsupervised and self-supervised learning. Our goal is to propose learning algorithms built upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelled data. We draw inspiration from the lateralisation and asymmetry in information processing of the two hemispheres of the human brain where, for most adults, sequential processing dominates the left hemisphere, and the right hemisphere has a focus on parallel processing (Bryden, 2012), but both hemispheres have been shown to have roles in literal and non-literal language comprehension (Coulson et al., 2005; Coulson & van Petten, 2007).
Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models, which have been widely applied in sentiment analysis tasks (Yang et al., 2016), and the linear/loglinear models, which have excelled at capturing attributional similarities of words and sentences (Arora et al., 2016; 2017; Hill et al., 2016; Turney & Pantel, 2010) for learning sentence representations. Previous work on unsupervised sentence representation learning based on distributional hypothesis can be roughly categorised into two types:
Generative objective: These models generally follow the encoder-decoder structure. The encoder learns to produce a vector representation for the current input, and the decoder learns to generate sentences in the adjacent context given the produced vector (Kiros et al., 2015; Hill et al., 2016; Gan et al., 2017; Tang et al., 2018). The idea is straightforward, yet its scalability for very large corpora is hindered by the slow decoding process that dominates training time, and also the decoder in each model is discarded after learning as the quality of generated sequences is not the main concern, which is a waste of parameters and learning effort.
Our first multi-view framework has a generative objective and uses an RNN as the encoder and an invertible linear projection as the decoder. The training time is drastically reduced as the decoder is simple, and the decoder is also utilised after learning. A regularisation is applied on the linear decoder to enforce invertibility, so that after learning, the inverse of the decoder can be applied as a linear encoder in addition to the RNN encoder.
Discriminative Objective: In these models, a classifier is learnt on top of the encoders to distinguish adjacent sentences from those that are not (Li & Hovy, 2014; Jernite et al., 2017; Nie et al., 2017; Logeswaran & Lee, 2018); these models make a prediction using a predefined differentiable similarity function on the representations of the input sentence pairs or triplets.
Our second multi-view framework has a discriminative objective and uses an RNN encoder and a linear encoder; it learns to maximise agreement among adjacent sentences. Compared to earlier work on multi-view learning (de Sa, 1993; Dhillon et al., 2011; Wang et al., 2015) that takes data from various sources or splits data into disjoint populations, our framework processes the exact same data in two distinctive ways. The two distinctive information processing views tend to encode different aspects of an input sentence; forcing agreement/alignment between these views encourages each view to be a better representation, and is beneficial to the future use of the learnt representations.
Our contribution is threefold:
• Two multi-view frameworks for learning sentence representations are proposed, in which one framework uses a generative objective and the other one adopts a discriminative objective. Two encoding functions, an RNN and a linear model, are learnt in both frameworks.
• The results show that in both frameworks, aligning representations from two views gives improved performance of each individual view on all evaluation tasks compared to their single-view trained counterparts, and furthermore ensures that the ensemble of two views provides even better results than each improved view alone.
• Models trained under our proposed frameworks achieve good performance on the unsupervised tasks, and overall outperform existing unsupervised learning models, and armed with various pooling functions, they also show solid results on supervised tasks, which are either comparable to or better than those of the best unsupervised transfer model.
It is shown (Hill et al., 2016) that the consistency between supervised and unsupervised evaluation tasks is much lower than that within either supervised or unsupervised evaluation tasks alone and that a model that performs well on supervised evaluation tasks may fail on unsupervised tasks. It is subsequently showed (Conneau et al., 2017; Subramanian et al., 2018) that, with large-scale labelled training corpora, the resulting representations of the sentences from the trained model excel in both supervised and unsupervised tasks, while the labelling process is costly. Our model is able to achieve good results on both groups of tasks without labelled information.

heading: 2 MODEL ARCHITECTURE

text:
Our goal is to marry RNN-based sentence encoder and the avg-on-word-vectors sentence encoder into multi-view frameworks with simple objectives. The motivation for the idea is that, RNN-based encoders process the sentences sequentially, and are able to capture complex syntactic interactions, while the avg-on-word-vectors encoder has been shown to be good at capturing the coarse meaning of a sentence which could be useful for finding paradigmatic parallels (Turney & Pantel, 2010).
We present two multi-view frameworks, each of which learns two different sentence encoders; after learning, the vectors produced from two encoders of the same input sentence are used to compose the sentence representation. The details of our learning frameworks are described as follows:

heading: 2.1 ENCODERS

text: In our multi-view frameworks, we first introduce two encoders that, after learning, can be used to build sentence representations. One encoder is a bi-directional Gated Recurrent Unit (Chung et al., 2014) f(s;φ), where s is the input sentence and φ is the parameter vector in the GRU. During learning, only hidden state at the last time step is sent to the next stage in learning. The other encoder is a linear avg-on-word-vectors model g(s;W ), which basically transforms word vectors in a sentence by a learnable weight matrixW and outputs an averaged vector.

heading: 2.2 GENERATIVE OBJECTIVE

text:
Given the finding (Tang et al., 2018) that neither an autoregressive nor an RNN decoder is necessary for learning sentence representations that excel on downstream tasks, our learning framework only learns to predict words in the next sentence. The framework has an RNN encoder f , and a linear decoder h. Given an input sentence si, the encoder produces a vector z f i = f(si;φ), and the decoder h projects the vector to xi = h(z f i ;U) = Uz f i , which has the same dimension as the word vectors vw. Negative sampling is applied to calculate the likelihood of generating the j-th word in the (i+ 1)-th sentence, shown in Eq. 1.
logP (wj |si) = log σ(x>i vwj ) + ∑K k=1 Ewk∼Pe(w) log σ(−x>i vwk) (1)
where vwk are pretrained word vectors for wk, the empirical distribution Pe(w) is the unigram distribution raised to power 0.75 (Mikolov et al., 2013), and K is the number of negative samples. The learning objective is to maximise the likelihood for words in all sentences in the training corpus.
Ideally, the inverse of h should be easy to compute so that during testing we can set g = h−1. As h is a linear projection, the simplest situation is when U is an orthogonal matrix and its inverse is equal to its transpose. Often, as the dimensionality of vector zfi doesn’t necessarily need to match that of word vectors vw, U is not a square matrix1. To enforce invertibility on U , a row-wise orthonormal regularisation on U is applied during training, which leads to UU> = I , where I is the identity matrix, thus the inverse function is simply h−1(x) = U>x, which is easily computed. The regularisation formula is ||UU> − I||F , where || · ||F is the Frobenius norm. Specifically, the update rule (Cissé et al., 2017) for the regularisation is:
U := (1 + β)U − β(UU>)U (2)
where β is set to 0.01. After learning, we set W = U>, then the inverse of the decoder h becomes the encoder g. Compared to prior work with generative objective, our framework reuses the decoding function rather than ignoring it for building sentence representations after learning, thus information encoded in the decoder is also utilised.

heading: 2.3 DISCRIMINATIVE OBJECTIVE

text:
Our multi-view framework with discriminative objective learns to maximise the agreement between the representations of a sentence pair across two views if one sentence in the pair is in the neighbourhood of the other one. An RNN encoder f(s;φ) and a linear avg-on-word-vectors g(s;W ) produce a vector representation zfi and z g i for i-th sentence respectively. The agreement between two views of a sentence pair (si, sj) is defined as aij = aji = cos(z f i , z g j ) + cos(z g i , z f j ). The training objective is to minimise the loss function:
L(φ,W ) = − ∑ |i−j|≤c log pij , where pij = eaij/τ∑i+N/2−1 n=i−N/2 e ain/τ
(3)
where τ is the trainable temperature term, which is essential for exaggerating the difference between adjacent sentences and those that are not. The neighbourhood/context window c, and the batch size N are hyperparameters.
1As often the dimension of sentence vectors are equal to or large than that of word vectors, U has more columns than rows. If it is not the case, then regulariser becomes ||U>U − I||F .
The choice of cosine similarity based loss is based on the observations (Turney & Pantel, 2010) that, of word vectors derived from distributional similarity, vector length tends to correlate with frequency of words, thus angular distance captures more important meaning-related information. Also, since our model is unsupervised/self-supervised, whatever similarity there is between neighbouring sentences is what is learnt as important for meaning.

heading: 2.4 POSTPROCESSING

text:
The postprocessing step (Arora et al., 2017), which removes the top principal component of a batch of representations, is applied on produced representations from f and g respectively after learning with a final l2 normalisation.
In addition, in our multi-view framework with discriminative objective, in order to reduce the discrepancy between training and testing, the top principal component is estimated by the power iteration method (Mises & Pollaczek-Geiringer, 1929) and removed during learning.

heading: 3 EXPERIMENTAL DESIGN

text:
Three unlabelled corpora from different genres are used in our experiments, including BookCorpus (Zhu et al., 2015), UMBC News (Han et al., 2013) and Amazon Book Review2(McAuley et al., 2015); six models are trained separately on each of three corpora with each of two objectives. The summary statistics of the three corpora can be found in Table 1. Adam optimiser (Kingma & Ba, 2014) and gradient clipping (Pascanu et al., 2013) are applied for stable training. Pretrained word vectors, fastText (Bojanowski et al., 2017), are used in our frameworks and fixed during learning.
All of our experiments including training and testing are done in PyTorch (Paszke et al., 2017). The modified SentEval (Conneau & Kiela, 2018) package with the step that removes the first principal component is used to evaluate our models on the downstream tasks. Hyperparameters, including negative samples K in the framework with generative objective, context window c in the one with discriminative objective, are tuned only on the averaged performance on STS14 of the model trained on the BookCorpus; STS14/G1 and STS14/D1 results are thus marked with a ? in Table 3 and Table 4 to indicate possible overfitting on that dataset/model only. Batch size N and dimension d in both frameworks are set to be the same for fair comparison. Hyperparameters are summarised in supplementary material.
2Largest subset of Amazon Review.

heading: 3.1 UNSUPERVISED EVALUATION - TEXTUAL SIMILARITY TASKS

text:
Representation: For a given sentence input s with M words, suggested by (Pennington et al., 2014; Levy et al., 2015), the representation is calculated as z = ( ẑf + ẑg ) /2, where ẑ refers to the post-processed and normalised vector, and is mentioned in Table 2.
Tasks: The unsupervised tasks include five tasks from SemEval Semantic Textual Similarity (STS) in 2012-2016 (Agirre et al., 2012; 2013; 2014; 2015; 2016) and the SemEval2014 Semantic Relatedness task (SICK-R) (Marelli et al., 2014).
Comparison: We compare our models with: • Unsupervised learning: We selected models with strong results from related work, including fastText, fastText+WR. • Semi-supervised learning: The word vectors are pretrained on each task (Wieting et al., 2015) without label information, and word vectors are averaged to serve as the vector representation for a given sentence (Arora et al., 2017). • Supervised learning: ParaNMT (Wieting & Gimpel, 2018) is included as a supervised learning method as the data collection requires a neural machine translation system trained in supervised fashion. The InferSent3 (Conneau et al., 2017) trained on SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2017) is included as well.
The results are presented in Table 3. Since the performance of FastSent (Hill et al., 2016) and QT (Logeswaran & Lee, 2018) were only evaluated on STS14, we compare to their results in Table 4.
All six models trained with our learning frameworks outperform other unsupervised and semisupervised learning methods, and the model trained on the UMBC News Corpus with discriminative objective gives the best performance likely because the STS tasks contain multiple news- and headlines-related datasets which is well matched by the domain of the UMBC News Corpus.

heading: 3.2 SUPERVISED EVALUATION

text:
The evaluation on these tasks involves learning a linear model on top of the learnt sentence representations produced by the model. Since a linear model is capable of selecting the most relevant dimensions in the feature vectors to make predictions, it is preferred to concatenate various types of representations to form a richer, and possibly more redundant feature vector, which allows the
3The released InferSent (Conneau et al., 2017) model is evaluated with the postprocessing step.
linear model to explore the combination of different aspects of encoding functions to provide better results.
Representation: Inspired by prior work (McCann et al., 2017; Shen et al., 2018), the representation zf is calculated by concatenating the outputs from the global mean-, max- and min-pooling on top of the hidden states H , and the last hidden state, and zg is calculated with three pooling functions as well. The post-processing and the normalisation step is applied individually. These two representations are concatenated to form a final sentence representation. Table 2 presents the details.
Tasks: Semantic relatedness (SICK) (Marelli et al., 2014), paraphrase detection (MRPC) (Dolan et al., 2004), question-type classification (TREC) (Li & Roth, 2002), movie review sentiment (MR) (Pang & Lee, 2005), Stanford Sentiment Treebank (SST) (Socher et al., 2013), customer product reviews (CR) (Hu & Liu, 2004), subjectivity/objectivity classification (SUBJ) (Pang & Lee, 2004), opinion polarity (MPQA) (Wiebe et al., 2005). The results are presented in Table 5.
Comparison: Our results as well as related results of supervised task-dependent training models, supervised learning models, and unsupervised learning models are presented in Table 5. Note that, for fair comparison, we collect the results of the best single model of MC-QT (Logeswaran & Lee, 2018) trained on BookCorpus.
Six models trained with our learning frameworks either outperform other existing methods, or achieve similar results on some tasks. The model trained on the Amazon Book Review gives the best performance on sentiment analysis tasks, since the corpus conveys strong sentiment information.

heading: 4 DISCUSSION

text: In both frameworks, RNN encoder and linear encoder perform well on all tasks, and generative objective and discriminative objective give similar performance.

heading: 4.1 GENERATIVE OBJECTIVE: REGULARISATION ON INVERTIBILITY

text:
The orthonormal regularisation applied on the linear decoder to enforce invertibility in our multiview framework encourages the vector representations produced by f and those by h−1, which is g in testing, to agree/align with each other. A direct comparison is to train our multi-view framework without the invertible constraint, and still directly use U> as an additional encoder in testing. The results of our framework with and without the invertible constraint are presented in Table 6.
The ensemble method of two views, f and g, on unsupervised evaluation tasks (STS12-16 and SICK14) is averaging, which benefits from aligning representations from f and g by applying invertible constraint, and the RNN encoder f gets improved on unsupervised tasks by learning to align with g. On supervised evaluation tasks, as the ensemble method is concatenation and a linear model is applied on top of the concatenated representations, as long as the encoders in two views process sentences distinctively, the linear classifier is capable of picking relevant feature dimensions from both views to make good predictions, thus there is no significant difference between our multi-view framework with and without invertible constraint.

heading: 4.2 DISCRIMINATIVE OBJECTIVE: MULTI-VIEW VS. SINGLE-VIEW

text:
In order to determine if the multi-view framework with two different views/encoding functions is helping the learning, we compare our framework with discriminative objective to other reasonable variants, including the multi-view model with two functions of the same type but parametrised independently, either two f -s or two g-s, and the single-view model with only one f or g. Table 6 presents the results of the models trained on UMBC News Corpus.
As specifically emphasised in previous work (Hill et al., 2016), linear/log-linear models, which include g in our model, produce better representations for unsupervised evaluation tasks than RNNbased models do. This can also be observed in Table 6 as well, where g consistently provides better results on unsupervised tasks than f . In addition, as expected, multi-view learning with f and g, improves the resulting performance of f on unsupervised tasks, also improves the resulting g on supervised evaluation tasks.
Provided the results of models with generative and discriminative objective in Table 6, we confidently show that, in our multi-view frameworks with f and g, the two encoding functions improve each other’s view.

heading: 4.3 ENSEMBLE IN MULTI-VIEW FRAMEWORKS

text:
In general, aligning the representations generated from two distinct encoding functions ensures that the ensemble of them performs better. The two encoding functions f and g encode the input sentence with emphasis on different aspects, and the subsequently trained linear model for each of the supervised downstream tasks benefits from this diversity leading to better predictions. However, on unsupervised evaluation tasks, simply averaging representations from two views without aligning them during learning leads to poor performance and it is worse than g (linear) encoding function solely. Our multi-view frameworks ensure that the ensemble of two views provides better performance on both supervised and unsupervised evaluation tasks.
Compared with the ensemble of two multi-view models, each with two encoding functions of the same type, our multi-view framework with f and g provides slightly better results on unsupervised tasks, and similar results on supervised evaluation tasks, while our model has much higher training efficiency. Compared with the ensemble of two single-view models, each with only one encoding function, the matching between f and g in our multi-view model produces better results.

heading: 5 CONCLUSION

text:
We proposed multi-view sentence representation learning frameworks with generative and discriminative objectives; each framework combines an RNN-based encoder and an average-on-wordvectors linear encoder and can be efficiently trained within a few hours on a large unlabelled corpus. The experiments were conducted on three large unlabelled corpora, and meaningful comparisons were made to demonstrate the generalisation ability and transferability of our learning frameworks and consolidate our claim. The produced sentence representations outperform existing unsupervised transfer methods on unsupervised evaluation tasks, and match the performance of the best unsupervised model on supervised evaluation tasks.
Our experimental results support the finding (Hill et al., 2016) that linear/log-linear models (g in our frameworks) tend to work better on the unsupervised tasks, while RNN-based models (f in our frameworks) generally perform better on the supervised tasks. As presented in our experiments, multi-view learning helps align f and g to produce better individual representations than when they are learned separately. In addition, the ensemble of both views leveraged the advantages of both, and provides rich semantic information of the input sentence. Future work should explore the impact of having various encoding architectures and learning under the multi-view framework.
Our multi-view learning frameworks were inspired by the asymmetric information processing in the two hemispheres of the human brain, in which the left hemisphere is thought to emphasise sequential processing and the right one more parallel processing (Bryden, 2012). Our experimental results raise an intriguing hypothesis about how these two types of information processing may complementarily help learning.

heading: 1 EVALUATION TASKS

text: The details, including size of each dataset and number of classes, about the evaluation tasks are presented below1.

heading: 2 POWER ITERATION

text:
The Power Iteration was proposed in Mises & Pollaczek-Geiringer (1929), and it is an efficient algorithm for estimating the top eigenvector of a given covariance matrix. Here, it is used to estimate the top principal component from the representations produced from f and g separately. We omit the superscription here, since the same step is applied to both f and g.
Suppose there is a batch of representations Z = [z1, z2 ..., zN ] ∈ R2d×N from either f or g, the Power Iteration method is applied here to estimate the top eigenvector of the covariance matrix2: C = ZZ>, and it is described in Algorithm 1:
Algorithm 1 Estimating the First Principal Component (Mises & Pollaczek-Geiringer, 1929)
Input: Covariance matrix C ∈ R2d×2d, number of iterations T Output: First principal component u ∈ R2d
1: Initialise a unit length vector u ∈ R2d 2: for t← 1, T do 3: u← Cu, 4: u← u
||u||
In our experiments, T is set to be 5.
1Provided by https://github.com/facebookresearch/SentEval 2In practice, often N is less than 2d, thus we estimate the top eigenvector of Z>Z ∈ RN×N .

heading: 3 TRAINING & MODEL DETAILS

text:
The hyperparameters we need to tune include the batch size N , the dimension of the GRU encoder d, and the context window c, and the number of negative samples K. The results we presented in this paper is based on the model trained with N = 512, d = 1024. Specifically, in discriminative objective, the context window is set c = 3, and in generative objective, the number of negative samples is set K = 5. It takes up to 8GB on a GTX 1080Ti GPU.
The initial learning rate is 5× 10−4, and we didn’t anneal the learning rate through the training. All weights in the model are initialised using the method proposed in He et al. (2015), and all gates in the bi-GRU are initialised to 1, and all biases in the single-layer neural network are zeroed before training. The word vectors are fixed to be those in the FastText (Bojanowski et al., 2017), and we don’t finetune them. Words that are not in the FastText’s vocabulary are fixed to 0 vectors through training. The temperature term is initialised as 1, and is tuned by the gradient descent during training.
The temperature term is used to convert the agreement aij to a probability distribution pij in Eq. 1 in the main paper. In our experiments, τ is a trainable parameter initialised to 1 that decreased consistently through training. Another model trained with fixed τ set to the final value performed similarly.

heading: 4 EFFECT OF POST-PROCESSING STEP

heading: 5 COMBINING BOTH GENERATIVE AND DISCRIMINATIVE OBJECTIVE IN OUR MULTI-VIEW FRAMEWORK

text:
Models with both generative and discriminative objectives are trained to see if further improvement can be provided by combining an RNN encoder, an inverse of a linear decoder in the generative objective and a linear encoder in the discriminative objective. The results of models trained on BookCorpus and UMBC News are presented in Table 3.
As presented in the table, no further improvement against models with only one objective is shown. In our understanding, the inverse of the linear decoder in generative objective behaves similarly to the linear encoder in the discriminative objective, which is presented in Table 6 in the main paper. Therefore, combining two objectives doesn’t perform better than only one of them.

heading: 6 NUMBER OF PARAMETERS

text:
The number of parameters of each of the selected models is:
1. Ours: ≈ 8.8M 2. Quick-thought (Logeswaran & Lee, 2018): ≈ 19.8M 3. Skip-thought (Kiros et al., 2015): ≈ 57.7M