Neural sequence-to-sequence models are currently the dominant approach in several natural language processing tasks, but require large parallel corpora. We present a sequence-to-sequence-to-sequence autoencoder ( SEQ 3 ), consisting of two chained encoder-decoder pairs, with words used as a sequence of discrete latent variables. We apply the proposed model to unsupervised abstractive sentence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle sequence is the compressed sentence. Constraining the length of the latent word sequences forces the model to distill important information from the input. A pretrained language model, acting as a prior over the latent sequences, encourages the compressed sentences to be human-readable. Continuous relaxations enable us to sample from categorical distributions, allowing gradient-based optimization, unlike alternatives that rely on reinforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets. Neural sequence-to-sequence models ( SEQ 2 SEQ ) perform impressively well in several natural language processing tasks, such as machine translation (Sutskever et al., 2014; Bahdanau et al., 2015) or syntactic constituency parsing (Vinyals et al., 2015). However, they require massive parallel training datasets (Koehn and Knowles, 2017). Consequently there has been extensive work on utilizing non-parallel corpora to boost the performance of SEQ 2 SEQ models (Sennrich et al., 2016; Gulcehre et al., 2015), mostly in neural machine translation where models that require absolutely no parallel corpora have also been pro1 ,  2 ,  ,   1 ,  2 ,  ,    1 ,  2 ,  ,   Compressor (encoder-decoder) Reconstructor ( encoder-decoder ) Reconstruction Loss LM Prior Loss Topic Loss Figure 1: Overview of the proposed SEQ 3 autoencoder. posed (Artetxe et al., 2018; Lample et al., 2018b). Unsupervised (or semi-supervised) SEQ 2 SEQ models have also been proposed for summarization tasks with no (or small) parallel text-summary sets, including unsupervised sentence compression. Current models, however, barely reach lead-N baselines (Fevry and Phang, 2018; Wang and Lee, 2018), and/or are non-differentiable (Wang and Lee, 2018; Miao and Blunsom, 2016), thus relying on reinforcement learning, which is unstable and inefficient. By contrast, we propose a sequence-to-sequence-to-sequence autoencoder, dubbed SEQ 3 , that can be trained end-to-end via gradient-based optimization. SEQ 3 employs differentiable approximations for sampling from categorical distributions (Maddison et al., 2017; Jang et al., 2017), which have been shown to outperform reinforcement learning ( Havrylov and Titov, 2017). Therefore it is a generic framework which can be easily extended to other tasks, e.g., machine translation and semantic parsing via task-specific losses. In this work, as a first step, we apply SEQ 3 to unsupervised abstractive sentence compression. SEQ 3 ( x 2) comprises two attentional encoder-decoder (Bahdanau et al., 2015) pairs (Fig. 1): a compressor C and a reconstructor R . C ( x 2.1) receives an input text x =  x 1 ; : : : ; x N  of N words, and generates a summary y =  y 1 ; : : : ; y M  of M words ( M < N ), y being a latent variable. R and C communicate only through the discrete words of the summary y ( x 2.2). R ( x 2.3) produces a sequence ^ x =  ^ x 1 ; : : : ; ^ x N  of N words from y , try 0  1  2  1  1  1  2       2      0  1  1  1  1  1    1  2       Topic Loss   1  2      Compressor Reconstructor LM prior Loss    0  1  0  1   1  2  1   1   Figure 2: More detailed illustration of SEQ 3 . ing to minimize a reconstruction loss LR = ( x ; ^ x ) ( x 2.5). A pretrained language model acts as a prior on y , introducing an additional loss LP ( x ; y ) that encourages SEQ 3 to produce human-readable summaries. A third loss LT ( x ; y ) rewards summaries y with similar topic-indicating words as x . Experiments ( x 3) on the Gigaword sentence compression dataset (Rush et al., 2015) and the DUC -2003 and DUC -2004 shared tasks (Over et al., 2007) produce promising results. Our contributions are: (1) a fully differentiable sequence-to-sequence-to-sequence ( SEQ 3 ) autoencoder that can be trained without parallel data via gradient optimization; (2) an application of SEQ 3 to unsupervised abstractive sentence compression, with additional task-specific loss functions; (3) state of the art performance in unsupervised abstractive sentence compression. This work is a step towards exploring the potential of SEQ 3 in other tasks, such as machine translation. The bottom left part of Fig. 2 illustrates the internals of the compressor C . An embedding layer projects the source sequence x to the word embeddings e s =  e s 1 ; : : : ; e sN  , which are then encoded by a bidirectional RNN , producing h s =  h s 1 ; : : : ; h sN  . Each h st is the concatenation of the corresponding left-to-right and right-to-left states (outputs in LSTM s) of the biRNN . h st = [ (cid:0)(cid:0)! RNN s ( e st ; (cid:0)! h st (cid:0) 1 ); (cid:0)(cid:0) RNN s ( e st ; (cid:0) h st +1 )] To generate the summary y , we employ the attentional RNN decoder of Luong et al. (2015), with their global attention and input feeding. Concretely, at each timestep ( t 2 f 1 ; : : : ; M g ) we compute a probability distribution a i over all the states h s 1 ; : : : ; h sN of the source encoder conditioned on the current state h ct of the compressor's decoder to produce a context vector c t . The matrix W a is learned. We obtain a probability distribution for y t over the vocabulary V by combining c t and the current state h ct of the decoder. o ct = tanh ( W o [ c t ; h ct ] + b o ) (1) u ct = W v o ct + b v (2) p ( y t j y <t ; x ) = softmax ( u ct ) (3) W o ; b o ; W v ; b v are learned. c t is also used when updating the state h ct of the decoder, along with the embedding e ct of y t and a countdown argument M (cid:0) t (scaled by a learnable w d ) indicating the number of the remaining words of the summary ( Fevry and Phang, 2018; Kikuchi et al., 2016). For each input x =  x 1 ; : : : ; x N  , we obtain a target length M for the summary y =  y 1 ; : : : ; y M  by sampling (and rounding) from a uniform distribution U ( (cid:11) N ; (cid:12) N ) ; (cid:11); (cid:12) are hyper-parameters ( (cid:11) < (cid:12) < 1 ); we set M = 5 , if the sampled M is smaller. Sampling M , instead of using a static compression ratio, allows us to train a model capable of producing summaries with varying (e.g., user-specified) compression ratios. Controlling the output length in encoder-decoder architectures has been explored in machine translation (Kikuchi et al., 2016) and summarization (Fan et al., 2018). To generate the summary, we need to sample its words y t from the categorical distributions p ( y t j y <t ; x ) , which is a non-differentiable process. Soft-Argmax Instead of sampling y t , a simple workaround during training is to pass as input to the next timestep of C 's decoder and to the corresponding timestep of R 's encoder a weighted sum of all the vocabulary's ( V ) word embeddings, using a peaked softmax function (Goyal et al., 2017): e ct = j V j  i e ( w i ) softmax ( u ct =(cid:28) ) (5) where u ct is the unnormalized score in Eq. 2 (i.e., the logit) of each word w i and (cid:28) 2 (0 ; 1 ) is the temperature. As (cid:28) ! 0 most of the probability mass in Eq. 5 goes to the most probable word, hence the operation approaches the arg max . Gumbel-Softmax We still want to be able to perform sampling, though, as it has the benefit of adding stochasticity and facilitating exploration of the parameter space. Hence, we use the Gumbel-Softmax ( GS ) reparametrization trick (Maddison et al., 2017; Jang et al., 2017) as a low variance approximation of sampling from categorical distributions. Sampling a specific word y t from the softmax (Eq. 3) is equivalent to adding (element-wise) to the logits an independent noise sample (cid:24) from the Gumbel distribution 1 and taking the arg max : y t (cid:24) softmax ( u ct ) $ y t = arg max( u ct + (cid:24) ) (6) Therefore, using the GS trick, Eq. 5 becomes: ~ e ct = j V j  i e ( w i ) softmax (( u ct + (cid:24) ) =(cid:28) ) (7) Straight-Through Both relaxations lead to mixtures of embeddings, which do not correspond to actual words. Even though this enables the compressor to communicate with the reconstructor using continuous values, thus fully utilizing the available embedding space, ultimately our aim is to constrain them to communicate using only natural language. In addition, an unwanted discrepancy is created between training (continuous embeddings) and test time (discrete embeddings). We alleviate these problems with the Straight-Through estimator ( ST ) (Bengio et al., 2013). Specifically, in the forward pass of training we dis-cretize ~ e ct by using the arg max (Eq. 6), whereas in the backward pass we compute the gradients using the GS (Eq. 7). This is a biased estimator due 1 (cid:24) i = (cid:0) log( (cid:0) log( x i )) ; x i (cid:24) U (0 ; 1) to the mismatch between the forward and backward passes, but works well in practice. ST GS reportedly outperforms scheduled sampling (Goyal et al., 2017) and converges faster than reinforcement learning (Havrylov and Titov, 2017). The reconstructor (upper right of Fig. 2) works like the compressor, but its encoder operates on the embeddings e c 1 ; : : : ; e cM of the words y 1 ; : : : ; y M of the summary (exact embeddings of the sampled words y t in the forward pass, approximate differentiable embeddings in the backward pass). We initialize the hidden state of each decoder using a transformation of the concatenation [ (cid:0)! h sN ; (cid:0) h s 1 ] of the last hidden states (from the two directions) of its bidirectional encoder and a length vector, following Mallinson et al. (2018). The length vector for the decoder of the compressor C consists of the target summary length M , scaled by a learnable parameter w v , and the compression ratio MN . Reconstruction Loss LR ( x ; ^ x ) is the (negative) log-likelihood assigned by the (decoder of) R to the input (correctly reconstructed) words x =  x 1 ; : : : ; x N  , where p R is the distribution of R . We do not expect LR ( x ; ^ x ) to decrease to zero, as there is information loss through the compression. However, we expect it to drive the compressor to produce such sentences that will increase the likelihood of the target words in the reconstruction. LM Prior Loss To ensure that the summaries y are readable, we pretrain an RNN language model (see Appendix) on the source texts of the full training set. We compute the Kullback-Leibler divergence DKL between the probability distributions of the (decoder of) the compressor ( p ( y t j y <t ; x ) , Eq. 3) and the language model ( p LM ( y t j y <t ; x ) ). Similar priors have been used in sentence compression (Miao and Blunsom, 2016) and agent communication (Havrylov and Titov, 2017). Topic Loss Words with high TF-IDF scores are indicative of the topic of a text (Ramos et al., 2003; Erkan and Radev, 2004). To encourage the compressor to preserve in the summary y the topic-indicating words of the input x , we compute the TF-IDF -weighted average v x of the word embeddings of x and the average v y of the word embeddings of y and use their cosine distance as an additional loss LT = 1 (cid:0) cos( v x ; v y ) . N  si MM Length Penalty A fourth loss LL (not shown in Fig. 1) helps the (decoder of the) compressor to predict the end-of-sequence ( EOS ) token at the target summary length M . LL is the cross-entropy between the distributions p ( y t j y <t ; x ) (Eq. 3) of the compressor at t = M + 1 and onward, with the one-hot distribution of the EOS token. Parameter Sharing We tie the weights of layers encoding similar information, to reduce the number of trainable parameters. First, we use a shared embedding layer for the encoders and decoders, initialized with 100-dimensional GloVe embeddings (Pennington et al., 2014). Additionally, we tie the shared embedding layer with the output layers of both decoders (Press and Wolf, 2017; Inan et al., 2017). Finally, we tie the encoders of the compressor and reconstructor (see Appendix). OOVs Out-of-vocabulary words are handled as in Fevry and Phang (2018) (see Appendix). Datasets We train SEQ 3 on the Gigaword sentence compression dataset (Rush et al., 2015). 2 It consists of pairs, each containing the first sentence of a news article ( x ) and the article's headline ( y ), a total of 3.8M/189k/1951 train/dev/test pairs. We also test (without retraining) SEQ 3 on DUC -2003 and DUC -2004 shared tasks (Over et al., 2007), containing 624/500 news articles each, paired with 4 reference summaries capped at 75 bytes. 2 github.com/harvardnlp/sent-summary sentences (sources) of the training pairs from Gigaword to train SEQ 3 ; our model is never exposed to target headlines (summaries) during training or evaluation, i.e., it is completely unsupervised. Our code is publicly available. 3 We compare SEQ 3 to other unsupervised sentence compression models. We note that the extractive model of Miao and Blunsom (2016) relies on a pre-trained attention model using at least 500K parallel sentences, which is crucial to mitigate the inefficiency of sampling-based variational inference and REINFORCE . Therefore it is not comparable, as it is semi-supervised. The results of the extractive model of Fevry and Phang (2018) are also not comparable, as they were obtained on a different, not publicly available test set. We note, however, that they report that their system performs worse than the LEAD -8 baseline in ROUGE -2 and ROUGE-L on Gigaword. The only directly comparable unsupervised model is the abstractive Pretrained Generator' of Wang and Lee (2018). The version of Adversarial REINFORCE ' that Wang and Lee (2018) consider unsupervised is actually weakly supervised, since its discriminator was exposed to the summaries of the same sources the rest of the model was trained on. As baselines, we use LEAD -8 for Gigaword, which simply selects the first 8 words of the source, and PREFIX for DUC , which includes the first 75 bytes of the source article. We also compare to supervised abstractive sentence compression methods (Tables 1-3). Following previous work, we report the average F1 of ROUGE 1, ROUGE -2, ROUGE-L (Lin, 2004). We implemented SEQ 3 with LSTM s (see Appendix) and during inference we perform greedy-sampling. Results Table 1 reports the Gigaword results. SEQ 3 outperforms the unsupervised Pretrained Generator across all metrics by a large margin. It also surpasses LEAD -8. If we remove the LM prior, performance drops, esp. in ROUGE -2 and ROUGEL . This makes sense, since the pretrained LM rewards correct word order. We also tried removing the topic loss, but the model failed to converge and results were extremely poor (Table 1). Topic loss acts as a bootstrap mechanism, biasing the compressor to generate words that maintain the topic of the input text. This greatly reduces variance due to sampling in early stages of training, alleviating the need to pretrain individual 3 https://github.com/cbaziotis/seq3 Type Supervision Methods R-1 R-2 R-L Supervised 3.8M ABS (Rush et al., 2015) 29.55 11.32 26.42 SEASS (Zhou et al., 2017) 36.15 17.54 33.63 words-lvt5k-1sent (Nallapati et al., 2016) 36.4 17.7 33.71 Weakly supervised (3.8M) Adversarial REINFORCE (Wang and Lee, 2018) 28.11 9.97 25.41 Unsupervised 0 LEAD -8 (Baseline) 21.86 7.66 20.45 Pretrained Generator (Wang and Lee, 2018) 21.26 5.60 18.89 SEQ 3 (Full) 25.39 8.21 22.68 SEQ 3 w/o LM prior loss 24.48 6.68 21.79 SEQ 3 w/o TOPIC loss 3.89 0.1 3.75 Table 1: Average results on the (English) Gigaword dataset for abstractive sentence compression methods. components, unlike works that rely on reinforcement learning (Miao and Blunsom, 2016; Wang and Lee, 2018). Overall, both losses work in synergy, with the topic loss driving what and the LM prior loss driving how words should be included in the summary. SEQ 3 behaves similarly on DUC 2003 and DUC -2004 (Tables 2-3), although it was trained on Gigaword. In DUC -2003, however, it does not surpass the PREFIX baseline. Finally, Fig. 3 illustrates three randomly sampled outputs of SEQ 3 on Gigaword. In the first one, SEQ 3 copies several words esp. from the beginning of the input (hence the high ROUGE-L ) exhibiting extractive capabilities, though still being adequately abstractive (bold words denote para-phrases). In the second one, SEQ 3 showcases its true abstractive power by paraphrasing and compressing multi-word expressions to single content words more heavily, still without losing the overall meaning. In the last example, SEQ 3 progressively becomes ungrammatical though interestingly retaining some content words from the input. input: the american sailors who thwarted somali pirates flew home to the u.s. on wednesday but without their cap-tain , who was still aboard a navy destroyer after being rescued from the hijackers . gold : us sailors who thwarted pirate hijackers fly home SEQ 3 : the american sailors who foiled somali pirates flew home after crew hijacked . input: the central election commission -lrbcec -rrbon monday decided that taiwan will hold another election of national assembly members on may # . gold : national <unk> election scheduled for may SEQ 3 : the central election commission -lrbcec UNK announced elections . input: dave bassett resigned as manager of struggling english premier league side nottingham forest on saturday after they were knocked out of the f.a. cup in the third round , according to local reports on saturday . gold : forest manager bassett quits . SEQ 3 : dave bassett resigned as manager of struggling en-glish premier league side UNK forest on knocked round press Figure 3: Good/bad example summaries on Gigaword. hypothesize that since the reconstructor is autoregressive, i.e., each word is conditioned on the previous one, errors occurring early in the generated sequence have cascading effects. This inevitably encourages the compressor to select the first words of the input. A possible workaround might be to modify SEQ 3 so that the first encoder-decoder pair would turn the inputs to longer sequences, and the second encoder-decoder would compress them trying to reconstruct the original inputs. In future work, we plan to explore the potential of SEQ 3 in other tasks, such as unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018) and caption generation (Xu et al., 2015). We would like to thank Ryan McDonald for helpful discussions and feedback. This work has been partially supported by computational time granted from the Greek Research & Technology Network (GR-NET) in the National HPC facility ARIS. We thank NVIDIA for donating a TitanX GPU.
abstain method objective abstain abstain method abstain abstain abstain other abstain abstain abstain objective abstain abstain objective abstain abstain abstain other abstain abstain abstain abstain objective objective abstain abstain abstain method method abstain abstain abstain abstain other method abstain method other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other