Neural encoder-decoder models have been successful in natural language generation tasks. However, real applications of abstractive summarization must consider additional constraint that a generated summary should not exceed a desired length. In this paper, we propose a simple but effective extension of a sinusoidal positional encoding (Vaswani et al., 2017) to enable neural encoder-decoder model to preserves the length constraint. Unlike in previous studies where that learn embeddings representing each length, the proposed method can generate a text of any length even if the target length is not present in training data. The experimental results show that the proposed method can not only control the generation length but also improve the ROUGE scores. Neural encoder-decoder models have been successfully applied to various natural language generation tasks including machine translation (Sutskever et al., 2014), summarization (Rush et al., 2015), and caption generation (Vinyals et al., 2015). Still, it is necessary to control the output length for abstractive summarization, which generates a summary for a given text while satisfying a space constraint. In fact, Figure 1 shows a large variance in output sequences produced by a widely used encoder-decoder model (Luong et al., 2015), which has no mechanism for controlling the length of the output sequences. Fan et al. (2018) trained embeddings that correspond to each output length to control the output sequence length. Since the embeddings for different lengths are independent, it is hard to generate a sequence of the length that is infrequent in training data. Thus, a method that can model any lengths continuously is required. Kikuchi et al. (2016) proposed two learning based methods for an LSTM encoder-decoder: LenEmb and LenInit. LenEmb inputs an embedding representing the remaining length in each decoding step. Since this approach also prepares embeddings for each length independently, it suffers from the same problem as that in Fan et al. (2018). On the other hand, LenInit can handle arbitrary lengths because it combines the scalar value of a desired length with a trainable embedding. LenInit initializes the LSTM cell of the decoder with the embedding depending on the scalar value of the desired length. Liu et al. (2018) incorporated such scalar values into the initial state of the decoder in a CNN encoder-decoder. These approaches deal with any length but it is reasonable to incorporate the distance to the desired terminal position into each decoding step such as in LenEmb. In this study, we focused on Transformer (Vaswani et al., 2017), which recently achieved the state-of-the-art score on the machine translation task. We extend the sinusoidal positional encoding, which represents a position of each token in Transformer (Vaswani et al., 2017), to represent a distance from a terminal position on the decoder side. In this way, the proposed method considers the remaining length explicitly at each decoding step. Moreover, the proposed method can handle any desired length regardless of its appearance in a training corpus because it uses the same continuous space for any length. We conduct experiments on the headline generation task. The experimental results show that our proposed method is able to not only control the output length but also improve the ROUGE scores from the baselines. Our code and constructed test data are publicly available at: https://github.com/takase/control-length. Transformer (Vaswani et al., 2017) uses a sinusoidal positional encoding to represent the position of an input. Transformer feeds the sum of the positional encoding and token embedding to the input layer of its encoder and decoder. Let pos be the position and d be the embedding size. Then, the i th dimension of the sinusoidal positional encoding P E ( pos,i ) is as follows: P E ( pos, 2 i ) = sin (cid:18) pos 10000 2 id (cid:19) , (1) P E ( pos, 2 i +1) = cos (cid:18) pos 10000 2 id (cid:19) . (2) In short, each dimension of the positional encoding corresponds to a sinusoid whose period is 10000 2 i/d  2  . Since this function returns an identical value at the same position pos , the above positional encoding can be interpreted as representing the absolute position of each input token. In this paper, we extend Equations (1) and (2) to depend on the given output length and the distance from the terminal position. We propose two extensions: length-difference positional encoding ( LDP E ) and length-ratio positional encoding ( LRP E ). Then we replace Equations (1) and (2) with (3) and (4) (or (5) and (6)) on the decoder side to control the output sequence length. We define LDP E and LRP E as follows: LDP E ( pos,len, 2 i ) = sin (cid:18) len  pos 10000 2 id (cid:19) , (3) LDP E ( pos,len, 2 i +1) = cos (cid:18) len  pos 10000 2 id (cid:19) , (4) LRP E ( pos,len, 2 i ) = sin (cid:18) pos len 2 id (cid:19) , (5) LRP E ( pos,len, 2 i +1) = cos (cid:18) pos len 2 id (cid:19) , (6) where len presents the given length constraint. LDP E returns an identical value at the position where the remaining length to the terminal position is the same. LRP E returns a similar value at the positions where the ratio of the remaining length to the terminal position is similar. Let us consider the d -th dimension as the simplest example. Since we obtain sin( pos/len ) (or cos( pos/len ) ) at this dimension, the equations yield the same value when the remaining length ratio is the same, e.g., pos = 5 , len = 10 and pos = 10 , len = 20 . We add LDP E (or LRP E ) to the input layer of Transformer in the same manner as in Vaswani et al. (2017). In the training step, we assign the length of the correct output to len . In the test phase, we control the output length by assigning the desired length to len . We conduct experiments on the headline generation task on Japanese and English datasets. The purpose of the experiments is to evaluate the ability of the proposed method to generate a summary of good quality within a specified length. We used JAMUL corpus as the Japanese test set (Hitomi et al., 2019). This test set contains three kinds of headlines for 1,181 1 news articles written by professional editors under the different upper bounds of headline lengths. The upper bounds are 10, 13, and 26 characters ( len = 10 , 13 , 26 ). This test set is suitable for simulating the real process of news production because it is constructed by a Japanese media company. In contrast, we have no English test sets that contain headlines of multiple lengths. Thus, we randomly extracted 3,000 sentence-headline 1 We obtained this test set by applying the pre-processing script at https://github.com/asahi-research/Gingo to the original JAMUL corpus. pairs that satisfy a length constraint from the test set constructed from annotated English Gigaword (Napoles et al., 2012) by pre-processing scripts of Rush et al. (2015) 2 . We set three configu-rations for the number of characters as the length constraint: 0 to 30 characters ( len = 30 ), 30 to 50 characters ( len = 50 ), and 50 to 75 characters ( len = 75 ). Moreover, we also evaluate the proposed method on the DUC-2004 task 1 (Over et al., 2007) for comparison with published scores in previous studies. Unfortunately, we have no large supervision data with multiple headlines of different lengths associated with each news article in both languages. Thus, we trained the proposed method on pairs with a one-to-one correspondences between the source articles and headlines. In the training step, we regarded the length of the target headline as the desired length len . For Japanese, we used the JNC corpus, which contains a pair of the lead three sentences of a news article and its headline (Hitomi et al., 2019). The training set contains about 1.6M pairs 3 . For English, we used sentence-headline pairs extracted from the annotated English Gigaword with the same pre-processing script used in the construction of the test set. The training set contains about 3.8M pairs. In this paper, we used a character-level decoder to control the number of characters. On the encoder side, we used subword units to construct the vocabulary (Sennrich et al., 2016; Kudo, 2018). We set the hyper-parameter to fit the vocabulary size to about 8k for Japanese and 16k for English. We implemented two methods proposed by previous studies to control the output length and handle arbitrary lengths. We employed them and Transformer as baselines. LenInit Kikuchi et al. (2016) proposed LenInit, which controls the output length by initializing the LSTM cell m of the decoder as follows: m = len  b, (7) where b is a trainable vector. We incorporated this method with a widely used LSTM encoder-decoder model (Luong et al., 2015) 4 . For a fair 2 https://github.com/facebookarchive/NAMAS 3 We obtained this training set by applying the preprocessing script at https://github.com/asahi-research/Gingo. 4 We used an implementation at https://github.com/mlpnlp/mlpnlp-nmt. comparison, we set the same hyper-parameters as in Takase et al. (2018) because they indicated that the LSTM encoder-decoder model trained with the hyper-parameters achieved a similar performance to the state-of-the-art on the headline generation. Length Control (LC) Liu et al. (2018) proposed a length control method that multiplies the desired length by input token embeddings. We trained the model with their hyper-parameters. Transformer Our proposed method is based on Transformer (Vaswani et al., 2017) 5 . We trained Transformer with the equal hyper-parameters as in the base model in Vaswani et al. (2017). Table 1 shows the recall-oriented ROUGE-1 (R-1), 2 (R-2), and L (R-L) scores of each method on the Japanese test set 6 . This table indicates that Transformer with the proposed method (Transformer+ LDP E and Transformer+ LRP E ) outperformed the baselines for all given constraints ( len = 10 , 13 , 26 ). Transformer+ LRP E performed slightly better than Transformer+ LDP E . Moreover, we improved the performance by incorporating the standard sinusoidal positional encoding (+ P E ) on len = 10 and 26 . The results imply that the absolute position also helps to generate better headlines while controlling the output length. Table 2 shows the recall-oriented ROUGE scores on the English Gigaword test set. This table indicates that LDP E and LRP E significantly improved the performance on len = 75 . Moreover, the absolute position ( P E ) also improved the performance in this test set. In particular, P E was very effective in the setting of very short headlines ( len = 30 ). However, the proposed method slightly lowered ROUGE-2 scores from the bare Transformer on len = 30 , 50 . We infer that the bare Transformer can generate headlines whose lengths are close to 30 and 50 because the majority of the training set consists of headlines whose lengths are less than or equal to 50. However, most of the generated headlines breached the length constraints, as explained in Section 3.4. 5 We used an implementation at https://github.com/pytorch/fairseq. 6 To calculate ROUGE scores on the Japanese dataset, we used https://github.com/asahi-research/Gingo. desired length ( len ) from the training data. The lower parts of Table 1 and 2 show ROUGE scores of the proposed method trained on the modified training data. These parts show that the proposed method achieved comparable scores to ones trained on whole training dataset. These results indicate that the proposed method can generate high-quality headlines even if the length does not appear in the training data. Table 3 shows the recall-oriented ROUGE scores on the DUC-2004 test set. Following the evaluation protocol (Over et al., 2007), we truncated characters over 75 bytes. The table indicates that LDP E and LRP E significantly improved the performance compared to the bare Transformer, and achieved better performance than the baselines except for R-2 of LenInit. This table also shows the scores reported in the previous studies. The proposed method outperformed the previous methods that control the output length and achieved the competitive score to the state-of-the-art scores. Since the proposed method consists of a character-based decoder, it sometimes generated Variance Japanese dataset English Gigaword Model len = 10 len = 13 len = 26 len = 30 len = 50 len = 75 BaselinesLenInit 0.047 0.144 0.058 0.114 0.112 0.091 LC 0.021 0.028 0.040 0.445 0.521 0.871 Transformer 181.261 115.431 38.169 193.119 138.566 620.887 Proposed method Transformer+ LDPE 0.000 0.000 0.000 0.015 0.012 0.013 + PE 0.003 0.001 0.001 0.016 0.009 0.007 Transformer+ LRPE 0.121 0.210 0.047 0.082 0.071 0.187 + PE 0.119 0.144 0.058 0.142 0.110 0.173 Proposed method trained on the dataset without headlines consisting of the target lengths Transformer+ LDPE 0.000 0.002 0.000 0.018 0.009 0.009 + PE 0.021 0.001 0.003 0.021 0.013 0.010 Transformer+ LRPE 0.191 0.362 0.043 0.120 0.058 0.133 + PE 0.183 0.406 0.052 0.138 0.081 0.154 Table 4: Variances of generated headlines. words unrelated to a source sentence. Thus, we applied a simple re-ranking to each n -best headlines generated by the proposed method ( n = 20 in this experiment) based on the contained words. Our re-ranking strategy selects a headline that contains source-side words the most. Table 3 shows that Transformer+ LRP E + P E with this re-ranking (+Re-ranking) achieved better scores than the state-of-the-art (Suzuki and Nagata, 2017). Following Liu et al. (2018), we used the variance of the generated summary lengths against the desired lengths as an indicator of the preciseness of the output lengths. We calculated variance ( var ) for n generated summaries as follows 7 : var = 1 n n (cid:88) i =1 | l i  len | 2 , (8) where len is the desired length and l i is the length of the generated summary. Table 4 shows the values of Equation (8) computed for each method and the desired lengths. This table indicates that LDP E could control the length of headlines precisely. In particular, LDP E could generate headlines with the identical length to the desired one in comparison with LenInit and LC. LRP E also generated headlines with a precise length but its variance is larger than those of previous studies in very short lengths, i.e., len = 10 and 13 in Japanese. However, we consider LRP E is enough for real applications because the averaged difference between its output and the desired length is small, e.g., 0 . 1 for len = 10 . 7 Liu et al. (2018) multiplies Equation (8) by 0 . 001 . The lower part of Table 4 shows the variances of the proposed method trained on the modified training data that does not contain headlines whose lengths are equal to the desired length, similar to the lower parts of Table 1 and 2. The variances for this part are comparable to the ones obtained when we trained the proposed method with whole training dataset. This fact indicates that the proposed method can generate an output that satisfies the constraint of the desired length even if the training data does not contain instances of such a length. In this paper, we proposed length-dependent positional encodings, LDP E and LRP E , that can control the output sequence length in Transformer. The experimental results demonstrate that the proposed method can generate a headline with the desired length even if the desired length is not present in the training data. Moreover, the proposed method significantly improved the quality of headlines on the Japanese headline generation task while preserving the given length constraint. For English, the proposed method also generated headlines with the desired length precisely and achieved the top ROUGE scores on the DUC-2004 test set. The research results have been achieved by Re-search and Development of Deep Learning Technology for Advanced Multilingual Speech Trans-lation, the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.
abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain method objective other abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain other