Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation. 1 1 Introduction Text summarization is an important natural language processing (NLP) task, aiming at generating concise summaries for given texts while preserving the key information. It has extensive real-world applications such as headline generation (Nenkova et al., 2011). In this paper, we focus on the setting of sentence summarization (Rush et al., 2015; Filippova et al., 2015). State-of-the-art text summarization models are typically trained in a supervised way with large training corpora, comprising pairs of long texts and their summaries (Zhang et al., 2020; Aghajanyan et al., 2020, 2021). However, such parallel data are expensive to obtain, preventing the applications to less popular domains and less spoken languages. Unsupervised text generation has been attracting increasing interest, because it does not require parallel data for training. One widely used approach 1 Our code, model, and output are released at: https: //github.com/MANGA-UOFA/NAUS is to compress a long text into a short one, and to reconstruct it to the long text by a cycle consistency loss (Miao and Blunsom, 2016; Wang and Lee, 2018; Baziotis et al., 2019). Due to the in-differentiability of the compressed sentence space, such an approach requires reinforcement learning (or its variants), which makes the training difficult (Kreutzer et al., 2021). Recently, Schumann et al. (2020) propose an edit-based approach for unsupervised summarization. Their model maximizes a heuristically defined scoring function that evaluates the quality (fluency and semantics) of the generated summary, achieving higher performance than cycle-consistency methods. However, the search approach is slow in inference because hundreds of search steps are needed for each data sample. Moreover, their approach can only select words from the input sentence with the word order preserved. Thus, it is restricted and may generate noisy summaries due to the local optimality of search algorithms. To address the above drawbacks, we propose a Non-Autoregressive approach to Unsupervised Summarization (NAUS). The idea is to perform search as in Schumann et al. (2020) and, inspired by Li et al. (2020), to train a machine learning model to smooth out such noise and to speed up the inference process. Different from Li et al. (2020), we propose to utilize non-autoregressive decoders, which generate all output tokens in parallel due to our following observations:  Non-autoregressive models are several times faster than autoregressive generation, which is important when the system is deployed. The input and output of the summarization task have a strong correspondence. Non-autoregressive generation supports encoder-only architectures, which can better utilize such inputoutput correspondence and even outperform autoregressive models for summarization. sign a length-control algorithm based on dynamic programming to satisfy the constraint of output lengths, which is typical in summarization applications but cannot be easily achieved with autoregressive models. We conducted experiments on Gigaword headline generation (Graff et al., 2003) and DUC2004 (Over and Yen, 2004) datasets. Experiments show that our NAUS achieves state-of-the-art performance on unsupervised summarization; especially, it outperforms its teacher (i.e., the search approach), confirming that NAUS can indeed smooth out the search noise. Regarding inference efficiency, our NAUS with truncating is 1000 times more efficient than the search approach; even with dynamic programming for length control, NAUS is still 100 times more efficient than search and several times more efficient than autoregressive models. Our NAUS is also able to perform length-transfer summary generation, i.e., generating summaries of different lengths from training. In our approach, we first follow Schumann et al. (2020) and obtain a summary by discrete search towards a heuristically defined objective function (2.1). Then, we propose a non-autoregressive model for the summarization task (2.2). We present the training strategy and the proposed length-control algorithm in 2.3. Consider a given source text x = (x 1 , x 2 , . . . , x n ) The goal of summarization is to find a shorter text y = (y 1 , y 2 , . . . , y m ) as the summary. Our work on unsupervised summarization follows the recent progress of search-based text generation (Liu et al., 2020, 2021a; Kumar et al., 2020). Schumann et al. (2020) formulate summarization as word-level extraction (with order preserved), and apply edit-based discrete local search to maximize a heuristically designed objective. Specifically, the objective function considers two aspects: (1) a language fluency score f LM ( y ) , given by the reciprocal of a language model's perplexity; and (2) a semantic similarity score f SIM ( y ; x ) , given by the cosine embeddings. The overall objective combines the two aspects as f ( y ; x ) = f LM ( y )  f SIM ( y ; x )  (1) where  is a weighting hyperparameter. Further, the desired summary length can be spec-ified as a hard constraint, achieved by searching only among sentences of the correct length. Suppose the desired summary length is T , the approach selects T random words from the input, and maximizes the scoring function (1) by changing the selection and non-selection of two words. A greedy hill-climbing algorithm determines whether the change is accepted or not. In other words, a change is accepted if the score improves, or rejected otherwise. Such a process continues until a (possibly local) optimum is found. A pilot analysis in Schumann et al. (2020) shows that words largely overlap between a source text and its reference summary. This explains the high performance of such a word extraction approach, being a state-of-the-art unsupervised summarization system and outperforming strong competitors, e.g., cycle consistency (Wang and Lee, 2018; Baziotis et al., 2019). Despite the high performance, such edit-based search has several drawbacks. First, the search process is slow because hundreds of local search steps are needed to obtain a high-quality summary. Second, their approach only extracts the original words with order preserved. Therefore, the generated summary is restricted and may be noisy. To this end, we propose a Non-Autoregressive approach to Unsupervised Summarization (NAUS) by learning from the search results. In this way, the machine learning model can smooth out the search noise and is much faster, largely alleviating the drawbacks of search-based summarization. Compared with training an autoregressive model from search (Li et al., 2020), non-autoregressive generation predicts all the words in parallel, further improving inference efficiency by several times. Moreover, a non-autoregressive model enables us to design an encoder-only architecture, which is more suited to the summarization task due to the strong correspondence between input and output, which cannot be fully utilized by encoderdecoder models, especially autoregressive ones. Specifically, we propose to use multi-layer Transformer (Vaswani et al., 2017) as the non-autoregressive architecture for summarization. Each Transformer layer is composed of a multihead attention sublayer and a feed-forward sublayer. Additionally, there is a residual connection in each sublayer, followed by layer normalization. Let X ( n )  RT  d be the representation at the n th layer, where T is the number of words and d is the dimension. Specially, the input layer X (0) is the embeddings of words. Suppose we have h attention heads. The output of the i th head in the n th attention sublayer is A ( n ) i = softmax (cid:16) Q i K (cid:62) i  d k (cid:17) V i , where Q i , K i , and V i are matrices calculated by three distinct multi-layer perceptrons (MLPs) from X ( n  1) ; d k is the attention dimension. Multiple attention heads are then concatenated: A ( n ) = Concat (cid:0) A ( n ) 1 , . . . , A ( n ) h (cid:1) WO where WO  R d  d is a weight matrix. Then, we have a residual connection and layer normalization by  A ( n ) = LayerNorm (cid:0) X ( n  1) + A ( n ) (cid:1) (2) Further, an MLP sublayer processes  A ( n ) , followed by residual connection and layer normalization, yielding the n th layer's representation X ( n ) = LayerNorm (cid:0)  A ( n ) + MLP(  A ( n ) ) (cid:1) (3) The last Transformer layer X ( N ) is fed to softmax to predict the words of the summary in a non-autoregressive manner, that is, the probability at the t th step is given by softmax( W x ( N ) t ) , where x ( N ) t is the t th row of the matrix X ( N ) and W is the weight matrix. It is emphasized that, in the vocabulary, we include a special blank token (cid:15) , which is handled by dynamic programming during both training and inference (2.3). This enables us to generate a shorter summary than the input with such a multi-layer Transformer. Our model can be thought of as an encoder-only architecture, differing from a typical encoder decoder model with cross attention (Vaswani et al., 2017; Baziotis et al., 2019; Zhou and Rush, 2019). Previously, Su et al. (2021) propose a seemingly similar model to us, but put multiple end-of-sequence (EOS) tokens at the end of the generation; thus, they are unable to maintain the correspondence between input and output. Instead, we allow blank tokens scattering over the entire sentence; the residual connections in Eqns (2) and (3) can better utilize such inputoutput correspondence for summarization. In this section, we first introduce the Connectionist Temporal Classification (CTC) training. Then, we propose a length-control decoding approach for summary generation. CTC Training. The Connectionist Temporal Classification (CTC, Graves et al., 2006) algorithm allows a special blank token (cid:15) in the vocabulary, and uses dynamic programming to marginalize out such blank tokens, known as latent alignment (Sa-haria et al., 2020). In addition, non-autoregressive generation suffers from a common problem that words may be repeated in consecutive steps (Gu et al., 2018; Lee et al., 2018); thus, CTC merges repeated words unless separated by (cid:15) . For example, the sequence of tokens a(cid:15)(cid:15)aabb(cid:15) is reduced to the text aab , denoted by ( a(cid:15)(cid:15)aabb(cid:15) ) = aab . Concretely, the predicted likelihood is marginal-ized over all possible fillings of (cid:15) , i.e., all possible token sequences that are reduced to the groundtruth text: P ( y | x ) = (cid:88) w :( w )= y P ( w | x ) (4) 7918 where P ( w | x ) is the probability of generating a sequence of tokens w . Although enumerating every candidate in { w : ( w ) = y } is intractable, such marginalization fortunately can be computed by dynamic programming in an efficient way. Let  s,t = (cid:80) w 1: s :( w 1: s )= y 1: t P ( w 1: s | x ) be the marginal probability of generating y 1: t up to the s th decoding slot. Moreover,  s, 0 is defined to be the probability that w 1: s is all (cid:15) , thus not having matched any word in y . The  s,t variable can be further decomposed into two terms  s,t =  (cid:15)s,t +   (cid:15) s,t , where the first term is such probability with w s = (cid:15) , and the second term w s (cid:54) = (cid:15) . Apparently, the initialization of  variables is  (cid:15) 1 , 0 = P (w 1 = (cid:15) | x ) (5)   (cid:15) 1 , 1 = P (w 1 = y 1 | x ) (6)  (cid:15) 1 ,t = 0 ,  t  1 (7)   (cid:15) 1 ,t = 0 ,  t > 1 or t = 0 (8) Eqn. (7) is because, at the first prediction slot, the empty token (cid:15) does not match any target words; Eqn. (8) is because the predicted non(cid:15) first token must match exactly the first target word. The recursion formula for  (cid:15)s,t is  (cid:15)s,t =  s  1 ,t P (w t = (cid:15) | x ) since the newly predicted token (cid:15) with probability P (w t = (cid:15) | x ) does not match any target word, inheriting  s  1 ,t . The recursion formula for   (cid:15) s,t is   (cid:15) s,t =   (cid:0)  (cid:15)s  1 ,t  1 +   (cid:15) s  1 ,t (cid:1) P (w s = y t | x ) , if y t = y t  1 (cid:0)  s  1 ,t  1 +   (cid:15) s  1 ,t (cid:1) P (w s = y t | x ) , otherwise Here, w s is not (cid:15) , so we must have w s = y t , having the predicted probability P (w s = y t | x ) . If y t = y t  1 , then we have two sub-cases: first, w 1: s  1 is reduced to y 1: t  1 with w s  1 = (cid:15) separating two repeating words in y , having probability  (cid:15)s  1 ,t  1 ; or second, w 1: s  1 is reduced to y 1: t with w s  1 = y t (cid:54) = (cid:15) , having probability   (cid:15) s  1 , which implies we are merging w s  1 and w s . If y t (cid:54) = y t  1 , w 1: s  1 is reduced to either y 1: t  1 or y 1: t . In the first case, w s  1 can be either (cid:15) or non(cid:15) , given by  s  1 ,t  1 =  (cid:15) s  1 ,t  1 +   (cid:15) s  1 ,t  1 . In the second case, we must have w s  1 (cid:54) = (cid:15) , which has a probability of   (cid:15) s  1 ,t . The CTC maximum likelihood estimation is to maximize the marginal probability, which is equivalent to minimizing the loss   | w | , | y | . Since the dynamic programming formulas are differentiable, the entire model can be trained by backpropagation in an end-to-end manner with auto-differentiation tools (such as PyTorch). Length-Control Inference. Controlling output length is the nature of the summarization task, for example, displaying a short news headline on a mo-bile device. Moreover, Schumann et al. (2020) show that the main evaluation metric ROUGE (Lin, 2004) is sensitive to the summary length, and longer summaries tend to achieve higher ROUGE scores. Thus, it is crucial to control the summary length for fair comparison. We propose a length-control algorithm by dynamic programming (DP), following the nature of CTC training. However, our DP is an approximate algorithm because of the dependencies introduced by removing consecutive repeated tokens. Thus, we equip our DP with a beam search mechanism. We define B s,t to be a set of topB sequences with s predicted tokens that are reduced to t words. B s,t is constructed by three scenarios. First, the blank token (cid:15) is predicted for the s th generation slot, and thus the summary length t remains the same, shown by the blue arrow in Figure 2. This yields a set of candidates B (1) s,t = (cid:8) b  (cid:15) : b  B s  1 ,t (cid:9) (9) where  refers to string/token concatenation. Second, a repeated word is predicted for the s th generation slot, i.e., b s  1 for a subsequence b of length s  1 . In this case, the summary length t also remains the same, also shown in the blue arrow in Figure 2. This gives a candidate set B (2) s,t = (cid:8) b  b s  1 : b  B s  1 ,t (cid:9) (10) Third, a non(cid:15) , non-repeating word w s is generated, increasing the summary length from t  1 to 7919 t , shown by the red arrow in Figure 2. This gives B (3) s,t = top B (cid:8) b  w : b  B s  1 ,t  1 , w s (cid:54) = (cid:15), w s (cid:54) = b s  1 (cid:9) (11) where top B selects the best B elements by the probability P (w s | x ) . Based on the three candidates sets, we select topB sequences to keep the beam size fixed: B s,t = top B ( B (1) s,t  B (2) s,t  B (3) s,t ) (12) where the sequences are ranked by their predicted joint probabilities. Theorem 1. (1) If repeating tokens are not merged, then the proposed length-control algorithm with beam size B = 1 finds the exact optimum B S,T being the most probable lengthT sentence given by S prediction slots. (2) If we merge repeating tokens predicted by CTC-trained models, the above algorithm may not be exact. Appendix A presents the proof of the theorem and provides a more detailed analysis, showing that our length-control algorithm, although being approximate inference, can generate a summary of the desired length properly. Compared with truncating an overlength output, our approach is able to generate more fluent and complete sentences. Also, our length-control algorithm is different from conventional beam search, shown in Appendix C. 3 Experiments 3.1 Setup Datasets. We evaluated our NAUS model on Gigaword headline generation and DUC2004 datasets. The headline generation dataset (Rush et al., 2015) is constructed from the Gigaword news corpus (Graff et al., 2003), where the first sentence of a news article is considered as input text and the news title is considered as the summary. The dataset contains 3.8M/198K/1951 samples for train-ing/validation/test. Based on the analysis of the training size in Appendix B, we used 3M samples for training NAUS. It should be emphasized that, when NAUS learns from search, we only use the input of the training corpus: we perform search (Schumann et al., 2020) for each input, and train our NAUS from the search results. Therefore, we do not utilize any labeled parallel data, and our approach is unsupervised. Moreover, we considered two settings with desired summary lengths of 8 and 10, following Schumann et al. (2020). Our NAUS is trained from respective search results. The DUC2004 dataset (Over and Yen, 2004) is designed for testing only with 500 samples, where we also take the first sentence of an article as the input text. Our NAUS is transferred from the above headline generation corpus. Based on the length of DUC2004 summaries, we trained NAUS from search results with 13 words, also following Schumann et al. (2020) for fair comparison. Evaluation Metrics. We evaluated the quality of predicted summaries by ROUGE scores 2 (Lin, 2004), which are the most widely used metrics in previous work (Wang and Lee, 2018; Baziotis et al., 2019; Zhou and Rush, 2019). Specifically, ROUGEn evaluates n -gram overlap between a predicted summary and its reference summary; ROUGE-L, instead, measures the longest common sequence between the predicted and reference summaries. Different ROUGE variants are adopted in previous work, depending on the dataset. We followed the standard evaluation scripts and evaluated headline generation by ROUGE F1 (Wang and Lee, 2018; Baziotis et al., 2019; Schumann et al., 2020) and DUC2004 by Truncate ROUGE Recall (Dorr et al., 2003; West et al., 2019). In addition to summary quality, we also evaluated the inference efficiency of different methods, as it is important for the deployment of deep learning models in real-time applications. We report the average inference time in seconds for each data sample, and compare the speedup with Schumann et al. (2020)'s search approach, which achieves (previous) state-of-the-art ROUGE scores. Our experiments were conducted on an i9-9940X CPU and an RTX6000 graphic card. Appendix B presents additional implementation details. Main Results. Table 1 presents the performance of our model and baselines on the Gigaword headline test set. For a fair comparison, we categorize all approaches by average summary lengths of ~8 and ~10 into Groups A and B, respectively. The Lead baseline extracts the first several words of the input sentence. Despite its simplicity, the Lead approach is a strong summarization baseline adopted in most previous work (Fvry and Phang, 2018; Baziotis et al., 2019). Wang and Lee (2018) utilize cycle consistency (Miao and Blunsom, 2016) for unsupervised summarization; the performance is relatively low, because the cycle consistency loss cannot ensure the generated text is a valid summary. Zhou and Rush (2019) perform beam search towards a step-by-step decomposable score of fluency and contextual matching. Both are unable to explicitly control the summary length: in a fair comparison of length 10 (Group B, Table 1), their performance is worse than the (previous) state-of-the-art approach (Schu-mann et al., 2020), 3 which performs edit-based local search. Our NAUS approach follows Schumann et al. (2020), but trains a non-autoregressive model from 3 Schumann et al. (2020) present a few variants that use additional datasets for training language models (in an unsupervised way). In our study, we focus on the setting without data augmentation, i.e., the language model is trained on nonparallel the Gigawords corpus. search results. We consider two settings for controlling the summary length: truncating longer summaries and decoding with our proposed length-control algorithm. Both of our variants outperform Schumann et al. (2020) by 1.212.73 in terms of the total ROUGE score (Rows 56 & 1314, Table 1). As mentioned, Schumann et al. (2020) only extract original words with order preserved, yielding noisy sentences. Our NAUS, as a student, learns from the search-based teacher model and is able to smooth out its noise. This is a compelling result, as our student model outperforms its teacher. Regarding inference efficiency, our NAUS method with truncating is more than 1300 times faster than Schumann et al. (2020), because we do not need iterative search. Even with dynamic programming and beam search for length control, NAUS is still over 100 times faster. This shows our NAUS is extremely efficient in inference, which is important for real-time applications. Although the efficiency of Wang and Lee (2018) and Zhou and Rush (2019) is not available, we still expect our approach to be a few times faster (despite our higher ROUGE scores) because their models are autoregressive. By contrast, our NAUS is non-autoregressive, meaning that it predicts all words simultaneously. We will provide a controlled comparison between autoregressive and non-autoregressive models in Table 3. Table 2 shows the results on the DUC2004 dataset. tis et al., 2019; West et al., 2019) does not perform well on this dataset, outperformed by an early rule-based syntax tree trimming approach (Za-jic et al., 2004) and the state-of-the-art edit-based search (Schumann et al., 2020). The performance of our NAUS model is consistent with Table 1, outperforming all previous methods in terms of the total ROUGE score, and being 1001000 times faster than the search approach (Schumann et al., 2020). In general, the proposed NAUS not only achieves state-of-the-art ROUGE scores for unsupervised summarization, but also is more efficient when deployed. Results are consistent on both datasets, demonstrating the generality of our NAUS. In-Depth Analyses. We conduct in-depth analyses on the proposed NAUS model in Table 3. Due to the limit of time and space, we chose the Gigaword headline generation as our testbed. All the autoregressive (AR) and non-autoregressive (NAR) variants learn from the search output of our replication (Rows 2 & 11), where we achieve very close results to those reported in Schumann et al. (2020). We first tried vanilla encoderdecoder NAR Transformer (Rows 4 & 13, Gu et al., 2018), where we set the number of decoding slots as the desired summary length; thus, the blank token and the length-control algorithm are not needed. As seen, a vanilla NAR model does not perform well, and CTC largely outperforms vanilla NAR in both groups (Rows 56 & 1415). Such results are highly consistent with the translation literature (Sa-haria et al., 2020; Chan et al., 2020; Gu and Kong, 2021; Qian et al., 2021; Huang et al., 2022). The proposed encoder-only NAUS model outperforms encoderdecoder ones in both groups in terms of the total ROUGE score, when the summary length is controlled by either truncating or length-control decoding (Rows 89 & 1718). Profoundly, our non-autoregressive NAUS is even better than the autoregressive Transformer (Rows 3 & 12). We also experimented with previous non-autoregressive work for supervised summarization (Su et al., 2021) 4 in our learning-from-search setting. Although their approach appears to be encoder-only, it adds end-of-sequence (EOS) tokens at the end of the generation, and thus is unable to utilize the inputoutput correspondence. Their performance is higher than vanilla NAR models, but lower than ours. By contrast, NAUS is able to capture such correspondence with the residual connections, i.e., Eqns. (2) and (3), in its encoder-only architecture. Generally, the efficiency of encoder-only NAR 5 (without length-control decoding) is ~2 times faster than encoderdecoder NAR and ~20 times faster than the AR Transformer. Further, our length-control decoding improves the total ROUGE score, compared with truncating, for both encoderdecoder CTC and encoder-only NAUS models (Rows 6, 9, 15, & 18), although its dynamic programming is slower. Nevertheless, our non-autoregressive NAUS with length control is ~200 times faster than search and ~3 times faster than the AR Transformer. results in our appendices: C. Analysis of Beam Search D. Case Study E. Human Evaluation F. Length-Transfer Summarization 4 To the best of our knowledge, the other two non-autoregressive supervised summarization models are Yang et al. (2021) and Qi et al. (2021). Their code and pretrained models are not available, making replication difficult. 5 The standard minimal encoderdecoder NAR model has 6 layers for the encoder and another 6 layers for the decoder (Vaswani et al., 2017). Our NAUS only has a 6-layer encoder. Our pilot study shows that more layers do not further improve performance in our encoder-only architecture. Summarization systems can be generally categorized into two paradigms: extractive and abstractive. Extractive systems extract certain sentences and clauses from input, for example, based on salient features (Zhou and Rush, 2019) or feature construction (He et al., 2012). Abstraction systems generate new utterances as the summary, e.g., by sequence-to-sequence models trained in a supervised way (Zhang et al., 2020; Liu et al., 2021b). Recently, unsupervised abstractive summarization is attracting increasing attention. Yang et al. (2020) propose to use the Lead baseline (first several sentences) as the pseudo-groundtruth. However, such an approach only works with well-structured articles (such as CNN/DailyMail). Wang and Lee (2018) and Baziotis et al. (2019) use cycle consistency for unsupervised summarization. Zhou and Rush (2019) propose a step-by-step decomposable scoring function and perform beam search for summary generation. Schumann et al. (2020) propose an edit-based local search approach, which allows a more comprehensive scoring function and outperforms cycle consistency and beam search. Our paper follows Schumann et al. (2020) but trains a machine learning model to improve efficiency and smooth out search noise. Previously, Li et al. (2020) fine-tune a GPT-2 model based on search results for unsupervised paraphrasing; Jolly et al. (2022) adopt the search-and-learning framework to improve the semantic coverage for few-shot data-to-text generation. We extend previous work in a non-trivial way by designing a non-autoregressive generator and further proposing a length-control decoding algorithm. The importance of controlling the output length is recently realized in the summarization community. Baziotis et al. (2019) and Su et al. (2021) adopt soft penalty to encourage shorter sentences; Yang et al. (2021) and Qi et al. (2021) control the summary length through POS tag and EOS predictions. None of these studies can control the length explicitly. Song et al. (2021) is able to precisely control the length by progressively filling a predetermined number of decoding slots, analogous to the vanilla NAR model in our non-autoregressive setting. Non-autoregressive generation is originally proposed for machine translation (Gu et al., 2018; Guo et al., 2020; Saharia et al., 2020), which is later extended to other text generation tasks. Wiseman et al. (2018) address the table-to-text generation task, and model output segments by a hidden semi-Markov model (Ostendorf et al., 1996), simultaneously generating tokens for all segments. Jia et al. (2021) apply non-autoregressive models to extractive document-level summarization. Su et al. (2021) stack a non-autoregressive BERT model with a conditional random field (CRF) for abstractive summarization; since the summary is shorter than the input text, their approach puts multiple end-to-sequence (EOS) tokens at the end of the sentence, and thus is unable to utilize the strong inputoutput correspondence in the summarization task. Yang et al. (2021) apply auxiliary part-of-speech (POS) loss and Qi et al. (2021) explore pretraining strategies for encoderdecoder non-autoregressive summarization. All these studies concern supervised summarization, while our paper focuses on unsupervised summarization. We adopt CTC training in our encoder-only architecture, allowing blank tokens to better align input and output words, which is more appropriate for summarization. In this work, we propose a non-autoregressive unsupervised summarization model (NAUS), where we further propose a length-control decoding algorithm based on dynamic programming. Experiments show that NAUS not only archives state-of-the-art unsupervised performance on Gigaword headline generation and DUC2004 datasets, but also is much more efficient than search methods and autoregressive models. Appendices present additional analyses and length-transfer experiments. Limitation and Future Work. Our paper focuses on unsupervised summarization due to the importance of low-data applications. One limitation is that we have not obtained rigorous empirical results for supervised summarization, where the developed model may also work. This is because previous supervised summarization studies lack explicit categorization of summary lengths (Yang et al., 2020; Qi et al., 2021), making comparisons unfair and problematic (Schumann et al., 2020). Such an observation is also evidenced by Su et al. (2021), where the same model may differ by a few ROUGE points when generating summaries of different lengths. Nevertheless, we have compared with Su et al. (2021) in our setting and show the superiority of the NAUS under fair comparison. We 7923 plan to explore supervised summarization in future work after we establish a rigorous experimental setup, which is beyond the scope of this paper. We thank Raphael Schumann for providing valuable suggestions on the work. We also thank the Action Editor and reviewers for their comments during ACL Rolling Review. The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR AI Chair Program, a UAHJIC project, a donation from DeepMind, and Compute Canada (www.computecanada.ca).
abstain objective objective result objective abstain method abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain method result abstain method method method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain method abstain abstain method method method method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain abstain method abstain method method abstain abstain abstain abstain abstain method abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result other other other other other other other other other abstain other objective other other other method other other other other other method method objective abstain abstain abstain method objective abstain abstain result objective other other other other