summaries: This paper proposes a generalization of the attention coverage penalty in seq2seq models (Wu et al 2016), where instead of encouraging a uniform coverage of the source, this work first predicts the expected coverage of each source word. During beam search, beam hypotheses that violate the expected attention coverage constraints receive a penalty. Experiments on 9 summarization benchmarks show that the proposed coverage loss outperforms using existing attention coverage losses during beam search. This paper thinks that cross attention distribution in transformer’s encoder-decoder framework can help the performance of beam search. At the same time, this paper also proposes an algorithm to predict distribution to help beam search. This paper conduct experiments on 9 datasets to prove its effectiveness.  This work proposes to add a global attention feature component to the beam search scoring procedure for conditional generation, which typically just considers the fitness of successor tokens conditioned on the predicted history of tokens in the beam. This augmentation basically keeps track of cumulative local autoregressive attention on the source tokens during beam search and penalizes major deviation from the predicted desired "global" attention over the source tokens. Specifically, this approach proposes a finetuning step of neural language generators, which is responsible for predicting cumulative global attention on each token and the training is done via regression. During training, the "reference" global attention is computed by using the full reference in the decoder transformer instead of just the left prefix of the reference to compute attention.  A relatively simple scheme is proposed to measure deviation of cumulative local attention from the global attention in a decomposable iterative manner. This scheme also involves a form of length reward that encourages the generation to match the length predicted by the global attention.  Experiments are performed on the task fo summarization with pretrained summarization system on 9 datasets. The results show that the proposed beam search augmentation outperforms standard beam search in general. Most strikingly, the ablation experiment with the "Oracle" global attention (instead of prediction, if we had access to the actual global attention and hence the actual reference length) significantly outperforms beam search and the proposed approach demonstrating the potential of global attention. This paper tackles the problem of suboptimality of beam search as it is not able to have a global perspective of the probability of the generated beams due to truncated beam size (less than vocabulary size) for space and time complexity purposes. This paper proposes a global-aware beam search which is calibrated using the global attention distribution to guide the local decisions at every time step during beam search. The global attention distribution prediction is formulated as a regression task, which is then fed into a global scoring function to guide the beam search-based generation. This approach solves the mismatch between the local optimality of beam search with the global optimal generation. This modified search strategy has been shown to boost the summarization of two SOTA models BART and PEGASUS.  This paper presents a method to improve beam search by learning and predicting the global attention. A global scoring function is developed to regulate beam search to generate summaries. In Phase I, the global attention distribution is predicted in order to be included as a protocol to calibrate beam search. In Phase II, a step-wise global scoring function is used to guide beam search based on the predicted global attention distribution.  Experiments on 9 datasets show that the global-aware inference improves state-of-the-art summarization models.  
abstractText: This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available.2
authors.name: Ye Ma
authors.name: Zixun Lan Lu Zong
authors.name: Kaizhu Huang
references.authors: Shuyang Cao Lu Wang
references.title: Attention head masking for inference time content selection in abstractive summarization
references.venue: In Proceedings of the 2021 Conference of the North American 10 Chapter of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Arman Cohan Franck Dernoncourt Doo Soon Kim Trung Bui Seokhwan Kim Walter Chang Nazli Goharian
references.title: A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
references.authors: Eldan Cohen Christopher Beck
references.title: Empirical analysis of beam search performance degradation in neural sequence models
references.venue: In International Conference on Machine Learning,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.authors: Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao Ming Zhou Hsiao-Wuen Hon
references.title: Unified language model pre-training for natural language understanding and generation
references.venue: Advances in Neural Information Processing Systems,
references.authors: Alexander Fabbri Irene Li Tianwei She Suyi Li Dragomir Radev
references.title: Multi-news: A largescale multi-document summarization dataset and abstractive hierarchical model
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Sebastian Gehrmann Yuntian Deng Alexander Rush
references.title: Bottom-up abstractive summarization
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Alex Graves
references.title: Sequence transduction with recurrent neural networks
references.venue: arXiv preprint arXiv:1211.3711,
references.authors: Max Grusky Mor Naaman Yoav Artzi
references.title: Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
references.authors: Karl Moritz Hermann Tomas Kocisky Edward Grefenstette Lasse Espeholt Will Kay Mustafa Suleyman Phil Blunsom
references.title: Teaching machines to read and comprehend
references.venue: Advances in neural information processing systems,
references.authors: Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi
references.title: The curious case of neural text degeneration
references.venue: In International Conference on Learning Representations,
references.authors: Zhiheng Huang Wei Xu Kai Yu
references.title: Bidirectional lstm-crf models for sequence tagging
references.venue: arXiv preprint arXiv:1508.01991,
references.authors: Nitish Shirish Keskar Bryan McCann Lav R Varshney Caiming Xiong Richard Socher
references.title: Ctrl: A conditional transformer language model for controllable generation
references.authors: Rik Koncel-Kedziorski Dhanush Bekal Yi Luan Mirella Lapata Hannaneh Hajishirzi
references.title: Text generation from knowledge graphs with graph transformers
references.authors: Anastassia Kornilova Vladimir Eidelman
references.title: BillSum: A corpus for automatic summarization of US legislation
references.venue: In Proceedings of the 2nd Workshop on New Frontiers in Summarization,
references.authors: Mahnaz Koupaee William Yang Wang
references.title: Wikihow: A large scale text summarization dataset
references.venue: arXiv preprint arXiv:1810.09305,
references.authors: Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer
references.title: BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension
references.venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
references.authors: Jiwei Li Will Monroe Dan Jurafsky
references.title: Learning to decode for future success
references.venue: arXiv preprint arXiv:1701.06549,
references.authors: Mingzhe Li Xiuying Chen Shen Gao Zhangming Chan Dongyan Zhao Rui Yan
references.title: Vmsmo: Learning to generate multimodal summary for video-based news articles
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Wei Li Xinyan Xiao Yajuan Lyu Yuanzhuo Wang
references.title: Improving neural abstractive document summarization with structural regularization
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Yanyang Li Tong Xiao Yinqiao Li Qiang Wang Changming Xu Jingbo Zhu
references.title: A simple and effective approach to coverage-aware neural machine translation
references.venue: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics
references.authors: Chin-Yew Lin
references.title: ROUGE: A package for automatic evaluation of summaries
references.venue: In Text Summarization Branches Out,
references.authors: Yinhan Liu Jiatao Gu Naman Goyal Xian Li Sergey Edunov Marjan Ghazvininejad Mike Lewis Luke Zettlemoyer
references.title: Multilingual denoising pre-training for neural machine translation
references.authors: Clara Meister Ryan Cotterell Tim Vieira
references.title: If beam search is the answer, what was the question
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
references.authors: Myle Ott Michael Auli David Grangier Marc’Aurelio Ranzato
references.title: Analyzing uncertainty in neural machine translation
references.venue: In International Conference on Machine Learning,
references.authors: Ratish Puduppully Li Dong Mirella Lapata
references.title: Data-to-text generation with content selection and planning
references.venue: In Proceedings of the AAAI conference on artificial intelligence,
references.authors: Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever
references.title: Language models are unsupervised multitask learners
references.venue: OpenAI Blog,
references.authors: Marc’Aurelio Ranzato Sumit Chopra Michael Auli Wojciech Zaremba
references.title: Sequence level training with recurrent neural networks
references.venue: arXiv preprint arXiv:1511.06732,
references.authors: Abigail See Peter J Liu Christopher D Manning
references.title: Get to the point: Summarization with pointer-generator networks
references.venue: In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
references.authors: Ilya Sutskever Oriol Vinyals Quoc V Le
references.title: Sequence to sequence learning with neural networks
references.venue: In Advances in neural information processing systems,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In Advances in neural information processing systems,
references.authors: Ashwin K Vijayakumar Michael Cogswell Ramprasaath R Selvaraju Qing Sun Stefan Lee David Crandall Dhruv Batra
references.title: Diverse beam search for improved description of complex scenes
references.venue: In Thirty-Second AAAI Conference on Artificial Intelligence,
references.authors: Oriol Vinyals Alexander Toshev Samy Bengio Dumitru Erhan
references.title: Show and tell: A neural image caption generator
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Michael Völske Martin Potthast Shahbaz Syed Benno Stein
references.title: TL;DR: Mining Reddit to learn automatic summarization
references.venue: In Proceedings of the Workshop on New Frontiers in Summarization,
references.authors: Sam Wiseman Alexander M. Rush
references.title: Sequence-to-sequence learning as beam-search optimization
references.venue: In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,
references.authors: Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault Rémi Louf Morgan Funtowicz Joe Davison Sam Shleifer Patrick von Platen Clara Ma Yacine Jernite Julien Plu Canwen Xu Teven Le Scao Sylvain Gugger Mariama Drame Quentin Lhoest Alexander M. Rush
references.title: Transformers: State-of-theart natural language processing
references.venue: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,
references.authors: Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey
references.title: Google’s neural machine translation system: Bridging the gap between human and machine translation
references.venue: arXiv preprint arXiv:1609.08144,
references.authors: Yilin Yang Liang Huang Mingbo Ma
references.title: Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Jingqing Zhang Yao Zhao Mohammad Saleh Peter Liu
references.title: Pegasus: Pre-training with extracted gap-sentences for abstractive summarization
references.venue: In International Conference on Machine Learning,
references.authors: Wen Zhang Yang Feng Fandong Meng Di You Qun Liu
references.title: Bridging the gap between training and inference for neural machine translation
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Juntang Zhuang Tommy Tang Yifan Ding Sekhar C Tatikonda Nicha Dvornek Xenophon Papademetris James Duncan
references.title: Adabelief optimizer: Adapting stepsizes by the belief in observed gradients
references.venue: Advances in Neural Information Processing Systems,
sections.heading: 1 Introduction
sections.text: As the barriers exist from the auto-regressive design of neural probabilistic text generators to predicting the global optimum directly [30], the heuristic algorithm beam search that factorizes global optimization to multiple local optimizations, has been popularly used for text decoding [24]. In the original beam search setting, the global optimum is a hypothesis g of the highest probability among all possible sentences, and consists of words in vocabulary V . Given the global optimum at step t denoted as g≤t, the local optimums l≤t refer to K candidate sequences with the highest probabilities at each step. While it is necessary to compromise on the beam size K V to ensure text quality [3, 25, 24] and search efficiency, beam search suffers from a major limitation due to its local property. Concretely, assuming that the global optimal hypothesis is within the K local optimal hypotheses of the highest probabilities, i.e. p(g≤t) ≥ p(l≤t), for all t until the termination T , it operates solely with the local information available at each step. In practice, such assumption may however fail in the case that the probability of the global optimum at step τ < T is less than those of the local optimums, i.e. p(g≤τ ) < p(l≤τ ), but is adjusted to a higher level in the later steps, p(g>τ |g≤τ ) > p(l>τ |l≤τ ) and p(g≤τ )p(g>τ |g≤τ ) > p(l≤τ )p(l>τ |l≤τ ). This often leads beam search to get stuck in the local optimum from step τ onward in generating texts.
  ∗corresponding author 2https://github.com/yema2018/global_aware
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  To cope with this limitation, this study proposes a calibrated beam-based algorithm with global awareness at all searching steps. Generally, our novel algorithm is implemented in two phases. Before the beam search (Phase I), the global attention distribution is predicted in order to be included as a protocol to calibrate beam search at each step, encouraging the generated hypotheses to attend to the source in a more near-global optimal way. Specifically, the global attention distribution describes how all reference tokens should assign the attention to each source token (illustrated in Figure 1.a), which could be predicted from the source by training an attention-prediction model. The training is fairly straightforward and resembles a sequence tagging task [12], except that the predicted attention distribution from the source is a regression result. There are several advantages of using the attention distribution as the global protocol. 1) Attention distributions are sensitive to the decoder input, suggesting that any input to the decoder leads to a unique attention distribution with fixed model parameters; 2) attention distributions are accessible for almost all text generation tasks thanks to the recent advances in attention models [27, 4, 31]; 3) relying on the source only, the global attention distribution can be predicted before beam search, thus offering a rigorous mechanism to calibrate a global-aware beam search.
  During beam search (Phase II), we develop a novel global scoring mechanism composed of attention scores and length rewards to guide beam search based on the predicted global attention distribution. As one main theoretical result, we show that the attention score can be considered as the probability that generated texts attend to sources in line with the predicted global attention distribution. Specifically, the generated tokens in each step update the local attention distribution to source tokens dynamically, where the attention values grow monotonically as the generation proceeds (see Figure 1.b). Since the desired situation is that the local distribution reaches exactly the global distribution at the terminal step, we regulate the inference by discouraging local attention from exceeding their corresponding predicted global attention at all steps.
  With regards to the core target to investigate the possible paradigm that improves beam search with global awareness during decoding, contributions of this study are summarized as follows:
  • We argue that the limitation of beam search roots from its defect in finding the global optimal hypothesis. We improve the algorithm by proposing a global protocol to regulate beam search step-by-step. This paper is the first to predict and deploy the global attention distribution to calibrate the inference in a rigorous way, thus returning a hypothesis that attends to source tokens in a more near-global optimal manner. In contrast, previous works [37, 7, 20, 29, 21] try to use attention distributions to improve beam search, but ignore that the global attention distribution is predictable.
  • A novel global scoring mechanism is designed to evaluate the generated sequences at each step based on the desired situation described in Figure 1.c. As theoretically justified, its major component can be elegantly integrated into beam search in the form of a probability so that merely O(K) of the time complexity is increased in each step (see Section § 3.1 for more details).
  • The proposed algorithm with global awareness manifests a robust and plug-and-play property in enhancing beam search for neural abstractive summarization. Without requiring any model or parameter modification, the global-aware inference shows excellent performance in generating meaningful texts, even if the attention distribution is corrupted or not of its own. Further, it is identified that summaries generated by global-aware inference are both higher-quality and different from beam search hypotheses (see Global-aware in Table 1). More interestingly, we find that the generation style of a dataset could be transferred by the designated global attention distribution. For instance, summaries of higher abstractness for CNN/DM could be generated by only replacing its global attention distribution with a highly abstractive distribution during inference, as presented in Global-aware† of Table 1.
  • On the empirical side, we show that the proposed global-aware inference can stably and significantly boost two state-of-the-art summarization models BART [17] and PEGASUS [39] to produce higher quality summaries on nine datasets, even if only the empirical hyper-parameters are used.
sections.heading: 2 Preliminary
sections.text: The proposed decoding strategy is applied in BART [17] and PEGASUS [39] to perform summarization. BART is a pre-trained seq-to-seq model whose structure essentially follows a vanilla Transformer encoder-decoder [31]. PEGASUS has a similar structure but is pre-trained on a larger dataset differently. Notably, the fine-tuned parameters of both models are downloaded from HuggingFace Models3 and are fixed in all subsequent operations, where “fine-tuned" means the pre-trained model has been fine-tuned on a specific dataset.
sections.heading: 2.1 Beam Search
sections.text: Since the decoder of the seq-to-seq model is an auto-regressive model, the probability of a target sequence y = (y0, · · · , yt, · · · , yT ) can be factorized to the probabilities conditional on the source x = (x1, · · · , xi, · · · , xn) and y<t = (y0, · · · , yt−1), i.e.,
  p(y|x) = T∏ t=1 p(yt|x,y<t), T ≥ 1 (1)
  When T = 0, y = (y0) and p(y0|x) = 1 because y0 is a fixed start token. Beam search [8] is a decoding strategy to predict a target sequence by maximizing this factorization. Given a vocabulary set V , at each inference step t, beam search selects a candidate beam set BKt = {bkt }Kk=1 (where each beam bkt = (b k 0 , · · · , bkt ) is a candidate sequence) from an all-possible beam set Bt of size K × |V|, namely, Bt = { bkt−1 ◦ v | bkt−1 ∈ BKt−1, v ∈ V } (2)
  3https://huggingface.co/models
  BKt = { bkt | bkt = argtopk (log p(bt|x)) , bt ∈ Bt } , t ≥ 1 (3)
  where argtopk(·) outputs K beams with the highest conditional probability, and ◦ is the concatenation operation. Besides, BK0 = {bk0}Kk=1 where bk0 is the start token. By Eq. 1, log p(bt|x) is an accumulated value. Its calculation can be simplified as:
  log p(bt|x) = { log p(bkt−1|x) + log p(v|x, bkt−1), t ≥ 2 log p(v|x, bk0), t = 1
  (4)
  where the value of log p(bkt−1|x) is computed from the previous step. Therefore, at each step, we only need calculate the condition probability of each token in the vocabulary set.
  A beam is terminated after it generates an ending token, and the beam set of K terminated beams is defined as Y . The final hypothesis y∗ is chosen from Y based on the beam probability normalized by lengtha where a is a hyper-parameter of length [37]:
  y∗ = argmax yk∈Y log p(yk | x) (|yk| − 1)a
  (5)
  where yk = (yk0 , · · · , ykT ). |yk| − 1 is used since the start token is not considered in calculating the length.
sections.heading: 2.2 Attention Distribution
sections.text: Attention distribution is a continuous vector whose element indicates the degree of attention paid to a source token. The element is formed by the accumulation of cross attention, i.e., ∑ t αt,i, where αt,i refers to the cross attention that the tth token in the target sequence gives to the ith source token.4 Specially, cross attention is a scaled dot-product [31] of hidden states of the source x and the target sequence y. Notably, since Transformer-decoder is an auto-regressive model, the cross attention assigned by tth target token is actually calculated by y<t = (y0, · · · , yt−1). Global Attention Distribution. The global attention distribution g = [g1, · · · , gi, · · · , gn] ∈ Rn is the attention distribution given by the reference, where global attention gi refers to the total attention that the reference attends to the ith source token, and n is the source length.
  Optimal Length. The summation of g, namely ∑n i=1 gi, is equal to ∑n i=1 ∑T t=1 αt,i = T due to∑n
  i=1 αt,i = 1 in softmax operation, where T is the reference length, or equivalently the optimal length Z.
  Local Attention Distribution. The local attention distribution lkt = [lkt,1, · · · , lkt,i, · · · , lkt,n] ∈ Rn is the attention distribution of the kth generated sequence and updated at each decoding step t. Thereinto, the local attention lkt,i denotes the total attention paid to the ith source token by the kth beam sequence (bk1 , · · · , bkt ) and is dependent on the sequence generated before t, i.e., bkt−1 = (bk0 , · · · , bkt−1).
sections.heading: 3 Proposed Global-aware Inference
sections.heading: 3.1 Global Scoring Mechanism
sections.text: The global scoring mechanism consists of an attention scoring function and a length reward function. Given the global attention distribution g, the attention scoring function A(·) at the step t depends on bkt−1,
  A(bkt−1) = ∑n i=1 min(l k t,i, gi)
  ζkt , ζkt = n∑ i=1 lkt,i, t ≥ 1 (6)
  where ζkt indicates the total attention that the generated sequence (b k 1 , · · · , bkt ) gives to the source, and ζkt = |bkt−1| = t because the assignable attention for each generated token is 1. Notably, Eq. 6 attains the maximum score provided that each lt,i ≤ gi. As mentioned in Section § 1, the reason for
  4Mean pooling is used for multi-layers and multi-heads
  this design is that we desire the local attention lT,i at the termination is exactly gi, since the final hypothesis is expected to attend to source tokens in the global-optimal manner. Meanwhile, we have lT,i > lt,i for t < T because lt,i monotonically increases on t with αlm,i > 0. Therefore, at any step, the local attention lt,i should not surpass gi. Otherwise, the attention score will decline, and the penalty depends on the total amount by which these lt,i exceed gi (see Theorem 1). Further, the attention score could be considered as the proportion of correctly assigned attention to the total attention given by the generated sequence, where correct assignment indicates that all parts of lt,i do not exceed gi. Also, it could be interpreted as the correct allocation probability of local attention against the global attention distribution (see Corollary 1.1). In this case, the total attention score can be expressed as the same multiplicative form as Eq. 1 to be elegantly integrated into beam search. Theorem 1. Let M = { s : lkt,s > gs } and ∆kt = { δ | ∀s ∈M, δ = lkt,s − gs } . Given ∆ =∑
  δ∈∆kt δ where ∆ ≥ 0, then A(bkt−1) decreases as ∆ increases.
  Corollary 1.1. The bound of A(bkt−1) is between 0 and 1.
  Proof. See App. A.
  In addition to the constraint for lt,i, we still desire lT,i = gi for each token. An ideal hypothesis should have two characteristics simultaneously. Namely, its attention score at the termination is the maximum 1, and its length equals the optimal length. Therefore, we introduce a length reward function to cooperate with the attention score to penalize the situation lT,i 6= gi, which will be discussed at the end of this subsection.
  As mentioned before, the total attention score at the decoding step t is defined as:
  A(bkt−1) = t∏ m=1 A(bkm−1) (7)
  Thus, the joint scoring function J(·) is modified from Eq. 4: J(bt,x) = log p(b k t−1|x) + β logA(bkt−1) + log p(v|x, bkt−1)
  = t−1∑ m=1 ( log p(bm|x, (bkt−1)<m) + β logA(bkm−1) ) + log p(v|x, bkt−1) + β logA(bkt−1)
  = J(bkt−1,x) + log p(v|x, bkt−1) + β logA(bkt−1), t ≥ 2 (8)
  and J(b1,x) = log p(v|x, bk0) + β logA(bk0), where β is a hyper-parameter to trade-off between the probability and attention score. Similar to log p(bt|x) in Eq. 4, J(bt,x) is also an accumulative score. Consequently, at each step t, we only need compute p(v|x, bkt−1) and A(bkt−1). Compared with Eq. 4, the time complexity of each step is only increased by O(K) as there are K attention scores. Replacing log p(bt|x) in Eq. 3 by J(bt,x), we can select the top K beams of each decoding step according to not only the probability distribution conditional on local information bkt−1 but also the score conditional on global information g.
  Considering the length reward function, the final hypothesis is thus defined as:
  y∗ = argmax yk∈Y
  ( J(yk,x)
  |yk| − 1 + βγR(ζkT , Z)
  ) (9)
  where R(·) is the length reward function dependent on the optimal length Z and the candidate hypothesis length ζkT . Exactly, ζ k T is the total attention that a candidate hypothesis (y k 1 , · · · , ykT ) pays to the source and equals |yk| − 1. Besides, the attention score and length reward are weighted by a hyper-parameter γ, and the role of β is to ensure that the two are at the same level relative to the probability. We remove a in Eq. 5 as it only adjusts the length preference without really controlling the length.
  The design of R(·) could be straightforward – one only need ensure that it increases as ζkT approaches Z, and reaches the maximum only at ζkT = Z. In this paper, we design a step-wise length reward functionR(ζkt , Z) to better balance the relationship between the attention score and the length reward and make the whole searching process as succinct as beam search. We put the design details of the step-wise length reward in App. B, and we regard Eq. 9 as the general scoring formulation of global-aware inference.
sections.heading: 3.2 Predict the Global Attention Distribution
sections.text: Since the reference is unknown practically, the global attention distribution could only be predicted from the source. We construct an attention-prediction model to learn the relationship between the source tokens and the global attention distribution.
  The input of the attention-prediction model is the fixed encoder output E ∈ Rn×d of BART or PEGASUS plus learnable positional encodings P ∈ Rn×d, where d is the dimension of hidden states. The input is fed to a learnable Transformer-encoder to obtain Ẽ ∈ Rn×d that is encoded with additional context information, followed by a linear transformation with an exponential function:
  ĝ = exp ( ẼWg + bg ) (10)
  where ĝ ∈ Rn refers to the prediction of g, Wg ∈ Rd×1 and bg ∈ Rn are the learnable weights and biases. The exponential function is imposed to ensure ĝ > 0. We choose the exponential function for this operation because it is shown stable in the training and testing stage. Given the objective of minimizing the distance between ĝ and g, the loss is defined as their Euclidean distance:
  L = ‖ĝ − g‖2 (11)
  The predicted optimal length Ẑ is the sum of elements in ĝ. Note that the length reward function is not affected no matter whether Ẑ is an integer or not.
sections.heading: 4 Experiment
sections.heading: 4.1 Setup
sections.text: Datasets. We evaluate the performance on totally 9 summarization datasets, where 2 datasets (CNN/DM [10], XSUM [5]) with BART [17] and 8 datasets (XSUM [5], BillSum [15], Multi-News [6], NewsRoom [9], WikiHow [16], Reddit TIFU [34], arXiv and PubMed [2]) with PEGASUS [39]. Thereinto, XSUM [5] is a highly abstractive dataset whose summaries are all expressed in a short sentence.
  Implementation Details. We adopt a randomly initialized 2-layer transformer-encoder in the attention-prediction model wherethe structure of each layer is the same as the BART-encoder layer. The optimizer is the Adabelief-optimizer [41] with eps 1e − 16, betas (0.9, 0.999), weight decay 1e − 4 and learning rate 2e − 5. The attention-prediction model is trained on the training set for about 50, 000 steps, and checkpoints are saved per 10, 000 steps to select the best checkpoints on the development set. Since the attention prediction is slightly different from common sequence tagging tasks, we have summarized two notable points after several attempts – the dropout rate should be 0, and a small learning rate is preferred. All experiments are conducted on 3 RTX 6000. We include the global-aware inference in the generation code of HuggingFace transformers [36]. At the time of evaluation, ROUGE-1, ROUGE-2 & ROUGE-L (R-1, R-2 & R-L) scores [22] are computed from the ROUGE code5 used by BART [17].
  Hyper-parameter Selection. Although the global-aware inference requires two new hyperparameters γ and β, some original hyper-parameters of beam search, namely length penalty, minimum and maximum length, are omitted. The searching scopes of β and γ are in {2, 4, 6, 10, 12, 15, 18, 20} and {0, 0.5, 1, 1.5, 2}, respectively. According to the numerical tests on the development set, we finally choose β = 12, γ = 1.5 for CNN/DM and β = 4, γ = 0 for XSUM. As limited improvement could be observed from larger γ’s, we recommend γ = 1 for normal or longer targets. When testing the global-aware inference with PEGASUS [39], we directly use empirical hyper-parameters for each dataset, namely β = 4, γ = 0 for XSUM and β = 12, γ = 1 for other 7 datasets. The beam size K follows the setups in BART [17] and PEGASUS [39].
sections.heading: 4.2 Results
sections.text: Comparison with Beam Search. Beam search is a hard-to-beat baseline which has stood the test of time and proven its superiority in practice for long [24]. In Table 2, we compare our global-aware
  5https://github.com/pltrdy/files2rouge
  inference to beam search with length regularizations (i.e., α in Eq. 5, accompanied with two hard constraints, namely minimum length and maximum length). We strictly follow the hyper-parameter setups of PEGASUS [39] in terms of beam search, while we only adopt empirical hyper-parameters for our method. Even so, significant improvements can be observed on all the data sets, especially when the summary is of normal or longer length.
  Table 3: Comparison with other methods.
  CNN/DM K = 4 R-1 R-2 R-L
  Beam search [17] 44.12 21.21 40.89 + Our coverage 44.74 21.69 41.48 + Repetition penalty [13] 44.11 21.14 40.87 + Attention masking [1] 45.54 22.24 42.44
  Global-aware 45.13 21.77 42.04
  XSUM K = 6 R-1 R-2 R-L
  Beam search [17] 45.38 22.32 37.15 + Our coverage 44.54 21.82 36.97 + Repetition penalty [13] 45.40 22.31 37.13 + Attention masking [1] 45.35 22.31 37.15
  Global-aware 45.57 22.60 37.61
  Table 4: ORACLE and ablation results.
  CNN/DM R-1 R-2 R-L
  ORACLE global-aware 51.85 28.13 48.68 -w/o length reward 50.46 27.53 47.43
  Global-aware 45.13 21.77 42.04 -w/o length reward 44.39 21.58 41.41 -w/o attention score 44.12 21.29 40.91
  XSUM R-1 R-2 R-L
  ORACLE global-aware 49.50 26.24 41.13 -w/o length reward 48.92 26.48 41.45
  Global-aware (γ = 1) 45.44 22.15 37.11 -w/o length reward 45.57 22.60 37.61 -w/o attention score 45.23 21.88 36.73
  Comparison with Other Attention Approaches. In this part, we focus on comparing other approaches which also exploit certain attention distributions to improve beam search. The first is the coverage penalty [37, 21]. To enhance its performance in summarization, we replace its preset attention distribution with our predicted global attention distribution. Note that the coverage function can only evaluate the generated sentences at the terminal step. Instead of comparing the global-aware inference to the methods [29, 20, 7] that aim to reduce repetition using the dynamic attention distribution, we compare our algorithm with the CTRL repetition penalty [13] which has similar motivation but is more systematical and independent of training. Table 3 lists the comparison results against different algorithms. It can be observed that our global-aware approach can improve the performance of beam search stably and fundamentally. We also observe that the attention head masking [1] appears to outperform the global-aware approach on CNN/DM, but it fails to gain any improvement on XSUM. To further show the advantage of the proposed approach, we will take a closer examination on the attention head masking [1] and our proposed approach in the next part.
  Further Comparison with Attention Head Masking. First, one should bear in mind that the attention head masking [1] acts on the model instead of beam search, which is opposite to us. Specifically, it selects contents during decoding by disabling partial attention heads for unimportant
  Table 7: Generate summaries with corrupted attention distributions.
  R-1 R-2 R-L
  Beam search 27.00 12.21 23.88 Global-aware (10K) 28.78 12.82 25.51 Global-aware (100K) 29.59 13.72 26.30
  Figure 2: Sensitive analysis in the test set.
  2 4 6 10 12 15 18 20 40.5
  41
  41.5
  42
  R -L
  f1 s
  co re
  s
  CNN/DM, =1
  global-aware beam search
  0 0.5 1 1.5 2 40.5
  41
  41.5
  42
  R -L
  f1 s
  co re
  s
  CNN/DM, =12
  2 4 6 10 12 15 18 20 37
  37.2
  37.4
  37.6
  R -L
  f1 s
  co re
  s
  XSUM, =0
  0 0.5 1 1.5 2 37
  37.2
  37.4
  37.6
  R -L
  f1 s
  co re
  s
  XSUM, =4
  tokens to decrease the attention to these tokens. According to the reported results presented in Table 3, we can see that although attention masking achieves amazing results on CNN/DM, it does fail completely on XSUM. Since hiding unimportant tokens from some heads results in the loss of context information of salient tokens, this would lead to its instability. Thus, it could be ineffective for tasks that require contextual information of the whole source such as XSUM. Taking a further comparison, we deploy the attention head combinations selected for CNN/DM and XSUM to examine its effect on PEGASUS [39]. These comparison results are shown in Table 5. Evidently, our method enjoys a robust feature that is able to boost summary inference on various datasets and models even with the same set of hyper-parameters. In contrast, attention masking [1] behaves much sensitive to the changes of models and datasets. Besides, attention masking has to construct its training saliency labels based on the longest common subsequences between a reference summary and the source. This may be hardly achieved in some text generation tasks (e.g. translation) where no common subsequence exists at all. Such drawback presents one main limitation for attention masking.
  ORACLE Results and Ablation Study. ORACLE refers to the global-aware inference combined with (true) global attention distribution instead of predicted global attention distribution. The related results have been presented in Table 4, and the significant boosting certifies that the proposed method could improve beam search with the global attention distribution. On the other hand, we conduct ablation experiments on ORACLE and global-aware. Both results indicate that length reward plays an important role in generating normal-length text but causes adverse effect on generating text of very short length. Besides, the performance declines significantly when only length reward is applied, due to the fact that sole length reward cannot calibrate beam search step-wise.
  Robustness. We intend to examine the robustness of our proposed global-aware algorithm in this and next part. To do so, we substitute the parameters in CNN/DM’s attention prediction model to XSUM’s to create more abstractive summaries for CNN/DM. For comparison, we set the minimum length of beam search as 0 to allow it to generate shorter summaries. Table 6 shows the F1 score, recall and precision of the shorter beam search and global-aware†. It is surprising that even by using the attention distribution from a different dataset with distinct properties, the proposed global-aware mechanism still manages to generate meaningful summaries with competitive F1 scores, proving the robustness of this algorithm. Moreover, the higher Precision and lower Recall of the global-aware suggest that although information is partially lost, the algorithm still summarizes core information in a concise format, compared to the standard beam search. On the other hand, we exploit a BART model fine-tuned on CNN/DM to generate summaries of NewsRoom directly, and the ROUGE scores of beam search are shown in Table 7. Next, we randomly select 10K and 100K samples from the training set and use them to fine-tune the attention-prediction model, where the global-aware improves beam search substantially. The experiment once again validates the robustness of the proposed inference algorithm, as it maintains a reasonably good performance even from learning a corrupted attention distribution from a BART without fine-tuning.
  Sensitive Analysis. We further examine the performance robustness with respect to the hyperparameters. From Figure 2, we could see the global-aware inference is always better than beam search in CNN/DM, no matter how its hyper-parameters change. Besides, the performance is less sensitive to the hyper-parameters when β ≥ 10 or γ ≥ 1. While in XSUM, the global-aware could improve beam search stably with γ = 0, but there is a significant decline when applying length
  reward. In fact, the attention score favors shorter hypotheses, and the length reward could alleviate the bias. However, if the references of a dataset are already very short such as XSUM, the length reward may lead to a counterproductive effect. Since the setup of CNN/DM is applicable to most datasets, we argue that the global-aware inference is robust to both hyper-parameters in most cases.
sections.heading: 5 Analysis on Global Attention Distribution
sections.heading: 5.1 Distribution Law
sections.text: The distribution law of global attention in BART [17] is shown in Figure 3. It is observed that most attention is assigned to the start token and full stops which are not semantically salient, and most of the remaining attention is unevenly allocated to (semantically) overlapped tokens between the source and the reference (i.e., important words). It is worth mentioning that the importance here is no longer a simple binary classification like in [7, 1], but a continuous numerical value decided by the model knowledge learned from data. In general, one should not simply equate the global attention with the word importance, but should be clear that it essentially reflects knowledge learned by the attention model such as the syntactic and semantic structure of sentences. Meanwhile, the distribution law indicates that attention distributions in pre-trained models may not be relevant with the uniform distribution at all. That is to say, it is not reasonable to still use an uniform attention threshold (like the threshold 1 preset in [37, 7]) to regulate the decoding of pre-trained models. Last but not least, our general motivation is to alleviate locality bias by aligning the attention distribution of reference and hypothesis, which does not really care how the global attention is distributed only if it is predictable. However, the proposed penalty mechanism is indeed insensitive to some distributions, and we will provide more thinking about this in App. G by applying the global-aware inference to translation.
sections.heading: 5.2 Why It can be Predicted from Source?
sections.text: Since the ORACLE experiments indicated that the global attention is helpful for decoding, the concern remains if it is predictable using only source. In App. E.1 we will show the predictability empirically, while here we just provide an interesting explanation. In our opinion, the global attention distribution is an interpretable representation of reference which not only has the characteristics of hidden representation but also can be explained by the source tokens. First of all, given the source, the global attention is calculated by the input reference and trained neural network parameters;
  this is similar to achieving hidden representation. Moreover, like hidden representation, the global attention distribution could also capture the semantic similarity, e.g., replacing a reference word with a semantically similar word leads to slighter attention changes than that with a random word (see Figure 4.a). Besides, it is observed from Figure 4.b and Figure 4.c that global attention is able to represent the sentence order and factual knowledge. On the other hand, the global attention distribution enjoys a distinct characteristic that each of its features can be explained as the degree of attention paid to a source token, which means changes of such representation are regular to some extent. For example, in Figure 4.c, we distort a numerical number in the reference to violate the fact stated in the source, and then find that the attention assigned to the actual number originally is most transferred to the special tokens and punctuation. Overall, similar sources should have similar global attention distributions, since similar sources often have similar references and global attention distribution is a representation of reference. Moreover, the global attention and source tokens are in an one-to-one correspondence. Thereby, we argue that it is convenient to use the source to predict the global attention distribution.
sections.heading: 6 Related Work
sections.text: In the field of text generation, efforts have been made to boost beam search with regards to the attention distribution. For instance, some studies engage the attention distribution to penalize the situation where sources fail to be fully covered in translation tasks [37, 21], while others [7, 29, 20] incorporate dynamic attention distributions to evade tokens that have been highly regarded to reduce repetition. However, none of the aforementioned studies attempts to apply the global attention distribution to acquire the knowledge that the level of attention should be allocated to each token from the reference. Further, the existing score functions used by those studies are rather different from the proposed global scoring mechanism. More details can be seen in App. D.
  In addition to the attention distribution, other techniques are developed to improve beam search in terms of length bias [38], diversity-less [32], vapidity [11] and degradation [25, 3]. These methods are not included for comparison because they are not suitable for summarization [38, 11], or do not aim to enhance beam search as the main purpose [32, 25, 3]. Besides, we argue that these patches to beam search are supposed to be hard for improving the performance stably and fundamentally (as shown in Table 3 given by our global-aware method) because they fail to specify what a final hypothesis should look like and are easy to trap into local optimums.
sections.heading: 7 Conclusion
sections.text: Beam search tends to fall into local optimums due to the lack of global information during inference. To calibrate beam search on the premise of global awareness, this study proposes a predictable protocol to stipulate how a global optimal hypothesis should attend to source tokens. By training a simple prediction model of the global attention distribution, a novel global scoring mechanism is then developed to regulate the inference on a step-wise basis. Our experiment shows that the proposed global-aware beam search generates higher-quality summaries, relative to beam search and other counterparts of the similar nature. Besides, the proposed algorithm is proven robust in generating meaningful texts even with corrupted attention distributions, implying its potential to cooperate with user-defined global attention distributions. We plan to focus our future study on generalizing the global-aware inference to a broader range of text generation tasks, including not only text2text but also image caption [33], multimodal summarization [19], graph2text [14] and data2text [26].
  Acknowledgments and Disclosure of Funding
  Funding in direct support of this work: XJTLU Key Programme Special Fund KSF-A-14 and KSFP-02, National Natural Science Foundation of China under No. 61876155 and Jiangsu Science and Technology Programme under No. BE2020006-4.
title: Global-aware Beam Search for Neural Abstractive Summarization