Inspired by Google's Neural Machine Translation (NMT) that models the one-to-one alignment in translation tasks with an uniform attention distribution during the inference, this study proposes an attention-aware inference algorithm for Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source contents with the optimal coverage. Unlike NMT, NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct an attention-prediction model to learn the dependency between the optimal attention distribution and the source . By refining the vanilla beam searchwith the attention-aware mechanism, significant improvements on the quality of summaries could be observed. Last but not the least, the attention-aware inference has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance .
This paper proposes a novel inference algorithm for seq-to-seq models. It corrects beam search step-by-step via the optimal attention distribution to make the generated text attend to source tokens in a controlled way. Experiments show the proposed attention-aware inference produces summaries rather differently from the beam search, and achieves promising improvements of higher scores and greater conciseness. The algorithm is also proven robust as it remains to outperform beam search significantly even with corrupted attention distributions .
R
Inspired by Google's Neural Machine Translation (NMT) that models the one-to-one alignment in translation tasks with an uniform attention distribution during the inference, this study proposes an attention-aware
This paper proposes a novel
R
Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source contents with the optimal coverage. Unlike NMT, NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct an attention-prediction model to learn the dependency between the optimal attention distribution and the source . By refining the vanilla beam searchwith the
seq-to-seq models. It corrects beam search step-by-step via the optimal attention distribution to make the generated text attend to source tokens in a controlled way. Experiments show the proposed
R
mechanism, significant improvements on the quality of summaries could be observed. Last but not the least, the attention-aware inference has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance
inference produces summaries rather differently from the beam search, and achieves promising improvements of higher scores and greater conciseness. The algorithm is also proven robust as it remains to outperform beam search significantly even with corrupted attention distributions