Inspired by Google's Neural Machine Translation (NMT) \mbox{%DIFAUXCMD Wu2016Google that models the one-to-one alignment in translation tasks with an optimal uniform attention distribution during the inference, this study proposes an attention-aware inference algorithm for Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source paragraphs/sentences with the optimal coverage. Unlike NMT, the attention-aware inference of NAS requires the prediction of the optimal attention distribution. Therefore, an attention-prediction model is constructed to learn the dependency between attention weights and sources. To apply the attention-aware inference on multi-document summarization, a Hierarchical Transformer (HT) is developed to accept lengthy inputs at the same time project cross-document information. Experiments on WikiSum \mbox{%DIFAUXCMD liu2018generating By refining the regular beam search with the attention-aware inference , significant improvements on the quality of summaries could be further observed. Last but not the least, the attention-aware inference could be adopted to single-document summarization with straightforward modifications according to the model architecture .
Inspired by Google's Neural Machine Translation (NMT) that models the one-to-one alignment in translation tasks with an uniform attention distribution during the inference, this study proposes an attention-aware inference algorithm for Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source contents with the optimal coverage. Unlike NMT, NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct an attention-prediction model to learn the dependency between the optimal attention distribution and the source. By refining the vanilla beam search with the attention-aware mechanism , significant improvements on the quality of summaries could be observed. Last but not the least, the attention-aware inference has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance .
D
\mbox{%DIFAUXCMD Wu2016Google
style
style style style
D
optimal
clarity
clarity clarity clarity
R
paragraphs/sentences
contents
meaning-changed
meaning-changed meaning-changed meaning-changed
R
the attention-aware inference of NAS requires the prediction of the optimal attention distribution. Therefore,
NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct
clarity
clarity clarity clarity
D
is constructed
coherence
coherence coherence coherence
R
attention weights and sources. To apply the attention-aware inference on multi-document summarization, a Hierarchical Transformer (HT) is developed to accept lengthy inputs at the same time project cross-document information. Experiments on WikiSum \mbox{%DIFAUXCMD liu2018generating
the optimal attention distribution and the source.
clarity
clarity clarity clarity
R
regular
vanilla
style
style style style
R
inference
mechanism
style
style style style
D
further
style
style style style
R
could be adopted to single-document summarization with straightforward modifications according to the model architecture
has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance
style
style style style