Inspired by Google's Neural Machine Translation (NMT) \mbox{%DIFAUXCMD Wu2016Google that models the one-to-one alignment in translation tasks with an optimal uniform attention distribution during the inference, this study proposes an attention-aware inference algorithm for Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source paragraphs/sentences with the optimal coverage. Unlike NMT, the attention-aware inference of NAS requires the prediction of the optimal attention distribution. Therefore, an attention-prediction model is constructed to learn the dependency between attention weights and sources. To apply the attention-aware inference on multi-document summarization, a Hierarchical Transformer (HT) is developed to accept lengthy inputs at the same time project cross-document information. Experiments on WikiSum \mbox{%DIFAUXCMD liu2018generating By refining the regular beam search with the attention-aware inference , significant improvements on the quality of summaries could be further observed. Last but not the least, the attention-aware inference could be adopted to single-document summarization with straightforward modifications according to the model architecture .
Inspired by Google's Neural Machine Translation (NMT) that models the one-to-one alignment in translation tasks with an uniform attention distribution during the inference, this study proposes an attention-aware inference algorithm for Neural Abstractive Summarization (NAS) to regulate generated summaries to attend to source contents with the optimal coverage. Unlike NMT, NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct an attention-prediction model to learn the dependency between the optimal attention distribution and the source. By refining the vanilla beam search with the attention-aware mechanism , significant improvements on the quality of summaries could be observed. Last but not the least, the attention-aware inference has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance .
D
\mbox{%DIFAUXCMD Wu2016Google
D
optimal
R
paragraphs/sentences
contents
R
the attention-aware inference of NAS requires the prediction of the optimal attention distribution. Therefore,
NAS is not based on one-to-one transformation. Instead, its attention distribution for the input should be irregular and depend on the content layout of the source documents. To address this matter, we construct
D
is constructed
R
attention weights and sources. To apply the attention-aware inference on multi-document summarization, a Hierarchical Transformer (HT) is developed to accept lengthy inputs at the same time project cross-document information. Experiments on WikiSum \mbox{%DIFAUXCMD liu2018generating
the optimal attention distribution and the source.
R
regular
vanilla
R
inference
mechanism
D
further
R
could be adopted to single-document summarization with straightforward modifications according to the model architecture
has strong universality that can be easily adopted to different hierarchical summarization models to promote the models' performance