This paper proposes a novel inference algorithm for seq-to-seq models. It corrects beam search step-by-step via the optimal attention distribution to make the generated text attend to source tokens in a controlled way. Experiments show the proposed attention-aware inferenceproduces summaries rather differently from the beam search , and achieves promising improvements of higher scores and greater conciseness. The algorithm is also proven robust as it remains to outperform beam search significantly even with corrupted attention distributions  .
This study develops a calibrated beam-based algorithm with global awareness for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring function is then developed to regulate beam search to generate summaries in a more near-global optimal fashion. This novel design enjoys a distinctive property, i.e. the global attention distribution could be predicted before inference, enabling stepwise improvements on the beam search through the global scoring function. Extensive experiments on 9 datasets show that the global-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions . The codes and a comprehensive set of examples are available .
R
paper proposes a novel inference algorithm for seq-to-seq models. It corrects beam search step-by-step via the optimal
study develops a calibrated beam-based algorithm with global awareness for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the
R
make the generated text attend to source tokens in a controlled way. Experiments show the proposed attention-aware inferenceproduces summaries rather differently from
stipulate how a global optimal hypothesis should attend to the source. A global scoring function is then developed to regulate beam search to generate summaries in a more near-global optimal fashion. This novel design enjoys a distinctive property, i.e. the global attention distribution could be predicted before inference, enabling stepwise improvements on
R
, and achieves promising improvements of higher scores and greater conciseness.
through the global scoring function. Extensive experiments on 9 datasets show that the global-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters.
R
outperform beam search significantly even
generate meaningful texts
A
. The codes and a comprehensive set of examples are available