How does their BERT-based model work?
It takes the query and document as input and encodes the query relevance, document context and salient meaning to be passed to the output layer to make the prediction.
The manual construction of a query-focused summarization corpus is costly and timeconsuming. The limited size of existing datasets renders training data-driven summarization models challenging. In this paper, we use Wikipedia to automatically collect a large query-focused summarization dataset (named as WIKIREF) of more than 280,000 examples, which can serve as a means of data augmentation. Moreover, we develop a query-focused summarization model based on BERT to extract summaries from the documents. Experimental results on three DUC benchmarks show that the model pre-trained on WIKIREF has already achieved reasonable performance. After fine-tuning on the specific datasets, the model with data augmentation outperforms the state of the art on the benchmarks.
Query-focused summarization aims to create a brief, well-organized and informative summary for a document with specifications described in the query. Various unsupervised methods BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5 and supervised methods BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 have been proposed for the purpose. The task is first introduced in DUC 2005 BIBREF11, with human annotated data released until 2007. The DUC benchmark datasets are of high quality. But the limited size renders training query-focused summarization models challenging, especially for the data-driven methods. Meanwhile, manually constructing a large-scale query-focused summarization dataset is quite costly and time-consuming. In order to advance query-focused summarization with limited data, we improve the summarization model with data augmentation. Specifically, we transform Wikipedia into a large-scale query-focused summarization dataset (named as WikiRef). To automatically construct query-focused summarization examples using Wikipedia, the statements' citations in Wikipedia articles as pivots to align the queries and documents. Figure FIGREF1 shows an example that is constructed by the proposed method. We first take the highlighted statement as the summary. Its supporting citation is expected to provide an adequate context to derive the statement, thus can serve as the source document. On the other hand, the section titles give a hint about which aspect of the document is the summary's focus. Therefore, we use the article title and the section titles of the statement to form the query. Given that Wikipedia is the largest online encyclopedia, we can automatically construct massive query-focused summarization examples. Most systems on the DUC benchmark are extractive summarization models. These systems are usually decomposed into two subtasks, i.e., sentence scoring and sentence selection. Sentence scoring aims to measure query relevance and sentence salience for each sentence, which mainly adopts feature-based methods BIBREF0, BIBREF7, BIBREF3. Sentence selection is used to generate the final summary with the minimal redundancy by selecting highest ranking sentences one by one. In this paper, we develop a BERT-based model for query-focused extractive summarization. The model takes the concatenation of the query and the document as input. The query-sentence and sentence-sentence relationships are jointly modeled by the self-attention mechanism BIBREF12. The model is fine-tuned to utilize the general language representations of BERT BIBREF13. Experimental results on three DUC benchmarks show that the model achieves competitive performance by fine-tuning and outperforms previous state-of-the-art summarization models with data augmentation. Meanwhile, the results demonstrate that we can use WikiRef as a large-scale dataset to advance query-focused summarization research.