query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : As a researcher focusing on long text summarization, I am keen to understand the key features that contribute to a high-quality summary. Which metrics would be most useful in assessing the quality of a summary? I intend to explore the correlation between the quality of summarization and certain NLP features, such as word diversity, entropy, and perplexity. I am curious to know if similar studies have been conducted previously. Furthermore, I am interested in examining self-supervised alignment techniques that could assist in aligning the semantic content of a summary with its original text. My plan is to develop a deep summarization model and an alignment model. The alignment model, potentially using an attention mechanism, could help align summaries with longer texts. The quality of this alignment could act as reward signals indicating the quality of the summaries, thereby driving further enhancements in my summarization model. The improved summarization model could, in turn, enhance the quality of the alignment model. This process would be iterative.

positive:
A quality abstractive summary should not only copy salient source texts as
summaries but should also tend to generate new conceptual words to express
concrete details. Inspired by the popular pointer generator
sequence-to-sequence model, this paper presents a concept pointer network for
improving these aspects of abstractive summarization. The network leverages
knowledge-based, context-aware conceptualizations to derive an extended set of
candidate concepts. The model then points to the most appropriate choice using
both the concept set and original source text. This joint approach generates
abstractive summaries with higher-level semantic concepts. The training model
is also optimized in a way that adapts to different data, which is based on a
novel method of distantly-supervised learning guided by reference summaries and
testing set. Overall, the proposed approach provides statistically significant
improvements over several state-of-the-art models on both the DUC-2004 and
Gigaword datasets. A human evaluation of the model's abstractive abilities also
supports the quality of the summaries produced within this framework.

positive:
In a world of proliferating data, the ability to rapidly summarize text is
growing in importance. Automatic summarization of text can be thought of as a
sequence to sequence problem. Another area of natural language processing that
solves a sequence to sequence problem is machine translation, which is rapidly
evolving due to the development of attention-based encoder-decoder networks.
This work applies these modern techniques to abstractive summarization. We
perform analysis on various attention mechanisms for summarization with the
goal of developing an approach and architecture aimed at improving the state of
the art. In particular, we modify and optimize a translation model with
self-attention for generating abstractive sentence summaries. The effectiveness
of this base model along with attention variants is compared and analyzed in
the context of standardized evaluation sets and test metrics. However, we show
that these metrics are limited in their ability to effectively score
abstractive summaries, and propose a new approach based on the intuition that
an abstractive model requires an abstractive evaluation.

positive:
Forum discussion data differ in both structure and properties from generic
form of textual data such as news. Henceforth, summarization techniques should,
in turn, make use of such differences, and craft models that can benefit from
the structural nature of discussion data. In this work, we propose attending to
the beginning of a document, to improve the performance of extractive
summarization models when applied to forum discussion data. Evaluations
demonstrated that with the help of bidirectional attention mechanism, attending
to the beginning of a document (initial comment/post) in a discussion thread,
can introduce a consistent boost in ROUGE scores, as well as introducing a new
State Of The Art (SOTA) ROUGE scores on the forum discussions dataset.
Additionally, we explored whether this hypothesis is extendable to other
generic forms of textual data. We make use of the tendency of introducing
important information early in the text, by attending to the first few
sentences in generic textual data. Evaluations demonstrated that attending to
introductory sentences using bidirectional attention, improves the performance
of extractive summarization models when even applied to more generic form of
textual data.

positive:
Discussions on Twitter involve participation from different communities with
different dialects and it is often necessary to summarize a large number of
posts into a representative sample to provide a synopsis. Yet, any such
representative sample should sufficiently portray the underlying dialect
diversity to present the voices of different participating communities
representing the dialects. Extractive summarization algorithms perform the task
of constructing subsets that succinctly capture the topic of any given set of
posts. However, we observe that there is dialect bias in the summaries
generated by common summarization approaches, i.e., they often return summaries
that under-represent certain dialects.
  The vast majority of existing "fair" summarization approaches require
socially salient attribute labels (in this case, dialect) to ensure that the
generated summary is fair with respect to the socially salient attribute.
Nevertheless, in many applications, these labels do not exist. Furthermore, due
to the ever-evolving nature of dialects in social media, it is unreasonable to
label or accurately infer the dialect of every social media post. To correct
for the dialect bias, we employ a framework that takes an existing text
summarization algorithm as a blackbox and, using a small set of dialect-diverse
sentences, returns a summary that is relatively more dialect-diverse.
Crucially, this approach does not need the posts being summarized to have
dialect labels, ensuring that the diversification process is independent of
dialect classification/identification models. We show the efficacy of our
approach on Twitter datasets containing posts written in dialects used by
different social groups defined by race or gender; in all cases, our approach
leads to improved dialect diversity compared to standard text summarization
approaches.

positive:
Due to the manifold ranking method has a significant effect on the ranking of
unknown data based on known data by using a weighted network, many researchers
use the manifold ranking method to solve the document summarization task.
However, their models only consider the original features but ignore the
semantic features of sentences when they construct the weighted networks for
the manifold ranking method. To solve this problem, we proposed two improved
models based on the manifold ranking method. One is combining the topic model
and manifold ranking method (JTMMR) to solve the document summarization task.
This model not only uses the original feature, but also uses the semantic
feature to represent the document, which can improve the accuracy of the
manifold ranking method. The other one is combining the lifelong topic model
and manifold ranking method (JLTMMR). On the basis of the JTMMR, this model
adds the constraint of knowledge to improve the quality of the topic. At the
same time, we also add the constraint of the relationship between documents to
dig out a better document semantic features. The JTMMR model can improve the
effect of the manifold ranking method by using the better semantic feature.
Experiments show that our models can achieve a better result than other
baseline models for multi-document summarization task. At the same time, our
models also have a good performance on the single document summarization task.
After combining with a few basic surface features, our model significantly
outperforms some model based on deep learning in recent years. After that, we
also do an exploring work for lifelong machine learning by analyzing the effect
of adding feedback. Experiments show that the effect of adding feedback to our
model is significant.

positive:
Document summarization provides an instrument for faster understanding the
collection of text documents and has several real-life applications. With the
growth of online text data, numerous summarization models have been proposed
recently. The Sequence-to-Sequence (Seq2Seq) based neural summarization model
is the most widely used in the summarization field due to its high performance.
This is because semantic information and structure information in the text is
adequately considered when encoding. However, the existing extractive
summarization models pay little attention to and use the central topic
information to assist the generation of summaries, which leads to models not
ensuring the generated summary under the primary topic. A lengthy document can
span several topics, and a single summary cannot do justice to all the topics.
Therefore, the key to generating a high-quality summary is determining the
central topic and building a summary based on it, especially for a long
document. We propose a topic-aware encoding for document summarization to deal
with this issue. This model effectively combines syntactic-level and
topic-level information to build a comprehensive sentence representation.
Specifically, a neural topic model is added in the neural-based sentence-level
representation learning to adequately consider the central topic information
for capturing the critical content in the original document. The experimental
results on three public datasets show that our model outperforms the
state-of-the-art models.

positive:
An accurate abstractive summary of a document should contain all its salient
information and should be logically entailed by the input document. We improve
these important aspects of abstractive summarization via multi-task learning
with the auxiliary tasks of question generation and entailment generation,
where the former teaches the summarization model how to look for salient
questioning-worthy details, and the latter teaches the model how to rewrite a
summary which is a directed-logical subset of the input document. We also
propose novel multi-task architectures with high-level (semantic)
layer-specific sharing across multiple encoder and decoder layers of the three
tasks, as well as soft-sharing mechanisms (and show performance ablations and
analysis examples of each contribution). Overall, we achieve statistically
significant improvements over the state-of-the-art on both the CNN/DailyMail
and Gigaword datasets, as well as on the DUC-2002 transfer setup. We also
present several quantitative and qualitative analysis studies of our model's
learned saliency and entailment skills.

positive:
Extractive summarization aims at selecting a set of indicative sentences from
a source document as a summary that can express the major theme of the
document. A general consensus on extractive summarization is that both
relevance and coverage are critical issues to address. The existing methods
designed to model coverage can be characterized by either reducing redundancy
or increasing diversity in the summary. Maximal margin relevance (MMR) is a
widely-cited method since it takes both relevance and redundancy into account
when generating a summary for a given document. In addition to MMR, there is
only a dearth of research concentrating on reducing redundancy or increasing
diversity for the spoken document summarization task, as far as we are aware.
Motivated by these observations, two major contributions are presented in this
paper. First, in contrast to MMR, which considers coverage by reducing
redundancy, we propose two novel coverage-based methods, which directly
increase diversity. With the proposed methods, a set of representative
sentences, which not only are relevant to the given document but also cover
most of the important sub-themes of the document, can be selected
automatically. Second, we make a step forward to plug in several
document/sentence representation methods into the proposed framework to further
enhance the summarization performance. A series of empirical evaluations
demonstrate the effectiveness of our proposed methods.

positive:
Aligning sentences in a reference summary with their counterparts in source
documents was shown as a useful auxiliary summarization task, notably for
generating training data for salience detection. Despite its assessed utility,
the alignment step was mostly approached with heuristic unsupervised methods,
typically ROUGE-based, and was never independently optimized or evaluated. In
this paper, we propose establishing summary-source alignment as an explicit
task, while introducing two major novelties: (1) applying it at the more
accurate proposition span level, and (2) approaching it as a supervised
classification task. To that end, we created a novel training dataset for
proposition-level alignment, derived automatically from available summarization
evaluation data. In addition, we crowdsourced dev and test datasets, enabling
model development and proper evaluation. Utilizing these data, we present a
supervised proposition alignment baseline model, showing improved
alignment-quality over the unsupervised approach.

positive:
Current summarization systems yield generic summaries that are disconnected
from users' preferences and expectations. To address this limitation, we
present CTRLsum, a novel framework for controllable summarization. Our approach
enables users to control multiple aspects of generated summaries by interacting
with the summarization system through textual input in the form of a set of
keywords or descriptive prompts. Using a single unified model, CTRLsum is able
to achieve a broad scope of summary manipulation at inference time without
requiring additional human annotations or pre-defining a set of control aspects
during training. We quantitatively demonstrate the effectiveness of our
approach on three domains of summarization datasets and five control aspects:
1) entity-centric and 2) length-controllable summarization, 3) contribution
summarization on scientific papers, 4) invention purpose summarization on
patent filings, and 5) question-guided summarization on news articles in a
reading comprehension setting. Moreover, when used in a standard, uncontrolled
summarization setting, CTRLsum achieves state-of-the-art results on the
CNN/DailyMail dataset. Code and model checkpoints are available at
https://github.com/salesforce/ctrl-sum

positive:
A critical point of multi-document summarization (MDS) is to learn the
relations among various documents. In this paper, we propose a novel
abstractive MDS model, in which we represent multiple documents as a
heterogeneous graph, taking semantic nodes of different granularities into
account, and then apply a graph-to-sequence framework to generate summaries.
Moreover, we employ a neural topic model to jointly discover latent topics that
can act as cross-document semantic units to bridge different documents and
provide global information to guide the summary generation. Since topic
extraction can be viewed as a special type of summarization that "summarizes"
texts into a more abstract format, i.e., a topic distribution, we adopt a
multi-task learning strategy to jointly train the topic and summarization
module, allowing the promotion of each other. Experimental results on the
Multi-News dataset demonstrate that our model outperforms previous
state-of-the-art MDS models on both Rouge metrics and human evaluation,
meanwhile learns high-quality topics.

positive:
In this paper, we introduce Iterative Text Summarization (ITS), an
iteration-based model for supervised extractive text summarization, inspired by
the observation that it is often necessary for a human to read an article
multiple times in order to fully understand and summarize its contents. Current
summarization approaches read through a document only once to generate a
document representation, resulting in a sub-optimal representation. To address
this issue we introduce a model which iteratively polishes the document
representation on many passes through the document. As part of our model, we
also introduce a selective reading mechanism that decides more accurately the
extent to which each sentence in the model should be updated. Experimental
results on the CNN/DailyMail and DUC2002 datasets demonstrate that our model
significantly outperforms state-of-the-art extractive systems when evaluated by
machines and by humans.

positive:
The availability of large-scale datasets has driven the development of neural
models that create summaries from single documents, for generic purposes. When
using a summarization system, users often have specific intents with various
language realizations, which, depending on the information need, can range from
a single keyword to a long narrative composed of multiple questions. Existing
summarization systems, however, often either fail to support or act robustly on
this query focused summarization task. We introduce LaQSum, the first unified
text summarization system that learns Latent Queries from documents for
abstractive summarization with any existing query forms. Under a deep
generative framework, our system jointly optimizes a latent query model and a
conditional language model, allowing users to plug-and-play queries of any type
at test time. Despite learning from only generic summarization data and
requiring no further optimization for downstream summarization tasks, our
system robustly outperforms strong comparison systems across summarization
benchmarks with different query types, document settings, and target domains.

positive:
A desirable property of a reference-based evaluation metric that measures the
content quality of a summary is that it should estimate how much information
that summary has in common with a reference. Traditional text overlap based
metrics such as ROUGE fail to achieve this because they are limited to matching
tokens, either lexically or via embeddings. In this work, we propose a metric
to evaluate the content quality of a summary using question-answering (QA).
QA-based methods directly measure a summary's information overlap with a
reference, making them fundamentally different than text overlap metrics. We
demonstrate the experimental benefits of QA-based metrics through an analysis
of our proposed metric, QAEval. QAEval out-performs current state-of-the-art
metrics on most evaluations using benchmark datasets, while being competitive
on others due to limitations of state-of-the-art models. Through a careful
analysis of each component of QAEval, we identify its performance bottlenecks
and estimate that its potential upper-bound performance surpasses all other
automatic metrics, approaching that of the gold-standard Pyramid Method.

positive:
Much progress has been made in text summarization, fueled by neural
architectures using large-scale training corpora. However, in the news domain,
neural models easily overfit by leveraging position-related features due to the
prevalence of the inverted pyramid writing style. In addition, there is an
unmet need to generate a variety of summaries for different users. In this
paper, we propose a neural framework that can flexibly control summary
generation by introducing a set of sub-aspect functions (i.e. importance,
diversity, position). These sub-aspect functions are regulated by a set of
control codes to decide which sub-aspect to focus on during summary generation.
We demonstrate that extracted summaries with minimal position bias is
comparable with those generated by standard models that take advantage of
position preference. We also show that news summaries generated with a focus on
diversity can be more preferred by human raters. These results suggest that a
more flexible neural summarization framework providing more control options
could be desirable in tailoring to different user preferences, which is useful
since it is often impractical to articulate such preferences for different
applications a priori.

positive:
The growth of online consumer health questions has led to the necessity for
reliable and accurate question answering systems. A recent study showed that
manual summarization of consumer health questions brings significant
improvement in retrieving relevant answers. However, the automatic
summarization of long questions is a challenging task due to the lack of
training data and the complexity of the related subtasks, such as the question
focus and type recognition. In this paper, we introduce a reinforcement
learning-based framework for abstractive question summarization. We propose two
novel rewards obtained from the downstream tasks of (i) question-type
identification and (ii) question-focus recognition to regularize the question
generation model. These rewards ensure the generation of semantically valid
questions and encourage the inclusion of key medical entities/foci in the
question summary. We evaluated our proposed method on two benchmark datasets
and achieved higher performance over state-of-the-art models. The manual
evaluation of the summaries reveals that the generated questions are more
diverse and have fewer factual inconsistencies than the baseline summaries

positive:
Recent advances in text autoencoders have significantly improved the quality
of the latent space, which enables models to generate grammatical and
consistent text from aggregated latent vectors. As a successful application of
this property, unsupervised opinion summarization models generate a summary by
decoding the aggregated latent vectors of inputs. More specifically, they
perform the aggregation via simple average. However, little is known about how
the vector aggregation step affects the generation quality. In this study, we
revisit the commonly used simple average approach by examining the latent space
and generated summaries. We found that text autoencoders tend to generate
overly generic summaries from simply averaged latent vectors due to an
unexpected $L_2$-norm shrinkage in the aggregated latent vectors, which we
refer to as summary vector degeneration. To overcome this issue, we develop a
framework Coop, which searches input combinations for the latent vector
aggregation using input-output word overlap. Experimental results show that
Coop successfully alleviates the summary vector degeneration issue and
establishes new state-of-the-art performance on two opinion summarization
benchmarks. Code is available at \url{https://github.com/megagonlabs/coop}.

positive:
Automatic text summarization aims to cut down readers time and cognitive
effort by reducing the content of a text document without compromising on its
essence. Ergo, informativeness is the prime attribute of document summary
generated by an algorithm, and selecting sentences that capture the essence of
a document is the primary goal of extractive document summarization. In this
paper, we employ Shannon entropy to capture informativeness of sentences. We
employ Non-negative Matrix Factorization (NMF) to reveal probability
distributions for computing entropy of terms, topics, and sentences in latent
space. We present an information theoretic interpretation of the computed
entropy, which is the bedrock of the proposed E-Summ algorithm, an unsupervised
method for extractive document summarization. The algorithm systematically
applies information theoretic principle for selecting informative sentences
from important topics in the document. The proposed algorithm is generic and
fast, and hence amenable to use for summarization of documents in real time.
Furthermore, it is domain-, collection-independent and agnostic to the language
of the document. Benefiting from strictly positive NMF factor matrices, E-Summ
algorithm is transparent and explainable too. We use standard ROUGE toolkit for
performance evaluation of the proposed method on four well known public
data-sets. We also perform quantitative assessment of E-Summ summary quality by
computing its semantic similarity w.r.t the original document. Our
investigation reveals that though using NMF and information theoretic approach
for document summarization promises efficient, explainable, and language
independent text summarization, it needs to be bolstered to match the
performance of deep neural methods.

positive:
Traditional sequence-to-sequence (seq2seq) models and other variations of the
attention-mechanism such as hierarchical attention have been applied to the
text summarization problem. Though there is a hierarchy in the way humans use
language by forming paragraphs from sentences and sentences from words,
hierarchical models have usually not worked that much better than their
traditional seq2seq counterparts. This effect is mainly because either the
hierarchical attention mechanisms are too sparse using hard attention or noisy
using soft attention. In this paper, we propose a method based on extracting
the highlights of a document; a key concept that is conveyed in a few
sentences. In a typical text summarization dataset consisting of documents that
are 800 tokens in length (average), capturing long-term dependencies is very
important, e.g., the last sentence can be grouped with the first sentence of a
document to form a summary. LSTMs (Long Short-Term Memory) proved useful for
machine translation. However, they often fail to capture long-term dependencies
while modeling long sequences. To address these issues, we have adapted Neural
Semantic Encoders (NSE) to text summarization, a class of memory-augmented
neural networks by improving its functionalities and proposed a novel
hierarchical NSE that outperforms similar previous models significantly. The
quality of summarization was improved by augmenting linguistic factors, namely
lemma, and Part-of-Speech (PoS) tags, to each word in the dataset for improved
vocabulary coverage and generalization. The hierarchical NSE model on factored
dataset outperformed the state-of-the-art by nearly 4 ROUGE points. We further
designed and used the first GPU-based self-critical Reinforcement Learning
model.

positive:
Reference-based metrics such as ROUGE or BERTScore evaluate the content
quality of a summary by comparing the summary to a reference. Ideally, this
comparison should measure the summary's information quality by calculating how
much information the summaries have in common. In this work, we analyze the
token alignments used by ROUGE and BERTScore to compare summaries and argue
that their scores largely cannot be interpreted as measuring information
overlap, but rather the extent to which they discuss the same topics. Further,
we provide evidence that this result holds true for many other summarization
evaluation metrics. The consequence of this result is that it means the
summarization community has not yet found a reliable automatic metric that
aligns with its research goal, to generate summaries with high-quality
information. Then, we propose a simple and interpretable method of evaluating
summaries which does directly measure information overlap and demonstrate how
it can be used to gain insights into model behavior that could not be provided
by other methods alone.

positive:
Text summarization refers to the process that generates a shorter form of
text from the source document preserving salient information. Many existing
works for text summarization are generally evaluated by using recall-oriented
understudy for gisting evaluation (ROUGE) scores. However, as ROUGE scores are
computed based on n-gram overlap, they do not reflect semantic meaning
correspondences between generated and reference summaries. Because Korean is an
agglutinative language that combines various morphemes into a word that express
several meanings, ROUGE is not suitable for Korean summarization. In this
paper, we propose evaluation metrics that reflect semantic meanings of a
reference summary and the original document, Reference and Document Aware
Semantic Score (RDASS). We then propose a method for improving the correlation
of the metrics with human judgment. Evaluation results show that the
correlation with human judgment is significantly higher for our evaluation
metrics than for ROUGE scores.

positive:
Summarizing texts is not a straightforward task. Before even considering text
summarization, one should determine what kind of summary is expected. How much
should the information be compressed? Is it relevant to reformulate or should
the summary stick to the original phrasing? State-of-the-art on automatic text
summarization mostly revolves around news articles. We suggest that considering
a wider variety of tasks would lead to an improvement in the field, in terms of
generalization and robustness. We explore meeting summarization: generating
reports from automatic transcriptions. Our work consists in segmenting and
aligning transcriptions with respect to reports, to get a suitable dataset for
neural summarization. Using a bootstrapping approach, we provide pre-alignments
that are corrected by human annotators, making a validation set against which
we evaluate automatic models. This consistently reduces annotators' efforts by
providing iteratively better pre-alignment and maximizes the corpus size by
using annotations from our automatic alignment models. Evaluation is conducted
on \publicmeetings, a novel corpus of aligned public meetings. We report
automatic alignment and summarization performances on this corpus and show that
automatic alignment is relevant for data annotation since it leads to large
improvement of almost +4 on all ROUGE scores on the summarization task.

positive:
As language models become more powerful, training and evaluation are
increasingly bottlenecked by the data and metrics used for a particular task.
For example, summarization models are often trained to predict human reference
summaries and evaluated using ROUGE, but both of these metrics are rough
proxies for what we really care about -- summary quality. In this work, we show
that it is possible to significantly improve summary quality by training a
model to optimize for human preferences. We collect a large, high-quality
dataset of human comparisons between summaries, train a model to predict the
human-preferred summary, and use that model as a reward function to fine-tune a
summarization policy using reinforcement learning. We apply our method to a
version of the TL;DR dataset of Reddit posts and find that our models
significantly outperform both human reference summaries and much larger models
fine-tuned with supervised learning alone. Our models also transfer to CNN/DM
news articles, producing summaries nearly as good as the human reference
without any news-specific fine-tuning. We conduct extensive analyses to
understand our human feedback dataset and fine-tuned models We establish that
our reward model generalizes to new datasets, and that optimizing our reward
model results in better summaries than optimizing ROUGE according to humans. We
hope the evidence from our paper motivates machine learning researchers to pay
closer attention to how their training loss affects the model behavior they
actually want.

positive:
The goal of a summary is to concisely state the most important information in
a document. With this principle in mind, we introduce new reference-free
summary evaluation metrics that use a pretrained language model to estimate the
information content shared between a document and its summary. These metrics
are a modern take on the Shannon Game, a method for summary quality scoring
proposed decades ago, where we replace human annotators with language models.
We also view these metrics as an extension of BLANC, a recently proposed
approach to summary quality measurement based on the performance of a language
model with and without the help of a summary. Using transformer based language
models, we empirically verify that our metrics achieve state-of-the-art
correlation with human judgement of the summary quality dimensions of both
coherence and relevance, as well as competitive correlation with human
judgement of consistency and fluency.

positive:
Neural network-based approaches have become widespread for abstractive text
summarization. Though previously proposed models for abstractive text
summarization addressed the problem of repetition of the same contents in the
summary, they did not explicitly consider its information structure. One of the
reasons these previous models failed to account for information structure in
the generated summary is that standard datasets include summaries of variable
lengths, resulting in problems in analyzing information flow, specifically, the
manner in which the first sentence is related to the following sentences.
Therefore, we use a dataset containing summaries with only three bullet points,
and propose a neural network-based abstractive summarization model that
considers the information structures of the generated summaries. Our
experimental results show that the information structure of a summary can be
controlled, thus improving the performance of the overall summarization.

positive:
Existing multi-document summarization systems usually rely on a specific
summarization model (i.e., a summarization method with a specific parameter
setting) to extract summaries for different document sets with different
topics. However, according to our quantitative analysis, none of the existing
summarization models can always produce high-quality summaries for different
document sets, and even a summarization model with good overall performance may
produce low-quality summaries for some document sets. On the contrary, a
baseline summarization model may produce high-quality summaries for some
document sets. Based on the above observations, we treat the summaries produced
by different summarization models as candidate summaries, and then explore
discriminative reranking techniques to identify high-quality summaries from the
candidates for difference document sets. We propose to extract a set of
candidate summaries for each document set based on an ILP framework, and then
leverage Ranking SVM for summary reranking. Various useful features have been
developed for the reranking process, including word-level features,
sentence-level features and summary-level features. Evaluation results on the
benchmark DUC datasets validate the efficacy and robustness of our proposed
approach.

positive:
In this work, we propose a method for incorporating question-answering (QA)
signals into a summarization model. Our method identifies salient noun phrases
(NPs) in the input document by automatically generating wh-questions that are
answered by the NPs and automatically determining whether those questions are
answered in the gold summaries. This QA-based signal is incorporated into a
two-stage summarization model which first marks salient NPs in the input
document using a classification model, then conditionally generates a summary.
Our experiments demonstrate that the models trained using QA-based supervision
generate higher-quality summaries than baseline methods of identifying salient
spans on benchmark summarization datasets. Further, we show that the content of
the generated summaries can be controlled based on which NPs are marked in the
input document. Finally, we propose a method of augmenting the training data so
the gold summaries are more consistent with the marked input spans used during
training and show how this results in models which learn to better exclude
unmarked document content.

positive:
Abstractive summarization at controllable lengths is a challenging task in
natural language processing. It is even more challenging for domains where
limited training data is available or scenarios in which the length of the
summary is not known beforehand. At the same time, when it comes to trusting
machine-generated summaries, explaining how a summary was constructed in
human-understandable terms may be critical. We propose Multi-level Summarizer
(MLS), a supervised method to construct abstractive summaries of a text
document at controllable lengths. The key enabler of our method is an
interpretable multi-headed attention mechanism that computes attention
distribution over an input document using an array of timestep independent
semantic kernels. Each kernel optimizes a human-interpretable syntactic or
semantic property. Exhaustive experiments on two low-resource datasets in the
English language show that MLS outperforms strong baselines by up to 14.70% in
the METEOR score. Human evaluation of the summaries also suggests that they
capture the key concepts of the document at various length-budgets.

positive:
Existing approaches to automatic summarization assume that a length limit for
the summary is given, and view content selection as an optimization problem to
maximize informativeness and minimize redundancy within this budget. This
framework ignores the fact that human-written summaries have rich internal
structure which can be exploited to train a summarization system. We present
NEXTSUM, a novel approach to summarization based on a model that predicts the
next sentence to include in the summary using not only the source article, but
also the summary produced so far. We show that such a model successfully
captures summary-specific discourse moves, and leads to better content
selection performance, in addition to automatically predicting how long the
target summary should be. We perform experiments on the New York Times
Annotated Corpus of summaries, where NEXTSUM outperforms lead and content-model
summarization baselines by significant margins. We also show that the lengths
of summaries produced by our system correlates with the lengths of the
human-written gold standards.

positive:
We propose a simple method for post-processing the outputs of a text
summarization system in order to refine its overall quality. Our approach is to
train text-to-text rewriting models to correct information redundancy errors
that may arise during summarization. We train on synthetically generated noisy
summaries, testing three different types of noise that introduce out-of-context
information within each summary. When applied on top of extractive and
abstractive summarization baselines, our summary denoising models yield metric
improvements while reducing redundancy.

positive:
We address the problem of unsupervised abstractive summarization of
collections of user generated reviews with self-supervision and control. We
propose a self-supervised setup that considers an individual document as a
target summary for a set of similar documents. This setting makes training
simpler than previous approaches by relying only on standard log-likelihood
loss. We address the problem of hallucinations through the use of control
codes, to steer the generation towards more coherent and relevant
summaries.Finally, we extend the Transformer architecture to allow for multiple
reviews as input. Our benchmarks on two datasets against graph-based and recent
neural abstractive unsupervised models show that our proposed method generates
summaries with a superior quality and relevance.This is confirmed in our human
evaluation which focuses explicitly on the faithfulness of generated summaries
We also provide an ablation study, which shows the importance of the control
setup in controlling hallucinations and achieve high sentiment and topic
alignment of the summaries with the input reviews.

negative:
Abstractive text summarization is a challenging task, and one need to design
a mechanism to effectively extract salient information from the source text and
then generate a summary. A parsing process of the source text contains critical
syntactic or semantic structures, which is useful to generate more accurate
summary. However, modeling a parsing tree for text summarization is not trivial
due to its non-linear structure and it is harder to deal with a document that
includes multiple sentences and their parsing trees. In this paper, we propose
to use a graph to connect the parsing trees from the sentences in a document
and utilize the stacked graph convolutional networks (GCNs) to learn the
syntactic representation for a document. The selective attention mechanism is
used to extract salient information in semantic and structural aspect and
generate an abstractive summary. We evaluate our approach on the CNN/Daily Mail
text summarization dataset. The experimental results show that the proposed
GCNs based selective attention approach outperforms the baselines and achieves
the state-of-the-art performance on the dataset.

negative:
Under special circumstances, summaries should conform to a particular style
with patterns, such as court judgments and abstracts in academic papers. To
this end, the prototype document-summary pairs can be utilized to generate
better summaries. There are two main challenges in this task: (1) the model
needs to incorporate learned patterns from the prototype, but (2) should avoid
copying contents other than the patternized words---such as irrelevant
facts---into the generated summaries. To tackle these challenges, we design a
model named Prototype Editing based Summary Generator (PESG). PESG first learns
summary patterns and prototype facts by analyzing the correlation between a
prototype document and its summary. Prototype facts are then utilized to help
extract facts from the input document. Next, an editing generator generates new
summary based on the summary pattern or extracted facts. Finally, to address
the second challenge, a fact checker is used to estimate mutual information
between the input document and generated summary, providing an additional
signal for the generator. Extensive experiments conducted on a large-scale
real-world text summarization dataset show that PESG achieves the
state-of-the-art performance in terms of both automatic metrics and human
evaluations.

negative:
We present a novel divide-and-conquer method for the neural summarization of
long documents. Our method exploits the discourse structure of the document and
uses sentence similarity to split the problem into an ensemble of smaller
summarization problems. In particular, we break a long document and its summary
into multiple source-target pairs, which are used for training a model that
learns to summarize each part of the document separately. These partial
summaries are then combined in order to produce a final complete summary. With
this approach we can decompose the problem of long document summarization into
smaller and simpler problems, reducing computational complexity and creating
more training examples, which at the same time contain less noise in the target
summaries compared to the standard approach. We demonstrate that this approach
paired with different summarization models, including sequence-to-sequence RNNs
and Transformers, can lead to improved summarization performance. Our best
models achieve results that are on par with the state-of-the-art in two two
publicly available datasets of academic articles.

negative:
A crucial difference between single- and multi-document summarization is how
salient content manifests itself in the document(s). While such content may
appear at the beginning of a single document, essential information is
frequently reiterated in a set of documents related to a particular topic,
resulting in an endorsement effect that increases information salience. In this
paper, we model the cross-document endorsement effect and its utilization in
multiple document summarization. Our method generates a synopsis from each
document, which serves as an endorser to identify salient content from other
documents. Strongly endorsed text segments are used to enrich a neural
encoder-decoder model to consolidate them into an abstractive summary. The
method has a great potential to learn from fewer examples to identify salient
content, which alleviates the need for costly retraining when the set of
documents is dynamically adjusted. Through extensive experiments on benchmark
multi-document summarization datasets, we demonstrate the effectiveness of our
proposed method over strong published baselines. Finally, we shed light on
future research directions and discuss broader challenges of this task using a
case study.

negative:
Text summarization aims to compress a textual document to a short summary
while keeping salient information. Extractive approaches are widely used in
text summarization because of their fluency and efficiency. However, most of
existing extractive models hardly capture inter-sentence relationships,
particularly in long documents. They also often ignore the effect of topical
information on capturing important contents. To address these issues, this
paper proposes a graph neural network (GNN)-based extractive summarization
model, enabling to capture inter-sentence relationships efficiently via
graph-structured document representation. Moreover, our model integrates a
joint neural topic model (NTM) to discover latent topics, which can provide
document-level features for sentence selection. The experimental results
demonstrate that our model not only substantially achieves state-of-the-art
results on CNN/DM and NYT datasets but also considerably outperforms existing
approaches on scientific paper datasets consisting of much longer documents,
indicating its better robustness in document genres and lengths. Further
discussions show that topical information can help the model preselect salient
contents from an entire document, which interprets its effectiveness in long
document summarization.

negative:
Recent neural sequence to sequence models have provided feasible solutions
for abstractive summarization. However, such models are still hard to tackle
long text dependency in the summarization task. A high-quality summarization
system usually depends on strong encoder which can refine important information
from long input texts so that the decoder can generate salient summaries from
the encoder's memory. In this paper, we propose an aggregation mechanism based
on the Transformer model to address the challenge of long text representation.
Our model can review history information to make encoder hold more memory
capacity. Empirically, we apply our aggregation mechanism to the Transformer
model and experiment on CNN/DailyMail dataset to achieve higher quality
summaries compared to several strong baseline models on the ROUGE metrics.

negative:
Text alignment finds application in tasks such as citation recommendation and
plagiarism detection. Existing alignment methods operate at a single,
predefined level and cannot learn to align texts at, for example, sentence and
document levels. We propose a new learning approach that equips previously
established hierarchical attention encoders for representing documents with a
cross-document attention component, enabling structural comparisons across
different levels (document-to-document and sentence-to-document). Our component
is weakly supervised from document pairs and can align at multiple levels. Our
evaluation on predicting document-to-document relationships and
sentence-to-document relationships on the tasks of citation recommendation and
plagiarism detection shows that our approach outperforms previously established
hierarchical, attention encoders based on recurrent and transformer
contextualization that are unaware of structural correspondence between
documents.

negative:
Abstractive summarization for long-document or multi-document remains
challenging for the Seq2Seq architecture, as Seq2Seq is not good at analyzing
long-distance relations in text. In this paper, we present BASS, a novel
framework for Boosting Abstractive Summarization based on a unified Semantic
graph, which aggregates co-referent phrases distributing across a long range of
context and conveys rich relations between phrases. Further, a graph-based
encoder-decoder model is proposed to improve both the document representation
and summary generation process by leveraging the graph structure. Specifically,
several graph augmentation methods are designed to encode both the explicit and
implicit relations in the text while the graph-propagation attention mechanism
is developed in the decoder to select salient content into the summary.
Empirical results show that the proposed architecture brings substantial
improvements for both long-document and multi-document summarization tasks.

negative:
Multi-document summaritazion is the process of taking multiple texts as input
and producing a short summary text based on the content of input texts. Up
until recently, multi-document summarizers are mostly supervised extractive.
However, supervised methods require datasets of large, paired document-summary
examples which are rare and expensive to produce. In 2018, an unsupervised
multi-document abstractive summarization method(Meansum) was proposed by Chu
and Liu, and demonstrated competitive performances comparing to extractive
methods. Despite good evaluation results on automatic metrics, Meansum has
multiple limitations, notably the inability of dealing with multiple aspects.
The aim of this work was to use Multi-Aspect Masker(MAM) as content selector to
address the issue with multi-aspect. Moreover, we propose a regularizer to
control the length of the generated summaries. Through a series of experiments
on the hotel dataset from Trip Advisor, we validate our assumption and show
that our improved model achieves higher ROUGE, Sentiment Accuracy than the
original Meansum method and also beats/ comprarable/close to the supervised
baseline.

negative:
Developed so far, multi-document summarization has reached its bottleneck due
to the lack of sufficient training data and diverse categories of documents.
Text classification just makes up for these deficiencies. In this paper, we
propose a novel summarization system called TCSum, which leverages plentiful
text classification data to improve the performance of multi-document
summarization. TCSum projects documents onto distributed representations which
act as a bridge between text classification and summarization. It also utilizes
the classification results to produce summaries of different styles. Extensive
experiments on DUC generic multi-document summarization datasets show that,
TCSum can achieve the state-of-the-art performance without using any
hand-crafted features and has the capability to catch the variations of summary
styles with respect to different text categories.

negative:
Traditional video summarization methods generate fixed video representations
regardless of user interest. Therefore such methods limit users' expectations
in content search and exploration scenarios. Multi-modal video summarization is
one of the methods utilized to address this problem. When multi-modal video
summarization is used to help video exploration, a text-based query is
considered as one of the main drivers of video summary generation, as it is
user-defined. Thus, encoding the text-based query and the video effectively are
both important for the task of multi-modal video summarization. In this work, a
new method is proposed that uses a specialized attention network and
contextualized word representations to tackle this task. The proposed model
consists of a contextualized video summary controller, multi-modal attention
mechanisms, an interactive attention network, and a video summary generator.
Based on the evaluation of the existing multi-modal video summarization
benchmark, experimental results show that the proposed model is effective with
the increase of +5.88% in accuracy and +4.06% increase of F1-score, compared
with the state-of-the-art method.

negative:
Neural summarization models suffer from the fixed-size input limitation: if
text length surpasses the model's maximal number of input tokens, some document
content (possibly summary-relevant) gets truncated Independently summarizing
windows of maximal input size disallows for information flow between windows
and leads to incoherent summaries. We propose windowing models for neural
abstractive summarization of (arbitrarily) long texts. We extend the
sequence-to-sequence model augmented with pointer generator network by (1)
allowing the encoder to slide over different windows of the input document and
(2) sharing the decoder and retaining its state across different input windows.
We explore two windowing variants: Static Windowing precomputes the number of
tokens the decoder should generate from each window (based on training corpus
statistics); in Dynamic Windowing the decoder learns to emit a token that
signals encoder's shift to the next input window. Empirical results render our
models effective in their intended use-case: summarizing long texts with
relevant content not bound to the very document beginning.

negative:
A key challenge for abstractive summarization is ensuring factual consistency
of the generated summary with respect to the original document. For example,
state-of-the-art models trained on existing datasets exhibit entity
hallucination, generating names of entities that are not present in the source
document. We propose a set of new metrics to quantify the entity-level factual
consistency of generated summaries and we show that the entity hallucination
problem can be alleviated by simply filtering the training data. In addition,
we propose a summary-worthy entity classification task to the training process
as well as a joint entity and summary generation approach, which yield further
improvements in entity level metrics.

negative:
In this paper, we aim to improve abstractive dialogue summarization quality
and, at the same time, enable granularity control. Our model has two primary
components and stages: 1) a two-stage generation strategy that generates a
preliminary summary sketch serving as the basis for the final summary. This
summary sketch provides a weakly supervised signal in the form of
pseudo-labeled interrogative pronoun categories and key phrases extracted using
a constituency parser. 2) A simple strategy to control the granularity of the
final summary, in that our model can automatically determine or control the
number of generated summary sentences for a given dialogue by predicting and
highlighting different text spans from the source text. Our model achieves
state-of-the-art performance on the largest dialogue summarization corpus
SAMSum, with as high as 50.79 in ROUGE-L score. In addition, we conduct a case
study and show competitive human evaluation results and controllability to
human-annotated summaries.

negative:
Recently, various neural encoder-decoder models pioneered by Seq2Seq
framework have been proposed to achieve the goal of generating more abstractive
summaries by learning to map input text to output text. At a high level, such
neural models can freely generate summaries without any constraint on the words
or phrases used. Moreover, their format is closer to human-edited summaries and
output is more readable and fluent. However, the neural model's abstraction
ability is a double-edged sword. A commonly observed problem with the generated
summaries is the distortion or fabrication of factual information in the
article. This inconsistency between the original text and the summary has
caused various concerns over its applicability, and the previous evaluation
methods of text summarization are not suitable for this issue. In response to
the above problems, the current research direction is predominantly divided
into two categories, one is to design fact-aware evaluation metrics to select
outputs without factual inconsistency errors, and the other is to develop new
summarization systems towards factual consistency. In this survey, we focus on
presenting a comprehensive review of these fact-specific evaluation methods and
text summarization models.

negative:
Automatic text summarization has been widely studied as an important task in
natural language processing. Traditionally, various feature engineering and
machine learning based systems have been proposed for extractive as well as
abstractive text summarization. Recently, deep learning based, specifically
Transformer-based systems have been immensely popular. Summarization is a
cognitively challenging task - extracting summary worthy sentences is
laborious, and expressing semantics in brief when doing abstractive
summarization is complicated. In this paper, we specifically look at the
problem of summarizing scientific research papers from multiple domains. We
differentiate between two types of summaries, namely, (a) LaySumm: A very short
summary that captures the essence of the research paper in layman terms
restricting overtly specific technical jargon and (b) LongSumm: A much longer
detailed summary aimed at providing specific insights into various ideas
touched upon in the paper. While leveraging latest Transformer-based models,
our systems are simple, intuitive and based on how specific paper sections
contribute to human summaries of the two types described above. Evaluations
against gold standard summaries using ROUGE metrics prove the effectiveness of
our approach. On blind test corpora, our system ranks first and third for the
LongSumm and LaySumm tasks respectively.

negative:
Despite the recent developments on neural summarization systems, the
underlying logic behind the improvements from the systems and its
corpus-dependency remains largely unexplored. Position of sentences in the
original text, for example, is a well known bias for news summarization.
Following in the spirit of the claim that summarization is a combination of
sub-functions, we define three sub-aspects of summarization: position,
importance, and diversity and conduct an extensive analysis of the biases of
each sub-aspect with respect to the domain of nine different summarization
corpora (e.g., news, academic papers, meeting minutes, movie script, books,
posts). We find that while position exhibits substantial bias in news articles,
this is not the case, for example, with academic papers and meeting minutes.
Furthermore, our empirical study shows that different types of summarization
systems (e.g., neural-based) are composed of different degrees of the
sub-aspects. Our study provides useful lessons regarding consideration of
underlying sub-aspects when collecting a new summarization dataset or
developing a new system.

negative:
We present a new summarization task, generating summaries of novel chapters
using summary/chapter pairs from online study guides. This is a harder task
than the news summarization task, given the chapter length as well as the
extreme paraphrasing and generalization found in the summaries. We focus on
extractive summarization, which requires the creation of a gold-standard set of
extractive summaries. We present a new metric for aligning reference summary
sentences with chapter sentences to create gold extracts and also experiment
with different alignment methods. Our experiments demonstrate significant
improvement over prior alignment approaches for our task as shown through
automatic metrics and a crowd-sourced pyramid analysis. We make our data
collection scripts available at
https://github.com/manestay/novel-chapter-dataset .

negative:
Automatic video summarization is still an unsolved problem due to several
challenges. We take steps towards making automatic video summarization more
realistic by addressing them. Firstly, the currently available datasets either
have very short videos or have few long videos of only a particular type. We
introduce a new benchmarking dataset VISIOCITY which comprises of longer videos
across six different categories with dense concept annotations capable of
supporting different flavors of video summarization and can be used for other
vision problems. Secondly, for long videos, human reference summaries are
difficult to obtain. We present a novel recipe based on pareto optimality to
automatically generate multiple reference summaries from indirect ground truth
present in VISIOCITY. We show that these summaries are at par with human
summaries. Thirdly, we demonstrate that in the presence of multiple ground
truth summaries (due to the highly subjective nature of the task), learning
from a single combined ground truth summary using a single loss function is not
a good idea. We propose a simple recipe VISIOCITY-SUM to enhance an existing
model using a combination of losses and demonstrate that it beats the current
state of the art techniques when tested on VISIOCITY. We also show that a
single measure to evaluate a summary, as is the current typical practice, falls
short. We propose a framework for better quantitative assessment of summary
quality which is closer to human judgment than a single measure, say F1. We
report the performance of a few representative techniques of video
summarization on VISIOCITY assessed using various measures and bring out the
limitation of the techniques and/or the assessment mechanism in modeling human
judgment and demonstrate the effectiveness of our evaluation framework in doing
so.

negative:
Exploring the tremendous amount of data efficiently to make a decision,
similar to answering a complicated question, is challenging with many
real-world application scenarios. In this context, automatic summarization has
substantial importance as it will provide the foundation for big data analytic.
Traditional summarization approaches optimize the system to produce a short
static summary that fits all users that do not consider the subjectivity aspect
of summarization, i.e., what is deemed valuable for different users, making
these approaches impractical in real-world use cases. This paper proposes an
interactive concept-based summarization model, called Adaptive Summaries, that
helps users make their desired summary instead of producing a single inflexible
summary. The system learns from users' provided information gradually while
interacting with the system by giving feedback in an iterative loop. Users can
choose either reject or accept action for selecting a concept being included in
the summary with the importance of that concept from users' perspectives and
confidence level of their feedback. The proposed approach can guarantee
interactive speed to keep the user engaged in the process. Furthermore, it
eliminates the need for reference summaries, which is a challenging issue for
summarization tasks. Evaluations show that Adaptive Summaries helps users make
high-quality summaries based on their preferences by maximizing the
user-desired content in the generated summaries.

negative:
In this paper we present VideoSET, a method for Video Summary Evaluation
through Text that can evaluate how well a video summary is able to retain the
semantic information contained in its original video. We observe that semantics
is most easily expressed in words, and develop a text-based approach for the
evaluation. Given a video summary, a text representation of the video summary
is first generated, and an NLP-based metric is then used to measure its
semantic distance to ground-truth text summaries written by humans. We show
that our technique has higher agreement with human judgment than pixel-based
distance metrics. We also release text annotations and ground-truth text
summaries for a number of publicly available video datasets, for use by the
computer vision community.

negative:
Standard neural machine translation (NMT) is on the assumption of
document-level context independent. Most existing document-level NMT methods
are satisfied with a smattering sense of brief document-level information,
while this work focuses on exploiting detailed document-level context in terms
of multiple forms of document embeddings, which is capable of sufficiently
modeling deeper and richer document-level context. The proposed document-aware
NMT is implemented to enhance the Transformer baseline by introducing both
global and local document-level clues on the source end. Experiments show that
the proposed method significantly improves the translation performance over
strong baselines and other related studies.

negative:
The scarcity of comprehensive up-to-date studies on evaluation metrics for
text summarization and the lack of consensus regarding evaluation protocols
continue to inhibit progress. We address the existing shortcomings of
summarization evaluation methods along five dimensions: 1) we re-evaluate 14
automatic evaluation metrics in a comprehensive and consistent fashion using
neural summarization model outputs along with expert and crowd-sourced human
annotations, 2) we consistently benchmark 23 recent summarization models using
the aforementioned automatic evaluation metrics, 3) we assemble the largest
collection of summaries generated by models trained on the CNN/DailyMail news
dataset and share it in a unified format, 4) we implement and share a toolkit
that provides an extensible and unified API for evaluating summarization models
across a broad range of automatic metrics, 5) we assemble and share the largest
and most diverse, in terms of model types, collection of human judgments of
model-generated summaries on the CNN/Daily Mail dataset annotated by both
expert judges and crowd-source workers. We hope that this work will help
promote a more complete evaluation protocol for text summarization as well as
advance research in developing evaluation metrics that better correlate with
human judgments.

negative:
Summarizing content contributed by individuals can be challenging, because
people make different lexical choices even when describing the same events.
However, there remains a significant need to summarize such content. Examples
include the student responses to post-class reflective questions, product
reviews, and news articles published by different news agencies related to the
same events. High lexical diversity of these documents hinders the system's
ability to effectively identify salient content and reduce summary redundancy.
In this paper, we overcome this issue by introducing an integer linear
programming-based summarization framework. It incorporates a low-rank
approximation to the sentence-word co-occurrence matrix to intrinsically group
semantically-similar lexical items. We conduct extensive experiments on
datasets of student responses, product reviews, and news documents. Our
approach compares favorably to a number of extractive baselines as well as a
neural abstractive summarization system. The paper finally sheds light on when
and why the proposed framework is effective at summarizing content with high
lexical variety.

negative:
Currently used metrics for assessing summarization algorithms do not account
for whether summaries are factually consistent with source documents. We
propose a weakly-supervised, model-based approach for verifying factual
consistency and identifying conflicts between source documents and a generated
summary. Training data is generated by applying a series of rule-based
transformations to the sentences of source documents. The factual consistency
model is then trained jointly for three tasks: 1) identify whether sentences
remain factually consistent after transformation, 2) extract a span in the
source documents to support the consistency prediction, 3) extract a span in
the summary sentence that is inconsistent if one exists. Transferring this
model to summaries generated by several state-of-the art models reveals that
this highly scalable approach substantially outperforms previous models,
including those trained with strong supervision using standard datasets for
natural language inference and fact checking. Additionally, human evaluation
shows that the auxiliary span extraction tasks provide useful assistance in the
process of verifying factual consistency.

negative:
Predicting the quality of a text document is a critical task when presented
with the problem of measuring the performance of a document before its release.
In this work, we evaluate various features including those extracted from the
text content (textual) and those describing higher-level characteristics of the
text (meta) features that are not directly available from the text, and show
how these features inform prediction of document quality in different ways.
Moreover, we also compare our methods on both social user-generated data such
as tweets, and scholarly user-generated data such as academic articles, showing
how the same features differently influence prediction of quality across these
disparate domains.

negative:
The technology of automatic document summarization is maturing and may
provide a solution to the information overload problem. Nowadays, document
summarization plays an important role in information retrieval. With a large
volume of documents, presenting the user with a summary of each document
greatly facilitates the task of finding the desired documents. Document
summarization is a process of automatically creating a compressed version of a
given document that provides useful information to users, and multi-document
summarization is to produce a summary delivering the majority of information
content from a set of documents about an explicit or implicit main topic. The
lexical cohesion structure of the text can be exploited to determine the
importance of a sentence/phrase. Lexical chains are useful tools to analyze the
lexical cohesion structure in a text .In this paper we consider the effect of
the use of lexical cohesion features in Summarization, And presenting a
algorithm base on the knowledge base. Ours algorithm at first find the correct
sense of any word, Then constructs the lexical chains, remove Lexical chains
that less score than other, detects topics roughly from lexical chains,
segments the text with respect to the topics and selects the most important
sentences. The experimental results on an open benchmark datasets from DUC01
and DUC02 show that our proposed approach can improve the performance compared
to sate-of-the-art summarization approaches.

negative:
Automatic Text Summarization strategies have been successfully employed to
digest text collections and extract its essential content. Usually, summaries
are generated using textual corpora that belongs to the same domain area where
the summary will be used. Nonetheless, there are special cases where it is not
found enough textual sources, and one possible alternative is to generate a
summary from a different domain. One manner to summarize texts consists of
using a graph model. This model allows giving more importance to words
corresponding to the main concepts from the target domain found in the
summarized text. This gives the reader an overview of the main text concepts as
well as their relationships. However, this kind of summarization presents a
significant number of repeated terms when compared to human-generated
summaries. In this paper, we present an approach to produce graph-model
extractive summaries of texts, meeting the target domain exigences and treating
the terms repetition problem. To evaluate the proposition, we performed a
series of experiments showing that the proposed approach statistically improves
the performance of a model based on Graph Centrality, achieving better
coverage, accuracy, and recall.

negative:
Prior work in document summarization has mainly focused on generating short
summaries of a document. While this type of summary helps get a high-level view
of a given document, it is desirable in some cases to know more detailed
information about its salient points that can't fit in a short summary. This is
typically the case for longer documents such as a research paper, legal
document, or a book. In this paper, we present a new method for generating
extended summaries of long papers. Our method exploits hierarchical structure
of the documents and incorporates it into an extractive summarization model
through a multi-task learning approach. We then present our results on three
long summarization datasets, arXiv-Long, PubMed-Long, and Longsumm. Our method
outperforms or matches the performance of strong baselines. Furthermore, we
perform a comprehensive analysis over the generated results, shedding insights
on future research for long-form summary generation task. Our analysis shows
that our multi-tasking approach can adjust extraction probability distribution
to the favor of summary-worthy sentences across diverse sections. Our datasets,
and codes are publicly available at
https://github.com/Georgetown-IR-Lab/ExtendedSumm

negative:
Neural abstractive summarization models are prone to generate content
inconsistent with the source document, i.e. unfaithful. Existing automatic
metrics do not capture such mistakes effectively. We tackle the problem of
evaluating faithfulness of a generated summary given its source document. We
first collected human annotations of faithfulness for outputs from numerous
models on two datasets. We find that current models exhibit a trade-off between
abstractiveness and faithfulness: outputs with less word overlap with the
source document are more likely to be unfaithful. Next, we propose an automatic
question answering (QA) based metric for faithfulness, FEQA, which leverages
recent advances in reading comprehension. Given question-answer pairs generated
from the summary, a QA model extracts answers from the document; non-matched
answers indicate unfaithful information in the summary. Among metrics based on
word overlap, embedding similarity, and learned language understanding models,
our QA-based metric has significantly higher correlation with human
faithfulness scores, especially on highly abstractive summaries.

negative:
Despite significant progress in neural abstractive summarization, recent
studies have shown that the current models are prone to generating summaries
that are unfaithful to the original context. To address the issue, we study
contrast candidate generation and selection as a model-agnostic post-processing
technique to correct the extrinsic hallucinations (i.e. information not present
in the source text) in unfaithful summaries. We learn a discriminative
correction model by generating alternative candidate summaries where named
entities and quantities in the generated summary are replaced with ones with
compatible semantic types from the source document. This model is then used to
select the best candidate as the final output summary. Our experiments and
analysis across a number of neural summarization systems show that our proposed
method is effective in identifying and correcting extrinsic hallucinations. We
analyze the typical hallucination phenomenon by different types of neural
summarization systems, in hope to provide insights for future work on the
direction.

negative:
We present BLANC, a new approach to the automatic estimation of document
summary quality. Our goal is to measure the functional performance of a summary
with an objective, reproducible, and fully automated method. Our approach
achieves this by measuring the performance boost gained by a pre-trained
language model with access to a document summary while carrying out its
language understanding task on the document's text. We present evidence that
BLANC scores have as good correlation with human evaluations as do the ROUGE
family of summary quality measurements. And unlike ROUGE, the BLANC method does
not require human-written reference summaries, allowing for fully human-free
summary quality estimation.

negative:
Owing to the rapidly growing multimedia content available on the Internet,
extractive spoken document summarization, with the purpose of automatically
selecting a set of representative sentences from a spoken document to concisely
express the most important theme of the document, has been an active area of
research and experimentation. On the other hand, word embedding has emerged as
a newly favorite research subject because of its excellent performance in many
natural language processing (NLP)-related tasks. However, as far as we are
aware, there are relatively few studies investigating its use in extractive
text or speech summarization. A common thread of leveraging word embeddings in
the summarization process is to represent the document (or sentence) by
averaging the word embeddings of the words occurring in the document (or
sentence). Then, intuitively, the cosine similarity measure can be employed to
determine the relevance degree between a pair of representations. Beyond the
continued efforts made to improve the representation of words, this paper
focuses on building novel and efficient ranking models based on the general
word embedding methods for extractive speech summarization. Experimental
results demonstrate the effectiveness of our proposed methods, compared to
existing state-of-the-art methods.

negative:
In this paper, we present a model for generating summaries of text documents
with respect to a query. This is known as query-based summarization. We adapt
an existing dataset of news article summaries for the task and train a
pointer-generator model using this dataset. The generated summaries are
evaluated by measuring similarity to reference summaries. Our results show that
a neural network summarization model, similar to existing neural network models
for abstractive summarization, can be constructed to make use of queries to
produce targeted summaries.

negative:
A major challenge for scaling machine learning is training models to perform
tasks that are very difficult or time-consuming for humans to evaluate. We
present progress on this problem on the task of abstractive summarization of
entire fiction novels. Our method combines learning from human feedback with
recursive task decomposition: we use models trained on smaller parts of the
task to assist humans in giving feedback on the broader task. We collect a
large volume of demonstrations and comparisons from human labelers, and
fine-tune GPT-3 using behavioral cloning and reward modeling to do
summarization recursively. At inference time, the model first summarizes small
sections of the book and then recursively summarizes these summaries to produce
a summary of the entire book. Our human labelers are able to supervise and
evaluate the models quickly, despite not having read the entire books
themselves. Our resulting model generates sensible summaries of entire books,
even matching the quality of human-written summaries in a few cases ($\sim5\%$
of books). We achieve state-of-the-art results on the recent BookSum dataset
for book-length summarization. A zero-shot question-answering model using these
summaries achieves state-of-the-art results on the challenging NarrativeQA
benchmark for answering questions about books and movie scripts. We release
datasets of samples from our model.

negative:
Long document summarization is an important and hard task in the field of
natural language processing. A good performance of the long document
summarization reveals the model has a decent understanding of the human
language. Currently, most researches focus on how to modify the attention
mechanism of the transformer to achieve a higher ROUGE score. The study of data
pre-processing and post-processing are relatively few. In this paper, we use
two pre-processing methods and a post-processing method and analyze the effect
of these methods on various long document summarization models.

negative:
Text generation is an important Natural Language Processing task with various
applications. Although several metrics have already been introduced to evaluate
the text generation methods, each of them has its own shortcomings. The most
widely used metrics such as BLEU only consider the quality of generated
sentences and neglect their diversity. For example, repeatedly generation of
only one high quality sentence would result in a high BLEU score. On the other
hand, the more recent metric introduced to evaluate the diversity of generated
texts known as Self-BLEU ignores the quality of generated texts. In this paper,
we propose metrics to evaluate both the quality and diversity simultaneously by
approximating the distance of the learned generative model and the real data
distribution. For this purpose, we first introduce a metric that approximates
this distance using n-gram based measures. Then, a feature-based measure which
is based on a recent highly deep model trained on a large text corpus called
BERT is introduced. Finally, for oracle training mode in which the generator's
density can also be calculated, we propose to use the distance measures between
the corresponding explicit distributions. Eventually, the most popular and
recent text generation models are evaluated using both the existing and the
proposed metrics and the preferences of the proposed metrics are determined.

negative:
Single document summarization generates summary by extracting the
representative sentences from the document. In this paper, we presented a novel
technique for summarization of domain-specific text from a single web document
that uses statistical and linguistic analysis on the text in a reference corpus
and the web document. The proposed summarizer uses the combinational function
of Sentence Weight (SW) and Subject Weight (SuW) to determine the rank of a
sentence, where SW is the function of number of terms (t_n) and number of words
(w_n) in a sentence, and term frequency (t_f) in the corpus and SuW is the
function of t_n and w_n in a subject, and t_f in the corpus. 30 percent of the
ranked sentences are considered to be the summary of the web document. We
generated three web document summaries using our technique and compared each of
them with the summaries developed manually from 16 different human subjects.
Results showed that 68 percent of the summaries produced by our approach
satisfy the manual summaries.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
In the light of exponentially increasing video content, video summarization
has attracted a lot of attention recently due to its ability to optimize time
and storage. Characteristics of a good summary of a video depend on the
particular domain under question. We propose a novel framework for domain
specific video summarization. Given a video of a particular domain, our system
can produce a summary based on what is important for that domain in addition to
possessing other desired characteristics like representativeness, coverage,
diversity etc. as suitable to that domain. Past related work has focused either
on using supervised approaches for ranking the snippets to produce summary or
on using unsupervised approaches of generating the summary as a subset of
snippets with the above characteristics. We look at the joint problem of
learning domain specific importance of segments as well as the desired summary
characteristic for that domain. Our studies show that the more efficient way of
incorporating domain specific relevances into a summary is by obtaining ratings
of shots as opposed to binary inclusion/exclusion information. We also argue
that ratings can be seen as unified representation of all possible ground truth
summaries of a video, taking us one step closer in dealing with challenges
associated with multiple ground truth summaries of a video. We also propose a
novel evaluation measure which is more naturally suited in assessing the
quality of video summary for the task at hand than F1 like measures. It
leverages the ratings information and is richer in appropriately modeling
desirable and undesirable characteristics of a summary. Lastly, we release a
gold standard dataset for furthering research in domain specific video
summarization, which to our knowledge is the first dataset with long videos
across several domains with rating annotations.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
The amount of text data available online is increasing at a very fast pace
hence text summarization has become essential. Most of the modern recommender
and text classification systems require going through a huge amount of data.
Manually generating precise and fluent summaries of lengthy articles is a very
tiresome and time-consuming task. Hence generating automated summaries for the
data and using it to train machine learning models will make these models space
and time-efficient. Extractive summarization and abstractive summarization are
two separate methods of generating summaries. The extractive technique
identifies the relevant sentences from the original document and extracts only
those from the text. Whereas in abstractive summarization techniques, the
summary is generated after interpreting the original text, hence making it more
complicated. In this paper, we will be presenting a comprehensive comparison of
a few transformer architecture based pre-trained models for text summarization.
For analysis and comparison, we have used the BBC news dataset that contains
text data that can be used for summarization and human generated summaries for
evaluating and comparing the summaries generated by machine learning models.

negative:
Automatic text summarization aims to produce a brief but crucial summary for
the input documents. Both extractive and abstractive methods have witnessed
great success in English datasets in recent years. However, there has been a
minimal exploration of text summarization in Chinese, limited by the lack of
large-scale datasets. In this paper, we present a large-scale Chinese news
summarization dataset CNewSum, which consists of 304,307 documents and
human-written summaries for the news feed. It has long documents with
high-abstractive summaries, which can encourage document-level understanding
and generation for current summarization models. An additional distinguishing
feature of CNewSum is that its test set contains adequacy and deducibility
annotations for the summaries. The adequacy level measures the degree of
summary information covered by the document, and the deducibility indicates the
reasoning ability the model needs to generate the summary. These annotations
can help researchers analyze and target their model performance bottleneck. We
examine recent methods on CNewSum and release our dataset to provide a solid
testbed for automatic Chinese summarization research.

negative:
Data-driven approaches to sequence-to-sequence modelling have been
successfully applied to short text summarization of news articles. Such models
are typically trained on input-summary pairs consisting of only a single or a
few sentences, partially due to limited availability of multi-sentence training
data. Here, we propose to use scientific articles as a new milestone for text
summarization: large-scale training data come almost for free with two types of
high-quality summaries at different levels - the title and the abstract. We
generate two novel multi-sentence summarization datasets from scientific
articles and test the suitability of a wide range of existing extractive and
abstractive neural network-based summarization approaches. Our analysis
demonstrates that scientific papers are suitable for data-driven text
summarization. Our results could serve as valuable benchmarks for scaling
sequence-to-sequence models to very long sequences.

negative:
Summarization is one of the key features of human intelligence. It plays an
important role in understanding and representation. With rapid and continual
expansion of texts, pictures and videos in cyberspace, automatic summarization
becomes more and more desirable. Text summarization has been studied for over
half century, but it is still hard to automatically generate a satisfied
summary. Traditional methods process texts empirically and neglect the
fundamental characteristics and principles of language use and understanding.
This paper summarizes previous text summarization approaches in a
multi-dimensional classification space, introduces a multi-dimensional
methodology for research and development, unveils the basic characteristics and
principles of language use and understanding, investigates some fundamental
mechanisms of summarization, studies the dimensions and forms of
representations, and proposes a multi-dimensional evaluation mechanisms.
Investigation extends to the incorporation of pictures into summary and to the
summarization of videos, graphs and pictures, and then reaches a general
summarization framework.

negative:
Text summarization is an interesting area for researchers to develop new
techniques to provide human like summaries for vast amounts of information.
Summarization techniques tend to focus on providing accurate representation of
content, and often the tone of the content is ignored. Tone of the content sets
a baseline for how a reader perceives the content. As such being able to
generate summary with tone that is appropriate for the reader is important. In
our work we implement Maximal Marginal Relevance [MMR] based multi-document
text summarization and propose a naive model to change tone of the
summarization by setting a bias to specific set of words and restricting other
words in the summarization output. This bias towards a specified set of words
produces a summary whose tone is same as tone of specified words.

negative:
Highlighting while reading is a natural behavior for people to track salient
content of a document. It would be desirable to teach an extractive summarizer
to do the same. However, a major obstacle to the development of a supervised
summarizer is the lack of ground-truth. Manual annotation of extraction units
is cost-prohibitive, whereas acquiring labels by automatically aligning human
abstracts and source documents can yield inferior results. In this paper we
describe a novel framework to guide a supervised, extractive summarization
system with question-answering rewards. We argue that quality summaries should
serve as a document surrogate to answer important questions, and such
question-answer pairs can be conveniently obtained from human abstracts. The
system learns to promote summaries that are informative, fluent, and perform
competitively on question-answering. Our results compare favorably with those
reported by strong summarization baselines as evaluated by automatic metrics
and human assessors.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
The development of summarization research has been significantly hampered by
the costly acquisition of reference summaries. This paper proposes an effective
way to automatically collect large scales of news-related multi-document
summaries with reference to social media's reactions. We utilize two types of
social labels in tweets, i.e., hashtags and hyper-links. Hashtags are used to
cluster documents into different topic sets. Also, a tweet with a hyper-link
often highlights certain key points of the corresponding document. We
synthesize a linked document cluster to form a reference summary which can
cover most key points. To this aim, we adopt the ROUGE metrics to measure the
coverage ratio, and develop an Integer Linear Programming solution to discover
the sentence set reaching the upper bound of ROUGE. Since we allow summary
sentences to be selected from both documents and high-quality tweets, the
generated reference summaries could be abstractive. Both informativeness and
readability of the collected summaries are verified by manual judgment. In
addition, we train a Support Vector Regression summarizer on DUC generic
multi-document summarization benchmarks. With the collected data as extra
training resource, the performance of the summarizer improves a lot on all the
test sets. We release this dataset for further research.

negative:
Neural conversation models are known to generate appropriate but
non-informative responses in general. A scenario where informativeness can be
significantly enhanced is Conversing by Reading (CbR), where conversations take
place with respect to a given external document. In previous work, the external
document is utilized by (1) creating a context-aware document memory that
integrates information from the document and the conversational context, and
then (2) generating responses referring to the memory. In this paper, we
propose to create the document memory with some anticipated responses in mind.
This is achieved using a teacher-student framework. The teacher is given the
external document, the context, and the ground-truth response, and learns how
to build a response-aware document memory from three sources of information.
The student learns to construct a response-anticipated document memory from the
first two sources, and the teacher's insight on memory creation. Empirical
results show that our model outperforms the previous state-of-the-art for the
CbR task.

negative:
We present a novel abstractive summarization framework that draws on the
recent development of a treebank for the Abstract Meaning Representation (AMR).
In this framework, the source text is parsed to a set of AMR graphs, the graphs
are transformed into a summary graph, and then text is generated from the
summary graph. We focus on the graph-to-graph transformation that reduces the
source semantic graph into a summary graph, making use of an existing AMR
parser and assuming the eventual availability of an AMR-to-text generator. The
framework is data-driven, trainable, and not specifically designed for a
particular domain. Experiments on gold-standard AMR annotations and system
parses show promising results. Code is available at:
https://github.com/summarization

negative:
Question-driven summarization has been recently studied as an effective
approach to summarizing the source document to produce concise but informative
answers for non-factoid questions. In this work, we propose a novel
question-driven abstractive summarization method, Multi-hop Selective Generator
(MSG), to incorporate multi-hop reasoning into question-driven summarization
and, meanwhile, provide justifications for the generated summaries.
Specifically, we jointly model the relevance to the question and the
interrelation among different sentences via a human-like multi-hop inference
module, which captures important sentences for justifying the summarized
answer. A gated selective pointer generator network with a multi-view coverage
mechanism is designed to integrate diverse information from different
perspectives. Experimental results show that the proposed method consistently
outperforms state-of-the-art methods on two non-factoid QA datasets, namely
WikiHow and PubMedQA.

negative:
Standard neural machine translation (NMT) is on the assumption that the
document-level context is independent. Most existing document-level NMT
approaches are satisfied with a smattering sense of global document-level
information, while this work focuses on exploiting detailed document-level
context in terms of a memory network. The capacity of the memory network that
detecting the most relevant part of the current sentence from memory renders a
natural solution to model the rich document-level context. In this work, the
proposed document-aware memory network is implemented to enhance the
Transformer NMT baseline. Experiments on several tasks show that the proposed
method significantly improves the NMT performance over strong Transformer
baselines and other related studies.

negative:
This paper introduces STRASS: Summarization by TRAnsformation Selection and
Scoring. It is an extractive text summarization method which leverages the
semantic information in existing sentence embedding spaces. Our method creates
an extractive summary by selecting the sentences with the closest embeddings to
the document embedding. The model learns a transformation of the document
embedding to minimize the similarity between the extractive summary and the
ground truth summary. As the transformation is only composed of a dense layer,
the training can be done on CPU, therefore, inexpensive. Moreover, inference
time is short and linear according to the number of sentences. As a second
contribution, we introduce the French CASS dataset, composed of judgments from
the French Court of cassation and their corresponding summaries. On this
dataset, our results show that our method performs similarly to the state of
the art extractive methods with effective training and inferring time.

negative:
Previous researchers have considered sentiment analysis as a document
classification task, in which input documents are classified into predefined
sentiment classes. Although there are sentences in a document that support
important evidences for sentiment analysis and sentences that do not, they have
treated the document as a bag of sentences. In other words, they have not
considered the importance of each sentence in the document. To effectively
determine polarity of a document, each sentence in the document should be dealt
with different degrees of importance. To address this problem, we propose a
document-level sentence classification model based on deep neural networks, in
which the importance degrees of sentences in documents are automatically
determined through gate mechanisms. To verify our new sentiment analysis model,
we conducted experiments using the sentiment datasets in the four different
domains such as movie reviews, hotel reviews, restaurant reviews, and music
reviews. In the experiments, the proposed model outperformed previous
state-of-the-art models that do not consider importance differences of
sentences in a document. The experimental results show that the importance of
sentences should be considered in a document-level sentiment classification
task.

negative:
In the Query Focused Multi-Document Summarization (QF-MDS) task, a set of
documents and a query are given where the goal is to generate a summary from
these documents based on the given query. However, one major challenge for this
task is the lack of availability of labeled training datasets. To overcome this
issue, in this paper, we propose a novel weakly supervised learning approach
via utilizing distant supervision. In particular, we use datasets similar to
the target dataset as the training data where we leverage pre-trained sentence
similarity models to generate the weak reference summary of each individual
document in a document set from the multi-document gold reference summaries.
Then, we iteratively train our summarization model on each single-document to
alleviate the computational complexity issue that occurs while training neural
summarization models in multiple documents (i.e., long sequences) at once.
Experimental results in Document Understanding Conferences (DUC) datasets show
that our proposed approach sets a new state-of-the-art result in terms of
various evaluation metrics.

negative:
Opinion summarization is the automatic creation of text reflecting subjective
information expressed in multiple documents, such as user reviews of a product.
The task is practically important and has attracted a lot of attention.
However, due to the high cost of summary production, datasets large enough for
training supervised models are lacking. Instead, the task has been
traditionally approached with extractive methods that learn to select text
fragments in an unsupervised or weakly-supervised way. Recently, it has been
shown that abstractive summaries, potentially more fluent and better at
reflecting conflicting information, can also be produced in an unsupervised
fashion. However, these models, not being exposed to actual summaries, fail to
capture their essential properties. In this work, we show that even a handful
of summaries is sufficient to bootstrap generation of the summary text with all
expected properties, such as writing style, informativeness, fluency, and
sentiment preservation. We start by training a conditional Transformer language
model to generate a new product review given other available reviews of the
product. The model is also conditioned on review properties that are directly
related to summaries; the properties are derived from reviews with no manual
effort. In the second stage, we fine-tune a plug-in module that learns to
predict property values on a handful of summaries. This lets us switch the
generator to the summarization mode. We show on Amazon and Yelp datasets that
our approach substantially outperforms previous extractive and abstractive
methods in automatic and human evaluation.

negative:
We present a method for generating comparative summaries that highlights
similarities and contradictions in input documents. The key challenge in
creating such summaries is the lack of large parallel training data required
for training typical summarization systems. To this end, we introduce a hybrid
generation approach inspired by traditional concept-to-text systems. To enable
accurate comparison between different sources, the model first learns to
extract pertinent relations from input documents. The content planning
component uses deterministic operators to aggregate these relations after
identifying a subset for inclusion into a summary. The surface realization
component lexicalizes this information using a text-infilling language model.
By separately modeling content selection and realization, we can effectively
train them with limited annotations. We implemented and tested the model in the
domain of nutrition and health -- rife with inconsistencies. Compared to
conventional methods, our framework leads to more faithful, relevant and
aggregation-sensitive summarization -- while being equally fluent.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
We present a robust approach for detecting intrinsic sentence importance in
news, by training on two corpora of document-summary pairs. When used for
single-document summarization, our approach, combined with the "beginning of
document" heuristic, outperforms a state-of-the-art summarizer and the
beginning-of-article baseline in both automatic and manual evaluations. These
results represent an important advance because in the absence of cross-document
repetition, single document summarizers for news have not been able to
consistently outperform the strong beginning-of-article baseline.

negative:
We present a contrasting learning approach with data augmentation techniques
to learn document representations in an unsupervised manner. Inspired by recent
contrastive self-supervised learning algorithms used for image and NLP
pretraining, we hypothesize that high-quality document embedding should be
invariant to diverse paraphrases that preserve the semantics of the original
document. With different backbones and contrastive learning frameworks, our
study reveals the enormous benefits of contrastive augmentation for document
representation learning with two additional insights: 1) including data
augmentation in a contrastive way can substantially improve the embedding
quality in unsupervised document representation learning, and 2) in general,
stochastic augmentations generated by simple word-level manipulation work much
better than sentence-level and document-level ones. We plug our method into a
classifier and compare it with a broad range of baseline methods on six
benchmark datasets. Our method can decrease the classification error rate by up
to 6.4% over the SOTA approaches on the document classification task, matching
or even surpassing fully-supervised methods.

negative:
Statistical methods have been widely employed in recent years to grasp many
language properties. The application of such techniques have allowed an
improvement of several linguistic applications, which encompasses machine
translation, automatic summarization and document classification. In the
latter, many approaches have emphasized the semantical content of texts, as it
is the case of bag-of-word language models. This approach has certainly yielded
reasonable performance. However, some potential features such as the structural
organization of texts have been used only on a few studies. In this context, we
probe how features derived from textual structure analysis can be effectively
employed in a classification task. More specifically, we performed a supervised
classification aiming at discriminating informative from imaginative documents.
Using a networked model that describes the local topological/dynamical
properties of function words, we achieved an accuracy rate of up to 95%, which
is much higher than similar networked approaches. A systematic analysis of
feature relevance revealed that symmetry and accessibility measurements are
among the most prominent network measurements. Our results suggest that these
measurements could be used in related language applications, as they play a
complementary role in characterizing texts.

negative:
With the increase of information, document classification as one of the
methods of text mining, plays vital role in many management and organizing
information. Document classification is the process of assigning a document to
one or more predefined category labels. Document classification includes
different parts such as text processing, term selection, term weighting and
final classification. The accuracy of document classification is very
important. Thus improvement in each part of classification should lead to
better results and higher precision. Term weighting has a great impact on the
accuracy of the classification. Most of the existing weighting methods exploit
the statistical information of terms in documents and do not consider semantic
relations between words. In this paper, an automated document classification
system is presented that uses a novel term weighting method based on semantic
relations between terms. To evaluate the proposed method, three standard
Persian corpuses are used. Experiment results show 2 to 4 percent improvement
in classification accuracy compared with the best previous designed system for
Persian documents.

negative:
Many Natural Language Processing and Computational Linguistics applications
involves the generation of new texts based on some existing texts, such as
summarization, text simplification and machine translation. However, there has
been a serious problem haunting these applications for decades, that is, how to
automatically and accurately assess quality of these applications. In this
paper, we will present some preliminary results on one especially useful and
challenging problem in NLP system evaluation: how to pinpoint content
differences of two text passages (especially for large pas-sages such as
articles and books). Our idea is intuitive and very different from existing
approaches. We treat one text passage as a small knowledge base, and ask it a
large number of questions to exhaustively identify all content points in it. By
comparing the correctly answered questions from two text passages, we will be
able to compare their content precisely. The experiment using 2007 DUC
summarization corpus clearly shows promising results.

negative:
The increasing amount of online content motivated the development of
multi-document summarization methods. In this work, we explore straightforward
approaches to extend single-document summarization methods to multi-document
summarization. The proposed methods are based on the hierarchical combination
of single-document summaries, and achieves state of the art results.

negative:
Document AI, or Document Intelligence, is a relatively new research topic
that refers to the techniques for automatically reading, understanding, and
analyzing business documents. It is an important research direction for natural
language processing and computer vision. In recent years, the popularity of
deep learning technology has greatly advanced the development of Document AI,
such as document layout analysis, visual information extraction, document
visual question answering, document image classification, etc. This paper
briefly reviews some of the representative models, tasks, and benchmark
datasets. Furthermore, we also introduce early-stage heuristic rule-based
document analysis, statistical machine learning algorithms, and deep learning
approaches especially pre-training methods. Finally, we look into future
directions for Document AI research.

negative:
Recent studies emphasize the need of document context in human evaluation of
machine translations, but little research has been done on the impact of user
interfaces on annotator productivity and the reliability of assessments. In
this work, we compare human assessment data from the last two WMT evaluation
campaigns collected via two different methods for document-level evaluation.
Our analysis shows that a document-centric approach to evaluation where the
annotator is presented with the entire document context on a screen leads to
higher quality segment and document level assessments. It improves the
correlation between segment and document scores and increases inter-annotator
agreement for document scores but is considerably more time consuming for
annotators.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.

negative:
We propose a sequence-to-sequence singing synthesizer, which avoids the need
for training data with pre-aligned phonetic and acoustic features. Rather than
the more common approach of a content-based attention mechanism combined with
an autoregressive decoder, we use a different mechanism suitable for
feed-forward synthesis. Given that phonetic timings in singing are highly
constrained by the musical score, we derive an approximate initial alignment
with the help of a simple duration model. Then, using a decoder based on a
feed-forward variant of the Transformer model, a series of self-attention and
convolutional layers refines the result of the initial alignment to reach the
target acoustic features. Advantages of this approach include faster inference
and avoiding the exposure bias issues that affect autoregressive models trained
by teacher forcing. We evaluate the effectiveness of this model compared to an
autoregressive baseline, the importance of self-attention, and the importance
of the accuracy of the duration model.

negative:
Additive Manufacturing (AM) is a crucial component of the smart industry. In
this paper, we propose an automated quality grading system for the AM process
using a deep convolutional neural network (CNN) model. The CNN model is trained
offline using the images of the internal and surface defects in the
layer-by-layer deposition of materials and tested online by studying the
performance of detecting and classifying the failure in AM process at different
extruder speeds and temperatures. The model demonstrates the accuracy of 94%
and specificity of 96%, as well as above 75% in three classifier measures of
the Fscore, the sensitivity, and precision for classifying the quality of the
printing process in five grades in real-time. The proposed online model adds an
automated, consistent, and non-contact quality control signal to the AM process
that eliminates the manual inspection of parts after they are entirely built.
The quality monitoring signal can also be used by the machine to suggest
remedial actions by adjusting the parameters in real-time. The proposed quality
predictive model serves as a proof-of-concept for any type of AM machines to
produce reliable parts with fewer quality hiccups while limiting the waste of
both time and materials.

negative:
We investigate the pertinence of methods from algebraic topology for text
data analysis. These methods enable the development of
mathematically-principled isometric-invariant mappings from a set of vectors to
a document embedding, which is stable with respect to the geometry of the
document in the selected metric space. In this work, we evaluate the utility of
these topology-based document representations in traditional NLP tasks,
specifically document clustering and sentiment classification. We find that the
embeddings do not benefit text analysis. In fact, performance is worse than
simple techniques like $\textit{tf-idf}$, indicating that the geometry of the
document does not provide enough variability for classification on the basis of
topic or sentiment in the chosen datasets.