query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : As a healthcare organization, our goal is to create a system that can effectively extract crucial information from Electronic Health Records (EHRs) to enhance patient care and medical research. Given the challenge of limited labeled data in EHRs, we intend to utilize transfer learning and cross-lingual methods to train a model for this task. Consequently, we are interested in identifying tasks that bear the highest similarity to information extraction from EHRs. Due to privacy concerns, we are unable to test the model on actual EHRs. Therefore, we require methods to generate datasets that closely resemble our format, using fictitious patient information, to test our model. Furthermore, since some EHRs may not convert doctors’ notes into text, the model should be capable of interpreting the doctor's handwriting and accurately extracting information from these notes. We have a preference for pre-trained models that have been benchmarked or models that come with training codes.

positive:
Electronic Health Records (EHRs) are commonly used by the machine learning
community for research on problems specifically related to health care and
medicine. EHRs have the advantages that they can be easily distributed and
contain many features useful for e.g. classification problems. What makes EHR
data sets different from typical machine learning data sets is that they are
often very sparse, due to their high dimensionality, and often contain
heterogeneous (mixed) data types. Furthermore, the data sets deal with
sensitive information, which limits the distribution of any models learned
using them, due to privacy concerns. For these reasons, using EHR data in
practice presents a real challenge. In this work, we explore using Generative
Adversarial Networks to generate synthetic, heterogeneous EHRs with the goal of
using these synthetic records in place of existing data sets for downstream
classification tasks. We will further explore applying differential privacy
(DP) preserving optimization in order to produce DP synthetic EHR data sets,
which provide rigorous privacy guarantees, and are therefore shareable and
usable in the real world. The performance (measured by AUROC, AUPRC and
accuracy) of our model's synthetic, heterogeneous data is very close to the
original data set (within 3 - 5% of the baseline) for the non-DP model when
tested in a binary classification task. Using strong $(1, 10^{-5})$ DP, our
model still produces data useful for machine learning tasks, albeit incurring a
roughly 17% performance penalty in our tested classification task. We
additionally perform a sub-population analysis and find that our model does not
introduce any bias into the synthetic EHR data compared to the baseline in
either male/female populations, or the 0-18, 19-50 and 51+ age groups in terms
of classification performance for either the non-DP or DP variant.

positive:
Extractive summarization is very useful for physicians to better manage and
digest Electronic Health Records (EHRs). However, the training of a supervised
model requires disease-specific medical background and is thus very expensive.
We studied how to utilize the intrinsic correlation between multiple EHRs to
generate pseudo-labels and train a supervised model with no external
annotation. Experiments on real-patient data validate that our model is
effective in summarizing crucial disease-specific information for patients.

positive:
Electronic health records (EHRs), digital collections of patient healthcare
events and observations, are ubiquitous in medicine and critical to healthcare
delivery, operations, and research. Despite this central role, EHRs are
notoriously difficult to process automatically. Well over half of the
information stored within EHRs is in the form of unstructured text (e.g.
provider notes, operation reports) and remains largely untapped for secondary
use. Recently, however, newer neural network and deep learning approaches to
Natural Language Processing (NLP) have made considerable advances,
outperforming traditional statistical and rule-based systems on a variety of
tasks. In this survey paper, we summarize current neural NLP methods for EHR
applications. We focus on a broad scope of tasks, namely, classification and
prediction, word embeddings, extraction, generation, and other topics such as
question answering, phenotyping, knowledge graphs, medical dialogue,
multilinguality, interpretability, etc.

positive:
Researchers require timely access to real-world longitudinal electronic
health records (EHR) to develop, test, validate, and implement machine learning
solutions that improve the quality and efficiency of healthcare. In contrast,
health systems value deeply patient privacy and data security. De-identified
EHRs do not adequately address the needs of health systems, as de-identified
data are susceptible to re-identification and its volume is also limited.
Synthetic EHRs offer a potential solution. In this paper, we propose EHR
Variational Autoencoder (EVA) for synthesizing sequences of discrete EHR
encounters (e.g., clinical visits) and encounter features (e.g., diagnoses,
medications, procedures). We illustrate that EVA can produce realistic EHR
sequences, account for individual differences among patients, and can be
conditioned on specific disease conditions, thus enabling disease-specific
studies. We design efficient, accurate inference algorithms by combining
stochastic gradient Markov Chain Monte Carlo with amortized variational
inference. We assess the utility of the methods on large real-world EHR
repositories containing over 250, 000 patients. Our experiments, which include
user studies with knowledgeable clinicians, indicate the generated EHR
sequences are realistic. We confirmed the performance of predictive models
trained on the synthetic data are similar with those trained on real EHRs.
Additionally, our findings indicate that augmenting real data with synthetic
EHRs results in the best predictive performance - improving the best baseline
by as much as 8% in top-20 recall.

positive:
Electronic Health Records (EHRs) provide vital contextual information to
radiologists and other physicians when making a diagnosis. Unfortunately,
because a given patient's record may contain hundreds of notes and reports,
identifying relevant information within these in the short time typically
allotted to a case is very difficult. We propose and evaluate models that
extract relevant text snippets from patient records to provide a rough case
summary intended to aid physicians considering one or more diagnoses. This is
hard because direct supervision (i.e., physician annotations of snippets
relevant to specific diagnoses in medical records) is prohibitively expensive
to collect at scale. We propose a distantly supervised strategy in which we use
groups of International Classification of Diseases (ICD) codes observed in
'future' records as noisy proxies for 'downstream' diagnoses. Using this we
train a transformer-based neural model to perform extractive summarization
conditioned on potential diagnoses. This model defines an attention mechanism
that is conditioned on potential diagnoses (queries) provided by the diagnosing
physician. We train (via distant supervision) and evaluate variants of this
model on EHR data from Brigham and Women's Hospital in Boston and MIMIC-III
(the latter to facilitate reproducibility). Evaluations performed by
radiologists demonstrate that these distantly supervised models yield better
extractive summaries than do unsupervised approaches. Such models may aid
diagnosis by identifying sentences in past patient reports that are clinically
relevant to a potential diagnosis.

positive:
The field of clinical natural language processing has been advanced
significantly since the introduction of deep learning models. The
self-supervised representation learning and the transfer learning paradigm
became the methods of choice in many natural language processing application,
in particular in the settings with the dearth of high quality manually
annotated data. Electronic health record systems are ubiquitous and the
majority of patients' data are now being collected electronically and in
particular in the form of free text. Identification of medical concepts and
information extraction is a challenging task, yet important ingredient for
parsing unstructured data into structured and tabulated format for downstream
analytical tasks. In this work we introduced a named-entity recognition model
for clinical natural language processing. The model is trained to recognise
seven categories: drug names, route, frequency, dosage, strength, form,
duration. The model was first self-supervisedly pre-trained by predicting the
next word, using a collection of 2 million free-text patients' records from
MIMIC-III corpora and then fine-tuned on the named-entity recognition task. The
model achieved a lenient (strict) micro-averaged F1 score of 0.957 (0.893)
across all seven categories. Additionally, we evaluated the transferability of
the developed model using the data from the Intensive Care Unit in the US to
secondary care mental health records (CRIS) in the UK. A direct application of
the trained NER model to CRIS data resulted in reduced performance of F1=0.762,
however after fine-tuning on a small sample from CRIS, the model achieved a
reasonable performance of F1=0.944. This demonstrated that despite a close
similarity between the data sets and the NER tasks, it is essential to
fine-tune on the target domain data in order to achieve more accurate results.

positive:
In this paper, we present our approach to extracting structured information
from unstructured Electronic Health Records (EHR) [2] which can be used to, for
example, study adverse drug reactions in patients due to chemicals in their
products. Our solution uses a combination of Natural Language Processing (NLP)
techniques and a web-based annotation tool to optimize the performance of a
custom Named Entity Recognition (NER) [1] model trained on a limited amount of
EHR training data. This work was presented at the first Health Search and Data
Mining Workshop (HSDM 2020) [26]. We showcase a combination of tools and
techniques leveraging the recent advancements in NLP aimed at targeting domain
shifts by applying transfer learning and language model pre-training techniques
[3]. We present a comparison of our technique to the current popular approaches
and show the effective increase in performance of the NER model and the
reduction in time to annotate data.A key observation of the results presented
is that the F1 score of model (0.734) trained with our approach with just 50%
of available training data outperforms the F1 score of the blank spaCy model
without language model component (0.704) trained with 100% of the available
training data. We also demonstrate an annotation tool to minimize domain expert
time and the manual effort required to generate such a training dataset.
Further, we plan to release the annotated dataset as well as the pre-trained
model to the community to further research in medical health records.

positive:
Deep learning models have shown tremendous potential in learning
representations, which are able to capture some key properties of the data.
This makes them great candidates for transfer learning: Exploiting
commonalities between different learning tasks to transfer knowledge from one
task to another. Electronic health records (EHR) research is one of the domains
that has witnessed a growing number of deep learning techniques employed for
learning clinically-meaningful representations of medical concepts (such as
diseases and medications). Despite this growth, the approaches to benchmark and
assess such learned representations (or, embeddings) is under-investigated;
this can be a big issue when such embeddings are shared to facilitate transfer
learning. In this study, we aim to (1) train some of the most prominent disease
embedding techniques on a comprehensive EHR data from 3.1 million patients, (2)
employ qualitative and quantitative evaluation techniques to assess these
embeddings, and (3) provide pre-trained disease embeddings for transfer
learning. This study can be the first comprehensive approach for clinical
concept embedding evaluation and can be applied to any embedding techniques and
for any EHR concept.

positive:
Electronic Health Record (EHR) data has been of tremendous utility in
Artificial Intelligence (AI) for healthcare such as predicting future clinical
events. These tasks, however, often come with many challenges when using
classical machine learning models due to a myriad of factors including class
imbalance and data heterogeneity (i.e., the complex intra-class variances). To
address some of these research gaps, this paper leverages the exciting
contrastive learning framework and proposes a novel contrastive regularized
clinical classification model. The contrastive loss is found to substantially
augment EHR-based prediction: it effectively characterizes the
similar/dissimilar patterns (by its "push-and-pull" form), meanwhile mitigating
the highly skewed class distribution by learning more balanced feature spaces
(as also echoed by recent findings). In particular, when naively exporting the
contrastive learning to the EHR data, one hurdle is in generating positive
samples, since EHR data is not as amendable to data augmentation as image data.
To this end, we have introduced two unique positive sampling strategies
specifically tailored for EHR data: a feature-based positive sampling that
exploits the feature space neighborhood structure to reinforce the feature
learning; and an attribute-based positive sampling that incorporates
pre-generated patient similarity metrics to define the sample proximity. Both
sampling approaches are designed with an awareness of unique high intra-class
variance in EHR data. Our overall framework yields highly competitive
experimental results in predicting the mortality risk on real-world COVID-19
EHR data with a total of 5,712 patients admitted to a large, urban health
system. Specifically, our method reaches a high AUROC prediction score of
0.959, which outperforms other baselines and alternatives: cross-entropy(0.873)
and focal loss(0.931).

positive:
The recent adoption of Electronic Health Records (EHRs) by health care
providers has introduced an important source of data that provides detailed and
highly specific insights into patient phenotypes over large cohorts. These
datasets, in combination with machine learning and statistical approaches,
generate new opportunities for research and clinical care. However, many
methods require the patient representations to be in structured formats, while
the information in the EHR is often locked in unstructured texts designed for
human readability. In this work, we develop the methodology to automatically
extract clinical features from clinical narratives from large EHR corpora
without the need for prior knowledge. We consider medical terms and sentences
appearing in clinical narratives as atomic information units. We propose an
efficient clustering strategy suitable for the analysis of large text corpora
and to utilize the clusters to represent information about the patient
compactly. To demonstrate the utility of our approach, we perform an
association study of clinical features with somatic mutation profiles from
4,007 cancer patients and their tumors. We apply the proposed algorithm to a
dataset consisting of about 65 thousand documents with a total of about 3.2
million sentences. We identify 341 significant statistical associations between
the presence of somatic mutations and clinical features. We annotated these
associations according to their novelty, and report several known associations.
We also propose 32 testable hypotheses where the underlying biological
mechanism does not appear to be known but plausible. These results illustrate
that the automated discovery of clinical features is possible and the joint
analysis of clinical and genetic datasets can generate appealing new
hypotheses.

negative:
Automatically extracting useful information from electronic medical records
along with conducting disease diagnoses is a promising task for both clinical
decision support(CDS) and neural language processing(NLP). Most of the existing
systems are based on artificially constructed knowledge bases, and then
auxiliary diagnosis is done by rule matching. In this study, we present a
clinical intelligent decision approach based on Convolutional Neural
Networks(CNN), which can automatically extract high-level semantic information
of electronic medical records and then perform automatic diagnosis without
artificial construction of rules or knowledge bases. We use collected 18,590
copies of the real-world clinical electronic medical records to train and test
the proposed model. Experimental results show that the proposed model can
achieve 98.67\% accuracy and 96.02\% recall, which strongly supports that using
convolutional neural network to automatically learn high-level semantic
features of electronic medical records and then conduct assist diagnosis is
feasible and effective.

negative:
Sharing electronic health records (EHRs) on a large scale may lead to privacy
intrusions. Recent research has shown that risks may be mitigated by simulating
EHRs through generative adversarial network (GAN) frameworks. Yet the methods
developed to date are limited because they 1) focus on generating data of a
single type (e.g., diagnosis codes), neglecting other data types (e.g.,
demographics, procedures or vital signs) and 2) do not represent constraints
between features. In this paper, we introduce a method to simulate EHRs
composed of multiple data types by 1) refining the GAN model, 2) accounting for
feature constraints, and 3) incorporating key utility measures for such
generation tasks. Our analysis with over $770,000$ EHRs from Vanderbilt
University Medical Center demonstrates that the new model achieves higher
performance in terms of retaining basic statistics, cross-feature correlations,
latent structural properties, feature constraints and associated patterns from
real data, without sacrificing privacy.

negative:
The wide adoption of Electronic Health Records (EHR) has resulted in large
amounts of clinical data becoming available, which promises to support service
delivery and advance clinical and informatics research. Deep learning
techniques have demonstrated performance in predictive analytic tasks using
EHRs yet they typically lack model result transparency or explainability
functionalities and require cumbersome pre-processing tasks. Moreover, EHRs
contain heterogeneous and multi-modal data points such as text, numbers and
time series which further hinder visualisation and interpretability. This paper
proposes a deep learning framework to: 1) encode patient pathways from EHRs
into images, 2) highlight important events within pathway images, and 3) enable
more complex predictions with additional intelligibility. The proposed method
relies on a deep attention mechanism for visualisation of the predictions and
allows predicting multiple sequential outcomes.

negative:
Evaluating the clinical similarities between pairwise patients is a
fundamental problem in healthcare informatics. A proper patient similarity
measure enables various downstream applications, such as cohort study and
treatment comparative effectiveness research. One major carrier for conducting
patient similarity research is Electronic Health Records(EHRs), which are
usually heterogeneous, longitudinal, and sparse. Though existing studies on
learning patient similarity from EHRs have shown being useful in solving real
clinical problems, their applicability is limited due to the lack of medical
interpretations. Moreover, most previous methods assume a vector-based
representation for patients, which typically requires aggregation of medical
events over a certain time period. As a consequence, temporal information will
be lost. In this paper, we propose a patient similarity evaluation framework
based on the temporal matching of longitudinal patient EHRs. Two efficient
methods are presented, unsupervised and supervised, both of which preserve the
temporal properties in EHRs. The supervised scheme takes a convolutional neural
network architecture and learns an optimal representation of patient clinical
records with medical concept embedding. The empirical results on real-world
clinical data demonstrate substantial improvement over the baselines. We make
our code and sample data available for further study.

negative:
The widespread availability of electronic health records (EHRs) promises to
usher in the era of personalized medicine. However, the problem of extracting
useful clinical representations from longitudinal EHR data remains challenging.
In this paper, we explore deep neural network models with learned medical
feature embedding to deal with the problems of high dimensionality and
temporality. Specifically, we use a multi-layer convolutional neural network
(CNN) to parameterize the model and is thus able to capture complex non-linear
longitudinal evolution of EHRs. Our model can effectively capture local/short
temporal dependency in EHRs, which is beneficial for risk prediction. To
account for high dimensionality, we use the embedding medical features in the
CNN model which hold the natural medical concepts. Our initial experiments
produce promising results and demonstrate the effectiveness of both the medical
feature embedding and the proposed convolutional neural network in risk
prediction on cohorts of congestive heart failure and diabetes patients
compared with several strong baselines.

negative:
Healthcare is becoming a more and more important research topic recently.
With the growing data in the healthcare domain, it offers a great opportunity
for deep learning to improve the quality of medical service. However, the
complexity of electronic health records (EHR) data is a challenge for the
application of deep learning. Specifically, the data produced in the hospital
admissions are monitored by the EHR system, which includes structured data like
daily body temperature, and unstructured data like free text and laboratory
measurements. Although there are some preprocessing frameworks proposed for
specific EHR data, the clinical notes that contain significant clinical value
are beyond the realm of their consideration. Besides, whether these different
data from various views are all beneficial to the medical tasks and how to best
utilize these data remain unclear. Therefore, in this paper, we first extract
the accompanying clinical notes from EHR and propose a method to integrate
these data, we also comprehensively study the different models and the data
leverage methods for better medical task prediction. The results on two medical
prediction tasks show that our fused model with different data outperforms the
state-of-the-art method that without clinical notes, which illustrates the
importance of our fusion method and the value of clinical note features. Our
code is available at https: //github.com/emnlp-mimic/mimic.

negative:
Electronic health records (EHR) systems contain vast amounts of medical
information about patients. These data can be used to train machine learning
models that can predict health status, as well as to help prevent future
diseases or disabilities. However, getting patients' medical data to obtain
well-trained machine learning models is a challenging task. This is because
sharing the patients' medical records is prohibited by law in most countries
due to patients privacy concerns. In this paper, we tackle this problem by
sharing the models instead of the original sensitive data by using the mimic
learning approach. The idea is first to train a model on the original sensitive
data, called the teacher model. Then, using this model, we can transfer its
knowledge to another model, called the student model, without the need to learn
the original data used in training the teacher model. The student model is then
shared to the public and can be used to make accurate predictions. To assess
the mimic learning approach, we have evaluated our scheme using different
medical datasets. The results indicate that the student model mimics the
teacher model performance in terms of prediction accuracy without the need to
access to the patients' original data records.

negative:
Electronic Health Records (EHR) have been heavily used in modern healthcare
systems for recording patients' admission information to hospitals. Many
data-driven approaches employ temporal features in EHR for predicting specific
diseases, readmission times, or diagnoses of patients. However, most existing
predictive models cannot fully utilize EHR data, due to an inherent lack of
labels in supervised training for some temporal events. Moreover, it is hard
for existing works to simultaneously provide generic and personalized
interpretability. To address these challenges, we first propose a hyperbolic
embedding method with information flow to pre-train medical code
representations in a hierarchical structure. We incorporate these pre-trained
representations into a graph neural network to detect disease complications,
and design a multi-level attention method to compute the contributions of
particular diseases and admissions, thus enhancing personalized
interpretability. We present a new hierarchy-enhanced historical prediction
proxy task in our self-supervised learning framework to fully utilize EHR data
and exploit medical domain knowledge. We conduct a comprehensive set of
experiments and case studies on widely used publicly available EHR datasets to
verify the effectiveness of our model. The results demonstrate our model's
strengths in both predictive tasks and interpretable abilities.

negative:
Early detection of preventable diseases is important for better disease
management, improved inter-ventions, and more efficient health-care resource
allocation. Various machine learning approacheshave been developed to utilize
information in Electronic Health Record (EHR) for this task. Majorityof
previous attempts, however, focus on structured fields and lose the vast amount
of information inthe unstructured notes. In this work we propose a general
multi-task framework for disease onsetprediction that combines both free-text
medical notes and structured information. We compareperformance of different
deep learning architectures including CNN, LSTM and hierarchical models.In
contrast to traditional text-based prediction models, our approach does not
require disease specificfeature engineering, and can handle negations and
numerical values that exist in the text. Ourresults on a cohort of about 1
million patients show that models using text outperform modelsusing just
structured data, and that models capable of using numerical values and
negations in thetext, in addition to the raw text, further improve performance.
Additionally, we compare differentvisualization methods for medical
professionals to interpret model predictions.

negative:
The development of neural networks for clinical artificial intelligence (AI)
is reliant on interpretability, transparency, and performance. The need to
delve into the black-box neural network and derive interpretable explanations
of model output is paramount. A task of high clinical importance is predicting
the likelihood of a patient being readmitted to hospital in the near future to
enable efficient triage. With the increasing adoption of electronic health
records (EHRs), there is great interest in applications of natural language
processing (NLP) to clinical free-text contained within EHRs. In this work, we
apply InfoCal, the current state-of-the-art model that produces extractive
rationales for its predictions, to the task of predicting hospital readmission
using hospital discharge notes. We compare extractive rationales produced by
InfoCal to competitive transformer-based models pretrained on clinical text
data and for which the attention mechanism can be used for interpretation. We
find each presented model with selected interpretability or feature importance
methods yield varying results, with clinical language domain expertise and
pretraining critical to performance and subsequent interpretability.

negative:
Electronic health records (EHRs) contain structured and unstructured data of
significant clinical and research value. Various machine learning approaches
have been developed to employ information in EHRs for risk prediction. The
majority of these attempts, however, focus on structured EHR fields and lose
the vast amount of information in the unstructured texts. To exploit the
potential information captured in EHRs, in this study we propose a multimodal
recurrent neural network model for cardiovascular risk prediction that
integrates both medical texts and structured clinical information. The proposed
multimodal bidirectional long short-term memory (BiLSTM) model concatenates
word embeddings to classical clinical predictors before applying them to a
final fully connected neural network. In the experiments, we compare
performance of different deep neural network (DNN) architectures including
convolutional neural network and long short-term memory in scenarios of using
clinical variables and chest X-ray radiology reports. Evaluated on a data set
of real world patients with manifest vascular disease or at high-risk for
cardiovascular disease, the proposed BiLSTM model demonstrates state-of-the-art
performance and outperforms other DNN baseline architectures.

negative:
Widespread adoption of electronic health records (EHRs) has fueled the
development of using machine learning to build prediction models for various
clinical outcomes. This process is often constrained by having a relatively
small number of patient records for training the model. We demonstrate that
using patient representation schemes inspired from techniques in natural
language processing can increase the accuracy of clinical prediction models by
transferring information learned from the entire patient population to the task
of training a specific model, where only a subset of the population is
relevant. Such patient representation schemes enable a 3.5% mean improvement in
AUROC on five prediction tasks compared to standard baselines, with the average
improvement rising to 19% when only a small number of patient records are
available for training the clinical prediction model.

negative:
The shift to electronic medical records (EMRs) has engendered research into
machine learning and natural language technologies to analyze patient records,
and to predict from these clinical outcomes of interest. Two observations
motivate our aims here. First, unstructured notes contained within EMR often
contain key information, and hence should be exploited by models. Second, while
strong predictive performance is important, interpretability of models is
perhaps equally so for applications in this domain. Together, these points
suggest that neural models for EMR may benefit from incorporation of attention
over notes, which one may hope will both yield performance gains and afford
transparency in predictions. In this work we perform experiments to explore
this question using two EMR corpora and four different predictive tasks, that:
(i) inclusion of attention mechanisms is critical for neural encoder modules
that operate over notes fields in order to yield competitive performance, but,
(ii) unfortunately, while these boost predictive performance, it is decidedly
less clear whether they provide meaningful support for predictions.

negative:
Deep learning models in healthcare may fail to generalize on data from unseen
corpora. Additionally, no quantitative metric exists to tell how existing
models will perform on new data. Previous studies demonstrated that NLP models
of medical notes generalize variably between institutions, but ignored other
levels of healthcare organization. We measured SciBERT diagnosis sentiment
classifier generalizability between medical specialties using EHR sentences
from MIMIC-III. Models trained on one specialty performed better on internal
test sets than mixed or external test sets (mean AUCs 0.92, 0.87, and 0.83,
respectively; p = 0.016). When models are trained on more specialties, they
have better test performances (p < 1e-4). Model performance on new corpora is
directly correlated to the similarity between train and test sentence content
(p < 1e-4). Future studies should assess additional axes of generalization to
ensure deep learning models fulfil their intended purpose across institutions,
specialties, and practices.

negative:
The rapid growth of Electronic Health Records (EHRs), as well as the
accompanied opportunities in Data-Driven Healthcare (DDH), has been attracting
widespread interests and attentions. Recent progress in the design and
applications of deep learning methods has shown promising results and is
forcing massive changes in healthcare academia and industry, but most of these
methods rely on massive labeled data. In this work, we propose a general deep
learning framework which is able to boost risk prediction performance with
limited EHR data. Our model takes a modified generative adversarial network
namely ehrGAN, which can provide plausible labeled EHR data by mimicking real
patient records, to augment the training dataset in a semi-supervised learning
manner. We use this generative model together with a convolutional neural
network (CNN) based prediction model to improve the onset prediction
performance. Experiments on two real healthcare datasets demonstrate that our
proposed framework produces realistic data samples and achieves significant
improvements on classification tasks with the generated data over several
stat-of-the-art baselines.

negative:
A crucial step within secondary analysis of electronic health records (EHRs)
is to identify the patient cohort under investigation. While EHRs contain
medical billing codes that aim to represent the conditions and treatments
patients may have, much of the information is only present in the patient
notes. Therefore, it is critical to develop robust algorithms to infer
patients' conditions and treatments from their written notes. In this paper, we
introduce a dataset for patient phenotyping, a task that is defined as the
identification of whether a patient has a given medical condition (also
referred to as clinical indication or phenotype) based on their patient note.
Nursing Progress Notes and Discharge Summaries from the Intensive Care Unit of
a large tertiary care hospital were manually annotated for the presence of
several high-context phenotypes relevant to treatment and risk of
re-hospitalization. This dataset contains 1102 Discharge Summaries and 1000
Nursing Progress Notes. Each Discharge Summary and Progress Note has been
annotated by at least two expert human annotators (one clinical researcher and
one resident physician). Annotated phenotypes include treatment non-adherence,
chronic pain, advanced/metastatic cancer, as well as 10 other phenotypes. This
dataset can be utilized for academic and industrial research in medicine and
computer science, particularly within the field of medical natural language
processing.

negative:
Biomedical documents such as Electronic Health Records (EHRs) contain a large
amount of information in an unstructured format. The data in EHRs is a hugely
valuable resource documenting clinical narratives and decisions, but whilst the
text can be easily understood by human doctors it is challenging to use in
research and clinical applications. To uncover the potential of biomedical
documents we need to extract and structure the information they contain. The
task at hand is Named Entity Recognition and Linking (NER+L). The number of
entities, ambiguity of words, overlapping and nesting make the biomedical area
significantly more difficult than many others. To overcome these difficulties,
we have developed the Medical Concept Annotation Tool (MedCAT), an open-source
unsupervised approach to NER+L. MedCAT uses unsupervised machine learning to
disambiguate entities. It was validated on MIMIC-III (a freely accessible
critical care database) and MedMentions (Biomedical papers annotated with
mentions from the Unified Medical Language System). In case of NER+L, the
comparison with existing tools shows that MedCAT improves the previous best
with only unsupervised learning (F1=0.848 vs 0.691 for disease detection;
F1=0.710 vs. 0.222 for general concept detection). A qualitative analysis of
the vector embeddings learnt by MedCAT shows that it captures latent medical
knowledge available in EHRs (MIMIC-III). Unsupervised learning can improve the
performance of large scale entity extraction, but it has some limitations when
working with only a couple of entities and a small dataset. In that case
options are supervised learning or active learning, both of which are supported
in MedCAT via the MedCATtrainer extension. Our approach can detect and link
millions of different biomedical concepts with state-of-the-art performance,
whilst being lightweight, fast and easy to use.

negative:
Extracting information from electronic health records (EHR) is a challenging
task since it requires prior knowledge of the reports and some natural language
processing algorithm (NLP). With the growing number of EHR implementations,
such knowledge is increasingly challenging to obtain in an efficient manner. We
address this challenge by proposing a novel methodology to analyze large sets
of EHRs using a modified Count Sketch data streaming algorithm termed DreamNLP.
By using DreamNLP, we generate a dictionary of frequently occurring terms or
heavy hitters in the EHRs using low computational memory compared to
conventional counting approach other NLP programs use. We demonstrate the
extraction of the most important breast diagnosis features from the EHRs in a
set of patients that underwent breast imaging. Based on the analysis,
extraction of these terms would be useful for defining important features for
downstream tasks such as machine learning for precision medicine.

negative:
Understanding patients' journeys in healthcare system is a fundamental
prepositive task for a broad range of AI-based healthcare applications. This
task aims to learn an informative representation that can comprehensively
encode hidden dependencies among medical events and its inner entities, and
then the use of encoding outputs can greatly benefit the downstream
application-driven tasks. A patient journey is a sequence of electronic health
records (EHRs) over time that is organized at multiple levels: patient, visits
and medical codes. The key challenge of patient journey understanding is to
design an effective encoding mechanism which can properly tackle the
aforementioned multi-level structured patient journey data with temporal
sequential visits and a set of medical codes. This paper proposes a novel
self-attention mechanism that can simultaneously capture the contextual and
temporal relationships hidden in patient journeys. A multi-level self-attention
network (MusaNet) is specifically designed to learn the representations of
patient journeys that is used to be a long sequence of activities. The MusaNet
is trained in end-to-end manner using the training data derived from EHRs. We
evaluated the efficacy of our method on two medical application tasks with
real-world benchmark datasets. The results have demonstrated the proposed
MusaNet produces higher-quality representations than state-of-the-art baseline
methods. The source code is available in https://github.com/xueping/MusaNet.

negative:
In longitudinal electronic health records (EHRs), the event records of a
patient are distributed over a long period of time and the temporal relations
between the events reflect sufficient domain knowledge to benefit prediction
tasks such as the rate of inpatient mortality. Medical concept embedding as a
feature extraction method that transforms a set of medical concepts with a
specific time stamp into a vector, which will be fed into a supervised learning
algorithm. The quality of the embedding significantly determines the learning
performance over the medical data. In this paper, we propose a medical concept
embedding method based on applying a self-attention mechanism to represent each
medical concept. We propose a novel attention mechanism which captures the
contextual information and temporal relationships between medical concepts. A
light-weight neural net, "Temporal Self-Attention Network (TeSAN)", is then
proposed to learn medical concept embedding based solely on the proposed
attention mechanism. To test the effectiveness of our proposed methods, we have
conducted clustering and prediction tasks on two public EHRs datasets comparing
TeSAN against five state-of-the-art embedding methods. The experimental results
demonstrate that the proposed TeSAN model is superior to all the compared
methods. To the best of our knowledge, this work is the first to exploit
temporal self-attentive relations between medical events.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
Recent grid-based document representations like BERTgrid allow the
simultaneous encoding of the textual and layout information of a document in a
2D feature map so that state-of-the-art image segmentation and/or object
detection models can be straightforwardly leveraged to extract key information
from documents. However, such methods have not achieved comparable performance
to state-of-the-art sequence- and graph-based methods such as LayoutLM and PICK
yet. In this paper, we propose a new multi-modal backbone network by
concatenating a BERTgrid to an intermediate layer of a CNN model, where the
input of CNN is a document image and the BERTgrid is a grid of word embeddings,
to generate a more powerful grid-based document representation, named
ViBERTgrid. Unlike BERTgrid, the parameters of BERT and CNN in our multimodal
backbone network are trained jointly. Our experimental results demonstrate that
this joint training strategy improves significantly the representation ability
of ViBERTgrid. Consequently, our ViBERTgrid-based key information extraction
approach has achieved state-of-the-art performance on real-world datasets.

negative:
In these notes, we wish to provide an introduction to the signature method,
focusing on its basic theoretical properties and recent numerical applications.
  The notes are split into two parts. The first part focuses on the definition
and fundamental properties of the signature of a path, or the path signature.
We have aimed for a minimalistic approach, assuming only familiarity with
classical real analysis and integration theory, and supplementing theory with
straightforward examples. We have chosen to focus in detail on the principle
properties of the signature which we believe are fundamental to understanding
its role in applications. We also present an informal discussion on some of its
deeper properties and briefly mention the role of the signature in rough paths
theory, which we hope could serve as a light introduction to rough paths for
the interested reader.
  The second part of these notes discusses practical applications of the path
signature to the area of machine learning. The signature approach represents a
non-parametric way for extraction of characteristic features from data. The
data are converted into a multi-dimensional path by means of various embedding
algorithms and then processed for computation of individual terms of the
signature which summarise certain information contained in the data. The
signature thus transforms raw data into a set of features which are used in
machine learning tasks. We will review current progress in applications of
signatures to machine learning problems.

negative:
In this work we addressed the problem of capturing sequential information
contained in longitudinal electronic health records (EHRs). Clinical notes,
which is a particular type of EHR data, are a rich source of information and
practitioners often develop clever solutions how to maximise the sequential
information contained in free-texts. We proposed a systematic methodology for
learning from chronological events available in clinical notes. The proposed
methodological {\it path signature} framework creates a non-parametric
hierarchical representation of sequential events of any type and can be used as
features for downstream statistical learning tasks. The methodology was
developed and externally validated using the largest in the UK secondary care
mental health EHR data on a specific task of predicting survival risk of
patients diagnosed with Alzheimer's disease. The signature-based model was
compared to a common survival random forest model. Our results showed a
15.4$\%$ increase of risk prediction AUC at the time point of 20 months after
the first admission to a specialist memory clinic and the signature method
outperformed the baseline mixed-effects model by 13.2 $\%$.

negative:
Clinicians spend a significant amount of time inputting free-form textual
notes into Electronic Health Records (EHR) systems. Much of this documentation
work is seen as a burden, reducing time spent with patients and contributing to
clinician burnout. With the aspiration of AI-assisted note-writing, we propose
a new language modeling task predicting the content of notes conditioned on
past data from a patient's medical record, including patient demographics,
labs, medications, and past notes. We train generative models using the public,
de-identified MIMIC-III dataset and compare generated notes with those in the
dataset on multiple measures. We find that much of the content can be
predicted, and that many common templates found in notes can be learned. We
discuss how such models can be useful in supporting assistive note-writing
features such as error-detection and auto-complete.

negative:
Information in electronic health records (EHR), such as clinical narratives,
examination reports, lab measurements, demographics, and other patient
encounter entries, can be transformed into appropriate data representations
that can be used for downstream clinical machine learning tasks using
representation learning. Learning better representations is critical to improve
the performance of downstream tasks. Due to the advances in machine learning,
we now can learn better and meaningful representations from EHR through
disentangling the underlying factors inside data and distilling large amounts
of information and knowledge from heterogeneous EHR sources. In this chapter,
we first introduce the background of learning representations and reasons why
we need good EHR representations in machine learning for medicine and
healthcare in Section 1. Next, we explain the commonly-used machine learning
and evaluation methods for representation learning using a deep learning
approach in Section 2. Following that, we review recent related studies of
learning patient state representation from EHR for clinical machine learning
tasks in Section 3. Finally, in Section 4 we discuss more techniques, studies,
and challenges for learning natural language representations when free texts,
such as clinical notes, examination reports, or biomedical literature are used.
We also discuss challenges and opportunities in these rapidly growing research
fields.

negative:
Crucial information about the practice of healthcare is recorded only in
free-form text, which creates an enormous opportunity for high-impact NLP.
However, annotated healthcare datasets tend to be small and expensive to
obtain, which raises the question of how to make maximally efficient uses of
the available data. To this end, we develop an LSTM-CRF model for combining
unsupervised word representations and hand-built feature representations
derived from publicly available healthcare ontologies. We show that this
combined model yields superior performance on five datasets of diverse kinds of
healthcare text (clinical, social, scientific, commercial). Each involves the
labeling of complex, multi-word spans that pick out different healthcare
concepts. We also introduce a new labeled dataset for identifying the treatment
relations between drugs and diseases.

negative:
The past decade has seen an explosion in the amount of digital information
stored in electronic health records (EHR). While primarily designed for
archiving patient clinical information and administrative healthcare tasks,
many researchers have found secondary use of these records for various clinical
informatics tasks. Over the same period, the machine learning community has
seen widespread advances in deep learning techniques, which also have been
successfully applied to the vast amount of EHR data. In this paper, we review
these deep EHR systems, examining architectures, technical aspects, and
clinical applications. We also identify shortcomings of current techniques and
discuss avenues of future research for EHR-based deep learning.

negative:
Oral drugs are becoming increasingly common in oncology care. In contrast to
intravenous chemotherapy, which is administered in the clinic and carefully
tracked via structure electronic health records (EHRs), oral drug treatment is
self-administered and therefore not tracked as well. Often, the details of oral
cancer treatment occur only in unstructured clinic notes. Extracting this
information is critical to understanding a patient's treatment history. Yet,
this a challenging task because treatment intervals must be inferred
longitudinally from both explicit mentions in the text as well as from document
timestamps. In this work, we present TIFTI (Temporally Integrated Framework for
Treatment Intervals), a robust framework for extracting oral drug treatment
intervals from a patient's unstructured notes. TIFTI leverages distinct sources
of temporal information by breaking the problem down into two separate
subtasks: document-level sequence labeling and date extraction. On a labeled
dataset of metastatic renal-cell carcinoma (RCC) patients, it exactly matched
the labeled start date in 46% of the examples (86% of the examples within 30
days), and it exactly matched the labeled end date in 52% of the examples (78%
of the examples within 30 days). Without retraining, the model achieved a
similar level of performance on a labeled dataset of advanced non-small-cell
lung cancer (NSCLC) patients.

negative:
In recent years, many methods have been developed to identify important
portions of text documents. Summarization tools can utilize these methods to
extract summaries from large volumes of textual information. However, to
identify concepts representing central ideas within a text document and to
extract the most informative sentences that best convey those concepts still
remain two crucial tasks in summarization methods. In this paper, we introduce
a graph-based method to address these two challenges in the context of
biomedical text summarization. We show that how a summarizer can discover
meaningful concepts within a biomedical text document using the Helmholtz
principle. The summarizer considers the meaningful concepts as the main topics
and constructs a graph based on the topics that the sentences share. The
summarizer can produce an informative summary by extracting those sentences
having higher values of the degree. We assess the performance of our method for
summarization of biomedical articles using the Recall-Oriented Understudy for
Gisting Evaluation (ROUGE) toolkit. The results show that the degree can be a
useful centrality measure to identify important sentences in this type of
graph-based modelling. Our method can improve the performance of biomedical
text summarization compared to some state-of-the-art and publicly available
summarizers. Combining a concept-based modelling strategy and a graph-based
approach to sentence extraction, our summarizer can produce summaries with the
highest scores of informativeness among the comparison methods. This research
work can be regarded as a start point to the study of small-world networks in
summarization of biomedical texts.

negative:
The data available in Electronic Health Records (EHRs) provides the
opportunity to transform care, and the best way to provide better care for one
patient is through learning from the data available on all other patients.
Temporal modelling of a patient's medical history, which takes into account the
sequence of past events, can be used to predict future events such as a
diagnosis of a new disorder or complication of a previous or existing disorder.
While most prediction approaches use mostly the structured data in EHRs or a
subset of single-domain predictions and outcomes, we present MedGPT a novel
transformer-based pipeline that uses Named Entity Recognition and Linking tools
(i.e. MedCAT) to structure and organize the free text portion of EHRs and
anticipate a range of future medical events (initially disorders). Since a
large portion of EHR data is in text form, such an approach benefits from a
granular and detailed view of a patient while introducing modest additional
noise. MedGPT effectively deals with the noise and the added granularity, and
achieves a precision of 0.344, 0.552 and 0.640 (vs LSTM 0.329, 0.538 and 0.633)
when predicting the top 1, 3 and 5 candidate future disorders on real world
hospital data from King's College Hospital, London, UK (\textasciitilde600k
patients). We also show that our model captures medical knowledge by testing it
on an experimental medical multiple choice question answering task, and by
examining the attentional focus of the model using gradient-based saliency
methods.

negative:
Clinical Named Entity Recognition (CNER) aims to automatically identity
clinical terminologies in Electronic Health Records (EHRs), which is a
fundamental and crucial step for clinical research. To train a high-performance
model for CNER, it usually requires a large number of EHRs with high-quality
labels. However, labeling EHRs, especially Chinese EHRs, is time-consuming and
expensive. One effective solution to this is active learning, where a model
asks labelers to annotate data which the model is uncertain of. Conventional
active learning assumes a single labeler that always replies noiseless answers
to queried labels. However, in real settings, multiple labelers provide diverse
quality of annotation with varied costs and labelers with low overall
annotation quality can still assign correct labels for some specific instances.
In this paper, we propose a Cost-Quality Adaptive Active Learning (CQAAL)
approach for CNER in Chinese EHRs, which maintains a balance between the
annotation quality, labeling costs, and the informativeness of selected
instances. Specifically, CQAAL selects cost-effective instance-labeler pairs to
achieve better annotation quality with lower costs in an adaptive manner.
Computational results on the CCKS-2017 Task 2 benchmark dataset demonstrate the
superiority and effectiveness of the proposed CQAAL.

negative:
Machine learning models depend on the quality of input data. As electronic
health records are widely adopted, the amount of data in health care is
growing, along with complaints about the quality of medical notes. We use two
prediction tasks, readmission prediction and in-hospital mortality prediction,
to characterize the value of information in medical notes. We show that as a
whole, medical notes only provide additional predictive power over structured
information in readmission prediction. We further propose a probing framework
to select parts of notes that enable more accurate predictions than using all
notes, despite that the selected information leads to a distribution shift from
the training data ("all notes"). Finally, we demonstrate that models trained on
the selected valuable information achieve even better predictive performance,
with only 6.8% of all the tokens for readmission prediction.

negative:
Predicting diagnoses from Electronic Health Records (EHRs) is an important
medical application of multi-label learning. We propose a convolutional
residual model for multi-label classification from doctor notes in EHR data. A
given patient may have multiple diagnoses, and therefore multi-label learning
is required. We employ a Convolutional Neural Network (CNN) to encode plain
text into a fixed-length sentence embedding vector. Since diagnoses are
typically correlated, a deep residual network is employed on top of the CNN
encoder, to capture label (diagnosis) dependencies and incorporate information
directly from the encoded sentence vector. A real EHR dataset is considered,
and we compare the proposed model with several well-known baselines, to predict
diagnoses based on doctor notes. Experimental results demonstrate the
superiority of the proposed convolutional residual model.

negative:
In recent years, the trend of deploying digital systems in numerous
industries has hiked. The health sector has observed an extensive adoption of
digital systems and services that generate significant medical records.
Electronic health records contain valuable information for prospective and
retrospective analysis that is often not entirely exploited because of the
complicated dense information storage. The crude purpose of condensing health
records is to select the information that holds most characteristics of the
original documents based on a reported disease. These summaries may boost
diagnosis and save a doctor's time during a saturated workload situation like
the COVID-19 pandemic. In this paper, we are applying a multi-head
attention-based mechanism to perform extractive summarization of meaningful
phrases on clinical notes. Our method finds major sentences for a summary by
correlating tokens, segments, and positional embeddings of sentences in a
clinical note. The model outputs attention scores that are statistically
transformed to extract critical phrases for visualization on the heat-mapping
tool and for human use.

negative:
Document-level Relation Extraction (RE) requires extracting relations
expressed within and across sentences. Recent works show that graph-based
methods, usually constructing a document-level graph that captures
document-aware interactions, can obtain useful entity representations thus
helping tackle document-level RE. These methods either focus more on the entire
graph, or pay more attention to a part of the graph, e.g., paths between the
target entity pair. However, we find that document-level RE may benefit from
focusing on both of them simultaneously. Therefore, to obtain more
comprehensive entity representations, we propose the Coarse-to-Fine Entity
Representation model (CFER) that adopts a coarse-to-fine strategy involving two
phases. First, CFER uses graph neural networks to integrate global information
in the entire graph at a coarse level. Next, CFER utilizes the global
information as a guidance to selectively aggregate path information between the
target entity pair at a fine level. In classification, we combine the entity
representations from both two levels into more comprehensive representations
for relation extraction. Experimental results on two document-level RE
datasets, DocRED and CDR, show that CFER outperforms existing models and is
robust to the uneven label distribution.

negative:
The modelling of Electronic Health Records (EHRs) has the potential to drive
more efficient allocation of healthcare resources, enabling early intervention
strategies and advancing personalised healthcare. However, EHRs are challenging
to model due to their realisation as noisy, multi-modal data occurring at
irregular time intervals. To address their temporal nature, we treat EHRs as
samples generated by a Temporal Point Process (TPP), enabling us to model what
happened in an event with when it happened in a principled way. We gather and
propose neural network parameterisations of TPPs, collectively referred to as
Neural TPPs. We perform evaluations on synthetic EHRs as well as on a set of
established benchmarks. We show that TPPs significantly outperform their
non-TPP counterparts on EHRs. We also show that an assumption of many Neural
TPPs, that the class distribution is conditionally independent of time, reduces
performance on EHRs. Finally, our proposed attention-based Neural TPP performs
favourably compared to existing models, whilst aligning with real world
interpretability requirements, an important step towards a component of
clinical decision support systems.

negative:
Electronic health records (EHRs) have contributed to the computerization of
patient records and can thus be used not only for efficient and systematic
medical services, but also for research on biomedical data science. However,
there are many missing values in EHRs when provided in matrix form, which is an
important issue in many biomedical EHR applications. In this paper, we propose
a two-stage framework that includes missing data imputation and disease
prediction to address the missing data problem in EHRs. We compared the disease
prediction performance of generative adversarial networks (GANs) and
conventional learning algorithms in combination with missing data prediction
methods. As a result, we obtained a level of accuracy of 0.9777, sensitivity of
0.9521, specificity of 0.9925, area under the receiver operating characteristic
curve (AUC-ROC) of 0.9889, and F-score of 0.9688 with a stacked autoencoder as
the missing data prediction method and an auxiliary classifier GAN (AC-GAN) as
the disease prediction method. The comparison results show that a combination
of a stacked autoencoder and an AC-GAN significantly outperforms other existing
approaches. Our results suggest that the proposed framework is more robust for
disease prediction from EHRs with missing data.

negative:
Databases of electronic health records (EHRs) are increasingly used to inform
clinical decisions. Machine learning methods can find patterns in EHRs that are
predictive of future adverse outcomes. However, statistical models may be built
upon patterns of health-seeking behavior that vary across patient
subpopulations, leading to poor predictive performance when training on one
patient population and predicting on another. This note proposes two tests to
better measure and understand model generalization. We use these tests to
compare models derived from two data sources: (i) historical medical records,
and (ii) electrocardiogram (EKG) waveforms. In a predictive task, we show that
EKG-based models can be more stable than EHR-based models across different
patient populations.

negative:
Document AI, or Document Intelligence, is a relatively new research topic
that refers to the techniques for automatically reading, understanding, and
analyzing business documents. It is an important research direction for natural
language processing and computer vision. In recent years, the popularity of
deep learning technology has greatly advanced the development of Document AI,
such as document layout analysis, visual information extraction, document
visual question answering, document image classification, etc. This paper
briefly reviews some of the representative models, tasks, and benchmark
datasets. Furthermore, we also introduce early-stage heuristic rule-based
document analysis, statistical machine learning algorithms, and deep learning
approaches especially pre-training methods. Finally, we look into future
directions for Document AI research.

negative:
We present a contrasting learning approach with data augmentation techniques
to learn document representations in an unsupervised manner. Inspired by recent
contrastive self-supervised learning algorithms used for image and NLP
pretraining, we hypothesize that high-quality document embedding should be
invariant to diverse paraphrases that preserve the semantics of the original
document. With different backbones and contrastive learning frameworks, our
study reveals the enormous benefits of contrastive augmentation for document
representation learning with two additional insights: 1) including data
augmentation in a contrastive way can substantially improve the embedding
quality in unsupervised document representation learning, and 2) in general,
stochastic augmentations generated by simple word-level manipulation work much
better than sentence-level and document-level ones. We plug our method into a
classifier and compare it with a broad range of baseline methods on six
benchmark datasets. Our method can decrease the classification error rate by up
to 6.4% over the SOTA approaches on the document classification task, matching
or even surpassing fully-supervised methods.

negative:
Patient notes contain a wealth of information of potentially great interest
to medical investigators. However, to protect patients' privacy, Protected
Health Information (PHI) must be removed from the patient notes before they can
be legally released, a process known as patient note de-identification. The
main objective for a de-identification system is to have the highest possible
recall. Recently, the first neural-network-based de-identification system has
been proposed, yielding state-of-the-art results. Unlike other systems, it does
not rely on human-engineered features, which allows it to be quickly deployed,
but does not leverage knowledge from human experts or from electronic health
records (EHRs). In this work, we explore a method to incorporate
human-engineered features as well as features derived from EHRs to a
neural-network-based de-identification system. Our results show that the
addition of features, especially the EHR-derived features, further improves the
state-of-the-art in patient note de-identification, including for some of the
most sensitive PHI types such as patient names. Since in a real-life setting
patient notes typically come with EHRs, we recommend developers of
de-identification systems to leverage the information EHRs contain.

negative:
The adoption of electronic health records (EHR) has become universal during
the past decade, which has afforded in-depth data-based research. By learning
from the large amount of healthcare data, various data-driven models have been
built to predict future events for different medical tasks, such as auto
diagnosis and heart-attack prediction. Although EHR is abundant, the population
that satisfies specific criteria for learning population-specific tasks is
scarce, making it challenging to train data-hungry deep learning models. This
study presents the Claim Pre-Training (Claim-PT) framework, a generic
pre-training model that first trains on the entire pediatric claims dataset,
followed by a discriminative fine-tuning on each population-specific task. The
semantic meaning of medical events can be captured in the pre-training stage,
and the effective knowledge transfer is completed through the task-aware
fine-tuning stage. The fine-tuning process requires minimal parameter
modification without changing the model architecture, which mitigates the data
scarcity issue and helps train the deep learning model adequately on small
patient cohorts. We conducted experiments on a real-world claims dataset with
more than one million patient records. Experimental results on two downstream
tasks demonstrated the effectiveness of our method: our general task-agnostic
pre-training framework outperformed tailored task-specific models, achieving
more than 10\% higher in model performance as compared to baselines. In
addition, our framework showed a great generalizability potential to transfer
learned knowledge from one institution to another, paving the way for future
healthcare model pre-training across institutions.

negative:
In this paper we study the problem of predicting clinical diagnoses from
textual Electronic Health Records (EHR) data. We show the importance of this
problem in medical community and present comprehensive historical review of the
problem and proposed methods. As the main scientific contributions we present a
modification of Bidirectional Encoder Representations from Transformers (BERT)
model for sequence classification that implements a novel way of
Fully-Connected (FC) layer composition and a BERT model pretrained only on
domain data. To empirically validate our model, we use a large-scale Russian
EHR dataset consisting of about 4 million unique patient visits. This is the
largest such study for the Russian language and one of the largest globally. We
performed a number of comparative experiments with other text representation
models on the task of multiclass classification for 265 disease subset of
ICD-10. The experiments demonstrate improved performance of our models compared
to other baselines, including a fine-tuned Russian BERT (RuBERT) variant. We
also show comparable performance of our model with a panel of experienced
medical experts. This allows us to hope that implementation of this system will
reduce misdiagnosis.

negative:
Large, pre-trained transformer models like BERT have achieved
state-of-the-art results on document understanding tasks, but most
implementations can only consider 512 tokens at a time. For many real-world
applications, documents can be much longer, and the segmentation strategies
typically used on longer documents miss out on document structure and
contextual information, hurting their results on downstream tasks. In our work
on legal agreements, we find that visual cues such as layout, style, and
placement of text in a document are strong features that are crucial to
achieving an acceptable level of accuracy on long documents. We measure the
impact of incorporating such visual cues, obtained via computer vision methods,
on the accuracy of document understanding tasks including document
segmentation, entity extraction, and attribute classification. Our method of
segmenting documents based on structural metadata out-performs existing methods
on four long-document understanding tasks as measured on the Contract
Understanding Atticus Dataset.

negative:
Events and entities are closely related; entities are often actors or
participants in events and events without entities are uncommon. The
interpretation of events and entities is highly contextually dependent.
Existing work in information extraction typically models events separately from
entities, and performs inference at the sentence level, ignoring the rest of
the document. In this paper, we propose a novel approach that models the
dependencies among variables of events, entities, and their relations, and
performs joint inference of these variables across a document. The goal is to
enable access to document-level contextual information and facilitate
context-aware predictions. We demonstrate that our approach substantially
outperforms the state-of-the-art methods for event extraction as well as a
strong baseline for entity extraction.

negative:
Document-level relation extraction (DocRE) aims to extract semantic relations
among entity pairs in a document. Typical DocRE methods blindly take the full
document as input, while a subset of the sentences in the document, noted as
the evidence, are often sufficient for humans to predict the relation of an
entity pair. In this paper, we propose an evidence-enhanced framework, Eider,
that empowers DocRE by efficiently extracting evidence and effectively fusing
the extracted evidence in inference. We first jointly train an RE model with a
lightweight evidence extraction model, which is efficient in both memory and
runtime. Empirically, even training the evidence model on silver labels
constructed by our heuristic rules can lead to better RE performance. We
further design a simple yet effective inference process that makes RE
predictions on both extracted evidence and the full document, then fuses the
predictions through a blending layer. This allows Eider to focus on important
sentences while still having access to the complete information in the
document. Extensive experiments show that Eider outperforms state-of-the-art
methods on three benchmark datasets (e.g., by 1.37/1.26 Ign F1/F1 on DocRED).

negative:
Hierarchical neural architectures are often used to capture long-distance
dependencies and have been applied to many document-level tasks such as
summarization, document segmentation, and sentiment analysis. However,
effective usage of such a large context can be difficult to learn, especially
in the case where there is limited labeled data available. Building on the
recent success of language model pretraining methods for learning flat
representations of text, we propose algorithms for pre-training hierarchical
document representations from unlabeled data. Unlike prior work, which has
focused on pre-training contextual token representations or context-independent
{sentence/paragraph} representations, our hierarchical document representations
include fixed-length sentence/paragraph representations which integrate
contextual information from the entire documents. Experiments on document
segmentation, document-level question answering, and extractive document
summarization demonstrate the effectiveness of the proposed pre-training
algorithms.

negative:
Standard neural machine translation (NMT) is on the assumption of
document-level context independent. Most existing document-level NMT methods
are satisfied with a smattering sense of brief document-level information,
while this work focuses on exploiting detailed document-level context in terms
of multiple forms of document embeddings, which is capable of sufficiently
modeling deeper and richer document-level context. The proposed document-aware
NMT is implemented to enhance the Transformer baseline by introducing both
global and local document-level clues on the source end. Experiments show that
the proposed method significantly improves the translation performance over
strong baselines and other related studies.

negative:
Recent years have seen numerous NLP datasets introduced to evaluate the
performance of fine-tuned models on natural language understanding tasks.
Recent results from large pretrained models, though, show that many of these
datasets are largely saturated and unlikely to be able to detect further
progress. What kind of datasets are still effective at discriminating among
strong models, and what kind of datasets should we expect to be able to detect
future improvements? To measure this uniformly across datasets, we draw on Item
Response Theory and evaluate 29 datasets using predictions from 18 pretrained
Transformer models on individual test examples. We find that Quoref, HellaSwag,
and MC-TACO are best suited for distinguishing among state-of-the-art models,
while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong
models. We also observe span selection task format, which is used for QA
datasets like QAMR or SQuAD2.0, is effective in differentiating between strong
and weak models.

negative:
Electronic Health Records (EHRs) have been heavily used to predict various
downstream clinical tasks such as readmission or mortality. One of the
modalities in EHRs, clinical notes, has not been fully explored for these tasks
due to its unstructured and inexplicable nature. Although recent advances in
deep learning (DL) enables models to extract interpretable features from
unstructured data, they often require a large amount of training data. However,
many tasks in medical domains inherently consist of small sample data with
lengthy documents; for a kidney transplant as an example, data from only a few
thousand of patients are available and each patient's document consists of a
couple of millions of words in major hospitals. Thus, complex DL methods cannot
be applied to these kinds of domains. In this paper, we present a comprehensive
ensemble model using vector space modeling and topic modeling. Our proposed
model is evaluated on the readmission task of kidney transplant patients and
improves 0.0211 in terms of c-statistics from the previous state-of-the-art
approach using structured data, while typical DL methods fail to beat this
approach. The proposed architecture provides the interpretable score for each
feature from both modalities, structured and unstructured data, which is shown
to be meaningful through a physician's evaluation.

negative:
Multiple entities in a document generally exhibit complex inter-sentence
relations, and cannot be well handled by existing relation extraction (RE)
methods that typically focus on extracting intra-sentence relations for single
entity pairs. In order to accelerate the research on document-level RE, we
introduce DocRED, a new dataset constructed from Wikipedia and Wikidata with
three features: (1) DocRED annotates both named entities and relations, and is
the largest human-annotated dataset for document-level RE from plain text; (2)
DocRED requires reading multiple sentences in a document to extract entities
and infer their relations by synthesizing all information of the document; (3)
along with the human-annotated data, we also offer large-scale distantly
supervised data, which enables DocRED to be adopted for both supervised and
weakly supervised scenarios. In order to verify the challenges of
document-level RE, we implement recent state-of-the-art methods for RE and
conduct a thorough evaluation of these methods on DocRED. Empirical results
show that DocRED is challenging for existing RE methods, which indicates that
document-level RE remains an open problem and requires further efforts. Based
on the detailed analysis on the experiments, we discuss multiple promising
directions for future research.

negative:
Electronic health records (EHRs) are longitudinal records of a patient's
interactions with healthcare systems. A patient's EHR data is organized as a
three-level hierarchy from top to bottom: patient journey - all the experiences
of diagnoses and treatments over a period of time; individual visit - a set of
medical codes in a particular visit; and medical code - a specific record in
the form of medical codes. As EHRs begin to amass in millions, the potential
benefits, which these data might hold for medical research and medical outcome
prediction, are staggering - including, for example, predicting future
admissions to hospitals, diagnosing illnesses or determining the efficacy of
medical treatments. Each of these analytics tasks requires a domain knowledge
extraction method to transform the hierarchical patient journey into a vector
representation for further prediction procedure. The representations should
embed a sequence of visits and a set of medical codes with a specific
timestamp, which are crucial to any downstream prediction tasks. Hence,
expressively powerful representations are appealing to boost learning
performance. To this end, we propose a novel self-attention mechanism that
captures the contextual dependency and temporal relationships within a
patient's healthcare journey. An end-to-end bidirectional temporal encoder
network (BiteNet) then learns representations of the patient's journeys, based
solely on the proposed attention mechanism. We have evaluated the effectiveness
of our methods on two supervised prediction and two unsupervised clustering
tasks with a real-world EHR dataset. The empirical results demonstrate the
proposed BiteNet model produces higher-quality representations than
state-of-the-art baseline methods.

negative:
Social and behavioral determinants of health (SBDoH) have important roles in
shaping people's health. In clinical research studies, especially comparative
effectiveness studies, failure to adjust for SBDoH factors will potentially
cause confounding issues and misclassification errors in either statistical
analyses and machine learning-based models. However, there are limited studies
to examine SBDoH factors in clinical outcomes due to the lack of structured
SBDoH information in current electronic health record (EHR) systems, while much
of the SBDoH information is documented in clinical narratives. Natural language
processing (NLP) is thus the key technology to extract such information from
unstructured clinical text. However, there is not a mature clinical NLP system
focusing on SBDoH. In this study, we examined two state-of-the-art
transformer-based NLP models, including BERT and RoBERTa, to extract SBDoH
concepts from clinical narratives, applied the best performing model to extract
SBDoH concepts on a lung cancer screening patient cohort, and examined the
difference of SBDoH information between NLP extracted results and structured
EHRs (SBDoH information captured in standard vocabularies such as the
International Classification of Diseases codes). The experimental results show
that the BERT-based NLP model achieved the best strict/lenient F1-score of
0.8791 and 0.8999, respectively. The comparison between NLP extracted SBDoH
information and structured EHRs in the lung cancer patient cohort of 864
patients with 161,933 various types of clinical notes showed that much more
detailed information about smoking, education, and employment were only
captured in clinical narratives and that it is necessary to use both clinical
narratives and structured EHRs to construct a more complete picture of
patients' SBDoH factors.

negative:
Sequence to sequence (Seq2Seq) learning has recently been used for
abstractive and extractive summarization. In current study, Seq2Seq models have
been used for eBay product description summarization. We propose a novel
Document-Context based Seq2Seq models using RNNs for abstractive and extractive
summarizations. Intuitively, this is similar to humans reading the title,
abstract or any other contextual information before reading the document. This
gives humans a high-level idea of what the document is about. We use this idea
and propose that Seq2Seq models should be started with contextual information
at the first time-step of the input to obtain better summaries. In this manner,
the output summaries are more document centric, than being generic, overcoming
one of the major hurdles of using generative models. We generate
document-context from user-behavior and seller provided information. We train
and evaluate our models on human-extracted-golden-summaries. The
document-contextual Seq2Seq models outperform standard Seq2Seq models.
Moreover, generating human extracted summaries is prohibitively expensive to
scale, we therefore propose a semi-supervised technique for extracting
approximate summaries and using it for training Seq2Seq models at scale.
Semi-supervised models are evaluated against human extracted summaries and are
found to be of similar efficacy. We provide side by side comparison for
abstractive and extractive summarizers (contextual and non-contextual) on same
evaluation dataset. Overall, we provide methodologies to use and evaluate the
proposed techniques for large document summarization. Furthermore, we found
these techniques to be highly effective, which is not the case with existing
techniques.

negative:
Cross-lingual document representations enable language understanding in
multilingual contexts and allow transfer learning from high-resource to
low-resource languages at the document level. Recently large pre-trained
language models such as BERT, XLM and XLM-RoBERTa have achieved great success
when fine-tuned on sentence-level downstream tasks. It is tempting to apply
these cross-lingual models to document representation learning. However, there
are two challenges: (1) these models impose high costs on long document
processing and thus many of them have strict length limit; (2) model
fine-tuning requires extra data and computational resources, which is not
practical in resource-limited settings. In this work, we address these
challenges by proposing unsupervised Language-Agnostic Weighted Document
Representations (LAWDR). We study the geometry of pre-trained sentence
embeddings and leverage it to derive document representations without
fine-tuning. Evaluated on cross-lingual document alignment, LAWDR demonstrates
comparable performance to state-of-the-art models on benchmark datasets.

negative:
Document grounded generation is the task of using the information provided in
a document to improve text generation. This work focuses on two different
document grounded generation tasks: Wikipedia Update Generation task and
Dialogue response generation. Our work introduces two novel adaptations of
large scale pre-trained encoder-decoder models focusing on building context
driven representation of the document and enabling specific attention to the
information in the document. Additionally, we provide a stronger BART baseline
for these tasks. Our proposed techniques outperform existing methods on both
automated (at least 48% increase in BLEU-4 points) and human evaluation for
closeness to reference and relevance to the document. Furthermore, we perform
comprehensive manual inspection of the generated output and categorize errors
to provide insights into future directions in modeling these tasks.

negative:
Most of the existing information extraction frameworks (Wadden et al., 2019;
Veysehet al., 2020) focus on sentence-level tasks and are hardly able to
capture the consolidated information from a given document. In our endeavour to
generate precise document-level information frames from lengthy textual
records, we introduce the task of Information Aggregation or Argument
Aggregation. More specifically, our aim is to filter irrelevant and redundant
argument mentions that were extracted at a sentence level and render a document
level information frame. Majority of the existing works have been observed to
resolve related tasks of document-level event argument extraction (Yang et al.,
2018a; Zheng et al., 2019a) and salient entity identification (Jain et
al.,2020) using supervised techniques. To remove dependency from large amounts
of labelled data, we explore the task of information aggregation using
weakly-supervised techniques. In particular, we present an extractive algorithm
with multiple sieves which adopts active learning strategies to work
efficiently in low-resource settings. For this task, we have annotated our own
test dataset comprising of 131 document information frames and have released
the code and dataset to further research prospects in this new domain. To the
best of our knowledge, we are the first to establish baseline results for this
task in English. Our data and code are publicly available at
https://github.com/DebanjanaKar/ArgFuse.

negative:
A variety of methods existing for generating synthetic electronic health
records (EHRs), but they are not capable of generating unstructured text, like
emergency department (ED) chief complaints, history of present illness or
progress notes. Here, we use the encoder-decoder model, a deep learning
algorithm that features in many contemporary machine translation systems, to
generate synthetic chief complaints from discrete variables in EHRs, like age
group, gender, and discharge diagnosis. After being trained end-to-end on
authentic records, the model can generate realistic chief complaint text that
preserves much of the epidemiological information in the original data. As a
side effect of the model's optimization goal, these synthetic chief complaints
are also free of relatively uncommon abbreviation and misspellings, and they
include none of the personally-identifiable information (PII) that was in the
training data, suggesting it may be used to support the de-identification of
text in EHRs. When combined with algorithms like generative adversarial
networks (GANs), our model could be used to generate fully-synthetic EHRs,
facilitating data sharing between healthcare providers and researchers and
improving our ability to develop machine learning methods tailored to the
information in healthcare data.

negative:
We present an overview of the CLEF-2019 Lab ProtestNews on Extracting
Protests from News in the context of generalizable natural language processing.
The lab consists of document, sentence, and token level information
classification and extraction tasks that were referred as task 1, task 2, and
task 3 respectively in the scope of this lab. The tasks required the
participants to identify protest relevant information from English local news
at one or more aforementioned levels in a cross-context setting, which is
cross-country in the scope of this lab. The training and development data were
collected from India and test data was collected from India and China. The lab
attracted 58 teams to participate in the lab. 12 and 9 of these teams submitted
results and working notes respectively. We have observed neural networks yield
the best results and the performance drops significantly for majority of the
submissions in the cross-country setting, which is China.

negative:
Augmentation of disease diagnosis and decision-making in healthcare with
machine learning algorithms is gaining much impetus in recent years. In
particular, in the current epidemiological situation caused by COVID-19
pandemic, swift and accurate prediction of disease diagnosis with machine
learning algorithms could facilitate identification and care of vulnerable
clusters of population, such as those having multi-morbidity conditions. In
order to build a useful disease diagnosis prediction system, advancement in
both data representation and development of machine learning architectures are
imperative. First, with respect to data collection and representation, we face
severe problems due to multitude of formats and lack of coherency prevalent in
Electronic Health Records (EHRs). This causes hindrance in extraction of
valuable information contained in EHRs. Currently, no universal global data
standard has been established. As a useful solution, we develop and publish a
Python package to transform public health dataset into an easy to access
universal format. This data transformation to an international health data
format facilitates researchers to easily combine EHR datasets with clinical
datasets of diverse formats. Second, machine learning algorithms that predict
multiple disease diagnosis categories simultaneously remain underdeveloped. We
propose two novel model architectures in this regard. First, DeepObserver,
which uses structured numerical data to predict the diagnosis categories and
second, ClinicalBERT_Multi, that incorporates rich information available in
clinical notes via natural language processing methods and also provides
interpretable visualizations to medical practitioners. We show that both models
can predict multiple diagnoses simultaneously with high accuracy.

negative:
Objective: Allowing patients to access their own electronic health record
(EHR) notes through online patient portals has the potential to improve
patient-centered care. However, medical jargon, which abounds in EHR notes, has
been shown to be a barrier for patient EHR comprehension. Existing knowledge
bases that link medical jargon to lay terms or definitions play an important
role in alleviating this problem but have low coverage of medical jargon in
EHRs. We developed a data-driven approach that mines EHRs to identify and rank
medical jargon based on its importance to patients, to support the building of
EHR-centric lay language resources.
  Methods: We developed an innovative adapted distant supervision (ADS) model
based on support vector machines to rank medical jargon from EHRs. For distant
supervision, we utilized the open-access, collaborative consumer health
vocabulary, a large, publicly available resource that links lay terms to
medical jargon. We explored both knowledge-based features from the Unified
Medical Language System and distributed word representations learned from
unlabeled large corpora. We evaluated the ADS model using physician-identified
important medical terms.
  Results: Our ADS model significantly surpassed two state-of-the-art automatic
term recognition methods, TF*IDF and C-Value, yielding 0.810 ROC-AUC versus
0.710 and 0.667, respectively. Our model identified 10K important medical
jargon terms after ranking over 100K candidate terms mined from over 7,500 EHR
narratives.
  Conclusion: Our work is an important step towards enriching lexical resources
that link medical jargon to lay terms/definitions to support patient EHR
comprehension. The identified medical jargon terms and their rankings are
available upon request.

negative:
Deep reading models for question-answering have demonstrated promising
performance over the last couple of years. However current systems tend to
learn how to cleverly extract a span of the source document, based on its
similarity with the question, instead of seeking for the appropriate answer.
Indeed, a reading machine should be able to detect relevant passages in a
document regarding a question, but more importantly, it should be able to
reason over the important pieces of the document in order to produce an answer
when it is required. To motivate this purpose, we present ReviewQA, a
question-answering dataset based on hotel reviews. The questions of this
dataset are linked to a set of relational understanding competencies that we
expect a model to master. Indeed, each question comes with an associated type
that characterizes the required competency. With this framework, it is possible
to benchmark the main families of models and to get an overview of what are the
strengths and the weaknesses of a given model on the set of tasks evaluated in
this dataset. Our corpus contains more than 500.000 questions in natural
language over 100.000 hotel reviews. Our setup is projective, the answer of a
question does not need to be extracted from a document, like in most of the
recent datasets, but selected among a set of candidates that contains all the
possible answers to the questions of the dataset. Finally, we present several
baselines over this dataset.

negative:
Measuring similarity between texts is an important task for several
applications. Available approaches to measure document similarity are
inadequate for document pairs that have non-comparable lengths, such as a long
document and its summary. This is because of the lexical, contextual and the
abstraction gaps between a long document of rich details and its concise
summary of abstract information. In this paper, we present a document matching
approach to bridge this gap, by comparing the texts in a common space of hidden
topics. We evaluate the matching algorithm on two matching tasks and find that
it consistently and widely outperforms strong baselines. We also highlight the
benefits of incorporating domain knowledge to text matching.

negative:
Modern electronic health records (EHRs) provide data to answer clinically
meaningful questions. The growing data in EHRs makes healthcare ripe for the
use of machine learning. However, learning in a clinical setting presents
unique challenges that complicate the use of common machine learning
methodologies. For example, diseases in EHRs are poorly labeled, conditions can
encompass multiple underlying endotypes, and healthy individuals are
underrepresented. This article serves as a primer to illuminate these
challenges and highlights opportunities for members of the machine learning
community to contribute to healthcare.

negative:
The increased adoption of Electronic Health Records(EHRs) has brought changes
to the way the patient care is carried out. The rich heterogeneous and temporal
data space stored in EHRs can be leveraged by machine learning models to
capture the underlying information and make clinically relevant predictions.
This can be exploited to support public health activities such as
pharmacovigilance and specifically mitigate the public health issue of Adverse
Drug Events(ADEs). The aim of this article is, therefore, to investigate the
various ways of handling temporal data for the purpose of detecting ADEs. Based
on a review of the existing literature, 11 articles from the last 10 years were
chosen to be studied. According to the literature retrieved the main methods
were found to fall into 5 different approaches: based on temporal abstraction,
graph-based, learning weights and data tables containing time series of
different length. To that end, EHRs are a valuable source that has led current
research to the automatic detection of ADEs. Yet there still exists a great
deal of challenges that concerns the exploitation of the heterogeneous, data
types with temporal information included in EHRs for predicting ADEs.

negative:
In the Query Focused Multi-Document Summarization (QF-MDS) task, a set of
documents and a query are given where the goal is to generate a summary from
these documents based on the given query. However, one major challenge for this
task is the lack of availability of labeled training datasets. To overcome this
issue, in this paper, we propose a novel weakly supervised learning approach
via utilizing distant supervision. In particular, we use datasets similar to
the target dataset as the training data where we leverage pre-trained sentence
similarity models to generate the weak reference summary of each individual
document in a document set from the multi-document gold reference summaries.
Then, we iteratively train our summarization model on each single-document to
alleviate the computational complexity issue that occurs while training neural
summarization models in multiple documents (i.e., long sequences) at once.
Experimental results in Document Understanding Conferences (DUC) datasets show
that our proposed approach sets a new state-of-the-art result in terms of
various evaluation metrics.

negative:
Text Mining is a field that aims at extracting information from textual data.
One of the challenges of such field of study comes from the pre-processing
stage in which a vector (and structured) representation should be extracted
from unstructured data. The common extraction creates large and sparse vectors
representing the importance of each term to a document. As such, this usually
leads to the curse-of-dimensionality that plagues most machine learning
algorithms. To cope with this issue, in this paper we propose a new supervised
feature extraction and reduction algorithm, named DCDistance, that creates
features based on the distance between a document to a representative of each
class label. As such, the proposed technique can reduce the features set in
more than 99% of the original set. Additionally, this algorithm was also
capable of improving the classification accuracy over a set of benchmark
datasets when compared to traditional and state-of-the-art features selection
algorithms.

negative:
In this paper we present a method for the unsupervised clustering of
high-dimensional binary data, with a special focus on electronic healthcare
records. We present a robust and efficient heuristic to face this problem using
tensor decomposition. We present the reasons why this approach is preferable
for tasks such as clustering patient records, to more commonly used
distance-based methods. We run the algorithm on two datasets of healthcare
records, obtaining clinically meaningful results.

negative:
As the application of deep learning has expanded to real-world problems with
insufficient volume of training data, transfer learning recently has gained
much attention as means of improving the performance in such small-data regime.
However, when existing methods are applied between heterogeneous architectures
and tasks, it becomes more important to manage their detailed configurations
and often requires exhaustive tuning on them for the desired performance. To
address the issue, we propose a novel transfer learning approach based on
meta-learning that can automatically learn what knowledge to transfer from the
source network to where in the target network. Given source and target
networks, we propose an efficient training scheme to learn meta-networks that
decide (a) which pairs of layers between the source and target networks should
be matched for knowledge transfer and (b) which features and how much knowledge
from each feature should be transferred. We validate our meta-transfer approach
against recent transfer learning methods on various datasets and network
architectures, on which our automated scheme significantly outperforms the
prior baselines that find "what and where to transfer" in a hand-crafted
manner.

negative:
This paper introduces a conceptually simple, scalable, and highly effective
BERT-based entity linking model, along with an extensive evaluation of its
accuracy-speed trade-off. We present a two-stage zero-shot linking algorithm,
where each entity is defined only by a short textual description. The first
stage does retrieval in a dense space defined by a bi-encoder that
independently embeds the mention context and the entity descriptions. Each
candidate is then re-ranked with a cross-encoder, that concatenates the mention
and entity text. Experiments demonstrate that this approach is state of the art
on recent zero-shot benchmarks (6 point absolute gains) and also on more
established non-zero-shot evaluations (e.g. TACKBP-2010), despite its relative
simplicity (e.g. no explicit entity embeddings or manually engineered mention
tables). We also show that bi-encoder linking is very fast with nearest
neighbour search (e.g. linking with 5.9 million candidates in 2 milliseconds),
and that much of the accuracy gain from the more expensive cross-encoder can be
transferred to the bi-encoder via knowledge distillation. Our code and models
are available at https://github.com/facebookresearch/BLINK.

negative:
We present Attract-Repel, an algorithm for improving the semantic quality of
word vectors by injecting constraints extracted from lexical resources.
Attract-Repel facilitates the use of constraints from mono- and cross-lingual
resources, yielding semantically specialised cross-lingual vector spaces. Our
evaluation shows that the method can make use of existing cross-lingual
lexicons to construct high-quality vector spaces for a plethora of different
languages, facilitating semantic transfer from high- to lower-resource ones.
The effectiveness of our approach is demonstrated with state-of-the-art results
on semantic similarity datasets in six languages. We next show that
Attract-Repel-specialised vectors boost performance in the downstream task of
dialogue state tracking (DST) across multiple languages. Finally, we show that
cross-lingual vector spaces produced by our algorithm facilitate the training
of multilingual DST models, which brings further performance improvements.

negative:
Handwriting is a natural and versatile method for human-computer interaction,
especially on small mobile devices such as smart phones. However, as
handwriting varies significantly from person to person, it is difficult to
design handwriting recognizers that perform well for all users. A natural
solution is to use machine learning to adapt the recognizer to the user. One
complicating factor is that, as the computer adapts to the user, the user also
adapts to the computer and probably changes their handwriting. This paper
investigates the dynamics of co-adaptation, a process in which both the
computer and the user are adapting their behaviors in order to improve the
speed and accuracy of the communication through handwriting. We devised an
information-theoretic framework for quantifying the efficiency of a handwriting
system where the system includes both the user and the computer. Using this
framework, we analyzed data collected from an adaptive handwriting recognition
system and characterized the impact of machine adaptation and of human
adaptation. We found that both machine adaptation and human adaptation have
significant impact on the input rate and must be considered together in order
to improve the efficiency of the system as a whole.

negative:
The success of pretrained cross-lingual language models relies on two
essential abilities, i.e., generalization ability for learning downstream tasks
in a source language, and cross-lingual transferability for transferring the
task knowledge to other languages. However, current methods jointly learn the
two abilities in a single-phase cross-lingual pretraining process, resulting in
a trade-off between generalization and cross-lingual transfer. In this paper,
we propose cross-lingual language model meta-pretraining, which learns the two
abilities in different training phases. Our method introduces an additional
meta-pretraining phase before cross-lingual pretraining, where the model learns
generalization ability on a large-scale monolingual corpus. Then, the model
focuses on learning cross-lingual transfer on a multilingual corpus.
Experimental results show that our method improves both generalization and
cross-lingual transfer, and produces better-aligned representations across
different languages.

negative:
We explore using multilingual document embeddings for nearest neighbor mining
of parallel data. Three document-level representations are investigated: (i)
document embeddings generated by simply averaging multilingual sentence
embeddings; (ii) a neural bag-of-words (BoW) document encoding model; (iii) a
hierarchical multilingual document encoder (HiDE) that builds on our
sentence-level model. The results show document embeddings derived from
sentence-level averaging are surprisingly effective for clean datasets, but
suggest models trained hierarchically at the document-level are more effective
on noisy data. Analysis experiments demonstrate our hierarchical models are
very robust to variations in the underlying sentence embedding quality. Using
document embeddings trained with HiDE achieves state-of-the-art performance on
United Nations (UN) parallel document mining, 94.9% P@1 for en-fr and 97.3% P@1
for en-es.

negative:
We present a novel model for the problem of ranking a collection of documents
according to their semantic similarity to a source (query) document. While the
problem of document-to-document similarity ranking has been studied, most
modern methods are limited to relatively short documents or rely on the
existence of "ground-truth" similarity labels. Yet, in most common real-world
cases, similarity ranking is an unsupervised problem as similarity labels are
unavailable. Moreover, an ideal model should not be restricted by documents'
length. Hence, we introduce SDR, a self-supervised method for document
similarity that can be applied to documents of arbitrary length. Importantly,
SDR can be effectively applied to extremely long documents, exceeding the 4,096
maximal token limits of Longformer. Extensive evaluations on large document
datasets show that SDR significantly outperforms its alternatives across all
metrics. To accelerate future research on unlabeled long document similarity
ranking, and as an additional contribution to the community, we herein publish
two human-annotated test sets of long documents similarity evaluation. The SDR
code and datasets are publicly available.

negative:
In this paper, we introduce Iterative Text Summarization (ITS), an
iteration-based model for supervised extractive text summarization, inspired by
the observation that it is often necessary for a human to read an article
multiple times in order to fully understand and summarize its contents. Current
summarization approaches read through a document only once to generate a
document representation, resulting in a sub-optimal representation. To address
this issue we introduce a model which iteratively polishes the document
representation on many passes through the document. As part of our model, we
also introduce a selective reading mechanism that decides more accurately the
extent to which each sentence in the model should be updated. Experimental
results on the CNN/DailyMail and DUC2002 datasets demonstrate that our model
significantly outperforms state-of-the-art extractive systems when evaluated by
machines and by humans.

negative:
Multi-document question generation focuses on generating a question that
covers the common aspect of multiple documents. Such a model is useful in
generating clarifying options. However, a naive model trained only using the
targeted ("positive") document set may generate too generic questions that
cover a larger scope than delineated by the document set. To address this
challenge, we introduce the contrastive learning strategy where given
"positive" and "negative" sets of documents, we generate a question that is
closely related to the "positive" set but is far away from the "negative" set.
This setting allows generated questions to be more specific and related to the
target document set. To generate such specific questions, we propose
Multi-Source Coordinated Question Generator (MSCQG), a novel framework that
includes a supervised learning (SL) stage and a reinforcement learning (RL)
stage. In the SL stage, a single-document question generator is trained. In the
RL stage, a coordinator model is trained to find optimal attention weights to
align multiple single-document generators, by optimizing a reward designed to
promote specificity of generated questions. We also develop an effective
auxiliary objective, named Set-induced Contrastive Regularization (SCR) that
improves the coordinator's contrastive learning during the RL stage. We show
that our model significantly outperforms several strong baselines, as measured
by automatic metrics and human evaluation. The source repository is publicly
available at \url{www.github.com/woonsangcho/contrast_qgen}.

negative:
Document-level relation extraction requires integrating information within
and across multiple sentences of a document and capturing complex interactions
between inter-sentence entities. However, effective aggregation of relevant
information in the document remains a challenging research question. Existing
approaches construct static document-level graphs based on syntactic trees,
co-references or heuristics from the unstructured text to model the
dependencies. Unlike previous methods that may not be able to capture rich
non-local interactions for inference, we propose a novel model that empowers
the relational reasoning across sentences by automatically inducing the latent
document-level graph. We further develop a refinement strategy, which enables
the model to incrementally aggregate relevant information for multi-hop
reasoning. Specifically, our model achieves an F1 score of 59.05 on a
large-scale document-level dataset (DocRED), significantly improving over the
previous results, and also yields new state-of-the-art results on the CDR and
GDA dataset. Furthermore, extensive analyses show that the model is able to
discover more accurate inter-sentence relations.

negative:
State-of-the-art systems in deep question answering proceed as follows: (1)
an initial document retrieval selects relevant documents, which (2) are then
processed by a neural network in order to extract the final answer. Yet the
exact interplay between both components is poorly understood, especially
concerning the number of candidate documents that should be retrieved. We show
that choosing a static number of documents -- as used in prior research --
suffers from a noise-information trade-off and yields suboptimal results. As a
remedy, we propose an adaptive document retrieval model. This learns the
optimal candidate number for document retrieval, conditional on the size of the
corpus and the query. We report extensive experimental results showing that our
adaptive approach outperforms state-of-the-art methods on multiple benchmark
datasets, as well as in the context of corpora with variable sizes.

negative:
Unstructured documents serving as external knowledge of the dialogues help to
generate more informative responses. Previous research focused on knowledge
selection (KS) in the document with dialogue. However, dialogue history that is
not related to the current dialogue may introduce noise in the KS processing.
In this paper, we propose a Compare Aggregate Transformer (CAT) to jointly
denoise the dialogue context and aggregate the document information for
response generation. We designed two different comparison mechanisms to reduce
noise (before and during decoding). In addition, we propose two metrics for
evaluating document utilization efficiency based on word overlap. Experimental
results on the CMUDoG dataset show that the proposed CAT model outperforms the
state-of-the-art approach and strong baselines.

negative:
Probabilistic topic models like Latent Dirichlet Allocation (LDA) have been
previously extended to the bilingual setting. A fundamental modeling assumption
in several of these extensions is that the input corpora are in the form of
document pairs whose constituent documents share a single topic distribution.
However, this assumption is strong for comparable corpora that consist of
documents thematically similar to an extent only, which are, in turn, the most
commonly available or easy to obtain. In this paper we relax this assumption by
proposing for the paired documents to have separate, yet bound topic
distributions. % a binding mechanism between the distributions of the paired
documents. We suggest that the strength of the bound should depend on each
pair's semantic similarity. To estimate the similarity of documents that are
written in different languages we use cross-lingual word embeddings that are
learned with shallow neural networks. We evaluate the proposed binding
mechanism by extending two topic models: a bilingual adaptation of LDA that
assumes bag-of-words inputs and a model that incorporates part of the text
structure in the form of boundaries of semantically coherent segments. To
assess the performance of the novel topic models we conduct intrinsic and
extrinsic experiments on five bilingual, comparable corpora of English
documents with French, German, Italian, Spanish and Portuguese documents. The
results demonstrate the efficiency of our approach in terms of both topic
coherence measured by the normalized point-wise mutual information, and
generalization performance measured by perplexity and in terms of Mean
Reciprocal Rank in a cross-lingual document retrieval task for each of the
language pairs.

negative:
Text alignment finds application in tasks such as citation recommendation and
plagiarism detection. Existing alignment methods operate at a single,
predefined level and cannot learn to align texts at, for example, sentence and
document levels. We propose a new learning approach that equips previously
established hierarchical attention encoders for representing documents with a
cross-document attention component, enabling structural comparisons across
different levels (document-to-document and sentence-to-document). Our component
is weakly supervised from document pairs and can align at multiple levels. Our
evaluation on predicting document-to-document relationships and
sentence-to-document relationships on the tasks of citation recommendation and
plagiarism detection shows that our approach outperforms previously established
hierarchical, attention encoders based on recurrent and transformer
contextualization that are unaware of structural correspondence between
documents.

negative:
We investigate the pertinence of methods from algebraic topology for text
data analysis. These methods enable the development of
mathematically-principled isometric-invariant mappings from a set of vectors to
a document embedding, which is stable with respect to the geometry of the
document in the selected metric space. In this work, we evaluate the utility of
these topology-based document representations in traditional NLP tasks,
specifically document clustering and sentiment classification. We find that the
embeddings do not benefit text analysis. In fact, performance is worse than
simple techniques like $\textit{tf-idf}$, indicating that the geometry of the
document does not provide enough variability for classification on the basis of
topic or sentiment in the chosen datasets.

negative:
In information retrieval (IR) and related tasks, term weighting approaches
typically consider the frequency of the term in the document and in the
collection in order to compute a score reflecting the importance of the term
for the document. In tasks characterized by the presence of training data (such
as text classification) it seems logical that the term weighting function
should take into account the distribution (as estimated from training data) of
the term across the classes of interest. Although `supervised term weighting'
approaches that use this intuition have been described before, they have failed
to show consistent improvements. In this article we analyse the possible
reasons for this failure, and call consolidated assumptions into question.
Following this criticism we propose a novel supervised term weighting approach
that, instead of relying on any predefined formula, learns a term weighting
function optimised on the training set of interest; we dub this approach
\emph{Learning to Weight} (LTW). The experiments that we run on several
well-known benchmarks, and using different learning methods, show that our
method outperforms previous term weighting approaches in text classification.

negative:
Lifelong learning has recently attracted attention in building machine
learning systems that continually accumulate and transfer knowledge to help
future learning. Unsupervised topic modeling has been popularly used to
discover topics from document collections. However, the application of topic
modeling is challenging due to data sparsity, e.g., in a small collection of
(short) documents and thus, generate incoherent topics and sub-optimal document
representations. To address the problem, we propose a lifelong learning
framework for neural topic modeling that can continuously process streams of
document collections, accumulate topics and guide future topic modeling tasks
by knowledge transfer from several sources to better deal with the sparse data.
In the lifelong process, we particularly investigate jointly: (1) sharing
generative homologies (latent topics) over lifetime to transfer prior
knowledge, and (2) minimizing catastrophic forgetting to retain the past
learning via novel selective data augmentation, co-training and topic
regularization approaches. Given a stream of document collections, we apply the
proposed Lifelong Neural Topic Modeling (LNTM) framework in modeling three
sparse document collections as future tasks and demonstrate improved
performance quantified by perplexity, topic coherence and information retrieval
task.

negative:
A significant roadblock in multilingual neural language modeling is the lack
of labeled non-English data. One potential method for overcoming this issue is
learning cross-lingual text representations that can be used to transfer the
performance from training on English tasks to non-English tasks, despite little
to no task-specific non-English data. In this paper, we explore a natural setup
for learning cross-lingual sentence representations: the dual-encoder. We
provide a comprehensive evaluation of our cross-lingual representations on a
number of monolingual, cross-lingual, and zero-shot/few-shot learning tasks,
and also give an analysis of different learned cross-lingual embedding spaces.

negative:
Document similarity is the problem of estimating the degree to which a given
pair of documents has similar semantic content. An accurate document similarity
measure can improve several enterprise relevant tasks such as document
clustering, text mining, and question-answering. In this paper, we show that a
document's thematic flow, which is often disregarded by bag-of-word techniques,
is pivotal in estimating their similarity. To this end, we propose a novel
semantic document similarity framework, called SimDoc. We model documents as
topic-sequences, where topics represent latent generative clusters of related
words. Then, we use a sequence alignment algorithm to estimate their semantic
similarity. We further conceptualize a novel mechanism to compute topic-topic
similarity to fine tune our system. In our experiments, we show that SimDoc
outperforms many contemporary bag-of-words techniques in accurately computing
document similarity, and on practical applications such as document clustering.

negative:
The organization and evolution of science has recently become itself an
object of scientific quantitative investigation, thanks to the wealth of
information that can be extracted from scientific documents, such as citations
between papers and co-authorship between researchers. However, only few studies
have focused on the concepts that characterize full documents and that can be
extracted and analyzed, revealing the deeper organization of scientific
knowledge. Unfortunately, several concepts can be so common across documents
that they hinder the emergence of the underlying topical structure of the
document corpus, because they give rise to a large amount of spurious and
trivial relations among documents. To identify and remove common concepts, we
introduce a method to gauge their relevance according to an objective
information-theoretic measure related to the statistics of their occurrence
across the document corpus. After progressively removing concepts that,
according to this metric, can be considered as generic, we find that the topic
organization displays a correspondingly more refined structure.

negative:
Recent developments in natural language representations have been accompanied
by large and expensive models that leverage vast amounts of general-domain text
through self-supervised pre-training. Due to the cost of applying such models
to down-stream tasks, several model compression techniques on pre-trained
language representations have been proposed (Sun et al., 2019; Sanh, 2019).
However, surprisingly, the simple baseline of just pre-training and fine-tuning
compact models has been overlooked. In this paper, we first show that
pre-training remains important in the context of smaller architectures, and
fine-tuning pre-trained compact models can be competitive to more elaborate
methods proposed in concurrent work. Starting with pre-trained compact models,
we then explore transferring task knowledge from large fine-tuned models
through standard knowledge distillation. The resulting simple, yet effective
and general algorithm, Pre-trained Distillation, brings further improvements.
Through extensive experiments, we more generally explore the interaction
between pre-training and distillation under two variables that have been
under-studied: model size and properties of unlabeled task data. One surprising
observation is that they have a compound effect even when sequentially applied
on the same data. To accelerate future research, we will make our 24
pre-trained miniature BERT models publicly available.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.

negative:
Research is a tertiary priority in the EHR, where the priorities are patient
care and billing. Because of this, the data is not standardized or formatted in
a manner easily adapted to machine learning approaches. Data may be missing for
a large variety of reasons ranging from individual input styles to differences
in clinical decision making, for example, which lab tests to issue. Few
patients are annotated at a research quality, limiting sample size and
presenting a moving gold standard. Patient progression over time is key to
understanding many diseases but many machine learning algorithms require a
snapshot, at a single time point, to create a usable vector form. Furthermore,
algorithms that produce black box results do not provide the interpretability
required for clinical adoption. This chapter discusses these challenges and
others in applying machine learning techniques to the structured EHR (i.e.
Patient Demographics, Family History, Medication Information, Vital Signs,
Laboratory Tests, Genetic Testing). It does not cover feature extraction from
additional sources such as imaging data or free text patient notes but the
approaches discussed can include features extracted from these sources.

negative:
The increasing amount of online content motivated the development of
multi-document summarization methods. In this work, we explore straightforward
approaches to extend single-document summarization methods to multi-document
summarization. The proposed methods are based on the hierarchical combination
of single-document summaries, and achieves state of the art results.

negative:
Standard neural machine translation (NMT) is on the assumption that the
document-level context is independent. Most existing document-level NMT
approaches are satisfied with a smattering sense of global document-level
information, while this work focuses on exploiting detailed document-level
context in terms of a memory network. The capacity of the memory network that
detecting the most relevant part of the current sentence from memory renders a
natural solution to model the rich document-level context. In this work, the
proposed document-aware memory network is implemented to enhance the
Transformer NMT baseline. Experiments on several tasks show that the proposed
method significantly improves the NMT performance over strong Transformer
baselines and other related studies.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
Under special circumstances, summaries should conform to a particular style
with patterns, such as court judgments and abstracts in academic papers. To
this end, the prototype document-summary pairs can be utilized to generate
better summaries. There are two main challenges in this task: (1) the model
needs to incorporate learned patterns from the prototype, but (2) should avoid
copying contents other than the patternized words---such as irrelevant
facts---into the generated summaries. To tackle these challenges, we design a
model named Prototype Editing based Summary Generator (PESG). PESG first learns
summary patterns and prototype facts by analyzing the correlation between a
prototype document and its summary. Prototype facts are then utilized to help
extract facts from the input document. Next, an editing generator generates new
summary based on the summary pattern or extracted facts. Finally, to address
the second challenge, a fact checker is used to estimate mutual information
between the input document and generated summary, providing an additional
signal for the generator. Extensive experiments conducted on a large-scale
real-world text summarization dataset show that PESG achieves the
state-of-the-art performance in terms of both automatic metrics and human
evaluations.

negative:
Electronic Health Records (EHRs) in hospital information systems contain
patients' diagnosis and treatments, so EHRs are essential to clinical data
mining. Of all the tasks in the mining process, Chinese Word Segmentation (CWS)
is a fundamental and important one, and most state-of-the-art methods greatly
rely on large-scale of manually-annotated data. Since annotation is
time-consuming and expensive, efforts have been devoted to techniques, such as
active learning, to locate the most informative samples for modeling. In this
paper, we follow the trend and present an active learning method for CWS in
EHRs. Specically, a new sampling strategy combining Normalized Entropy with
Loss Prediction (NE-LP) is proposed to select the most representative data.
Meanwhile, to minimize the computational cost of learning, we propose a joint
model including a word segmenter and a loss prediction model. Furthermore, to
capture interactions between adjacent characters, bigram features are also
applied in the joint model. To illustrate the effectiveness of NE-LP, we
conducted experiments on EHRs collected from the Shuguang Hospital Affiliated
to Shanghai University of Traditional Chinese Medicine. The results demonstrate
that NE-LP consistently outperforms conventional uncertainty-based sampling
strategies for active learning in CWS.

negative:
The selection of a suitable document representation approach plays a crucial
role in the performance of a document clustering task. Being able to pick out
representative words within a document can lead to substantial improvements in
document clustering. In the case of web documents, the HTML markup that defines
the layout of the content provides additional structural information that can
be further exploited to identify representative words. In this paper we
introduce a fuzzy term weighing approach that makes the most of the HTML
structure for document clustering. We set forth and build on the hypothesis
that a good representation can take advantage of how humans skim through
documents to extract the most representative words. The authors of web pages
make use of HTML tags to convey the most important message of a web page
through page elements that attract the readers' attention, such as page titles
or emphasized elements. We define a set of criteria to exploit the information
provided by these page elements, and introduce a fuzzy combination of these
criteria that we evaluate within the context of a web page clustering task. Our
proposed approach, called Abstract Fuzzy Combination of Criteria (AFCC), can
adapt to datasets whose features are distributed differently, achieving good
results compared to other similar fuzzy logic based approaches and TF-IDF
across different datasets.

negative:
We introduce a novel type of text representation that preserves the 2D layout
of a document. This is achieved by encoding each document page as a
two-dimensional grid of characters. Based on this representation, we present a
generic document understanding pipeline for structured documents. This pipeline
makes use of a fully convolutional encoder-decoder network that predicts a
segmentation mask and bounding boxes. We demonstrate its capabilities on an
information extraction task from invoices and show that it significantly
outperforms approaches based on sequential text or document images.

negative:
Keyphrase extraction from documents is useful to a variety of applications
such as information retrieval and document summarization. This paper presents
an end-to-end method called DivGraphPointer for extracting a set of diversified
keyphrases from a document. DivGraphPointer combines the advantages of
traditional graph-based ranking methods and recent neural network-based
approaches. Specifically, given a document, a word graph is constructed from
the document based on word proximity and is encoded with graph convolutional
networks, which effectively capture document-level word salience by modeling
long-range dependency between words in the document and aggregating multiple
appearances of identical words into one node. Furthermore, we propose a
diversified point network to generate a set of diverse keyphrases out of the
word graph in the decoding process. Experimental results on five benchmark data
sets show that our proposed method significantly outperforms the existing
state-of-the-art approaches.

negative:
Neural conversation models are known to generate appropriate but
non-informative responses in general. A scenario where informativeness can be
significantly enhanced is Conversing by Reading (CbR), where conversations take
place with respect to a given external document. In previous work, the external
document is utilized by (1) creating a context-aware document memory that
integrates information from the document and the conversational context, and
then (2) generating responses referring to the memory. In this paper, we
propose to create the document memory with some anticipated responses in mind.
This is achieved using a teacher-student framework. The teacher is given the
external document, the context, and the ground-truth response, and learns how
to build a response-aware document memory from three sources of information.
The student learns to construct a response-anticipated document memory from the
first two sources, and the teacher's insight on memory creation. Empirical
results show that our model outperforms the previous state-of-the-art for the
CbR task.

negative:
Models of object vision have been of great interest in computer vision and
visual neuroscience. During the last decades, several models have been
developed to extract visual features from images for object recognition tasks.
Some of these were inspired by the hierarchical structure of primate visual
system, and some others were engineered models. The models are varied in
several aspects: models that are trained by supervision, models trained without
supervision, and models (e.g. feature extractors) that are fully hard-wired and
do not need training. Some of the models come with a deep hierarchical
structure consisting of several layers, and some others are shallow and come
with only one or two layers of processing. More recently, new models have been
developed that are not hand-tuned but trained using millions of images, through
which they learn how to extract informative task-related features. Here I will
survey all these different models and provide the reader with an intuitive, as
well as a more detailed, understanding of the underlying computations in each
of the models.

negative:
Previous researchers have considered sentiment analysis as a document
classification task, in which input documents are classified into predefined
sentiment classes. Although there are sentences in a document that support
important evidences for sentiment analysis and sentences that do not, they have
treated the document as a bag of sentences. In other words, they have not
considered the importance of each sentence in the document. To effectively
determine polarity of a document, each sentence in the document should be dealt
with different degrees of importance. To address this problem, we propose a
document-level sentence classification model based on deep neural networks, in
which the importance degrees of sentences in documents are automatically
determined through gate mechanisms. To verify our new sentiment analysis model,
we conducted experiments using the sentiment datasets in the four different
domains such as movie reviews, hotel reviews, restaurant reviews, and music
reviews. In the experiments, the proposed model outperformed previous
state-of-the-art models that do not consider importance differences of
sentences in a document. The experimental results show that the importance of
sentences should be considered in a document-level sentiment classification
task.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
Electronic Health Records (EHRs) provide a wealth of information for machine
learning algorithms to predict the patient outcome from the data including
diagnostic information, vital signals, lab tests, drug administration, and
demographic information. Machine learning models can be built, for example, to
evaluate patients based on their predicted mortality or morbidity and to
predict required resources for efficient resource management in hospitals. In
this paper, we demonstrate that an attacker can manipulate the machine learning
predictions with EHRs easily and selectively at test time by backdoor attacks
with the poisoned training data. Furthermore, the poison we create has
statistically similar features to the original data making it hard to detect,
and can also attack multiple machine learning models without any knowledge of
the models. With less than 5% of the raw EHR data poisoned, we achieve average
attack success rates of 97% on mortality prediction tasks with MIMIC-III
database against Logistic Regression, Multilayer Perceptron, and Long
Short-term Memory models simultaneously.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
Traditional learning-based coreference resolvers operate by training the
mention-pair model for determining whether two mentions are coreferent or not.
Though conceptually simple and easy to understand, the mention-pair model is
linguistically rather unappealing and lags far behind the heuristic-based
coreference models proposed in the pre-statistical NLP era in terms of
sophistication. Two independent lines of recent research have attempted to
improve the mention-pair model, one by acquiring the mention-ranking model to
rank preceding mentions for a given anaphor, and the other by training the
entity-mention model to determine whether a preceding cluster is coreferent
with a given mention. We propose a cluster-ranking approach to coreference
resolution, which combines the strengths of the mention-ranking model and the
entity-mention model, and is therefore theoretically more appealing than both
of these models. In addition, we seek to improve cluster rankers via two
extensions: (1) lexicalization and (2) incorporating knowledge of anaphoricity
by jointly modeling anaphoricity determination and coreference resolution.
Experimental results on the ACE data sets demonstrate the superior performance
of cluster rankers to competing approaches as well as the effectiveness of our
two extensions.

negative:
We introduce the "Incremental Implicitly-Refined Classi-fication (IIRC)"
setup, an extension to the class incremental learning setup where the incoming
batches of classes have two granularity levels. i.e., each sample could have a
high-level (coarse) label like "bear" and a low-level (fine) label like "polar
bear". Only one label is provided at a time, and the model has to figure out
the other label if it has already learnfed it. This setup is more aligned with
real-life scenarios, where a learner usually interacts with the same family of
entities multiple times, discovers more granularity about them, while still
trying not to forget previous knowledge. Moreover, this setup enables
evaluating models for some important lifelong learning challenges that cannot
be easily addressed under the existing setups. These challenges can be
motivated by the example "if a model was trained on the class bear in one task
and on polar bear in another task, will it forget the concept of bear, will it
rightfully infer that a polar bear is still a bear? and will it wrongfully
associate the label of polar bear to other breeds of bear?". We develop a
standardized benchmark that enables evaluating models on the IIRC setup. We
evaluate several state-of-the-art lifelong learning algorithms and highlight
their strengths and limitations. For example, distillation-based methods
perform relatively well but are prone to incorrectly predicting too many labels
per image. We hope that the proposed setup, along with the benchmark, would
provide a meaningful problem setting to the practitioners

negative:
Recent studies emphasize the need of document context in human evaluation of
machine translations, but little research has been done on the impact of user
interfaces on annotator productivity and the reliability of assessments. In
this work, we compare human assessment data from the last two WMT evaluation
campaigns collected via two different methods for document-level evaluation.
Our analysis shows that a document-centric approach to evaluation where the
annotator is presented with the entire document context on a screen leads to
higher quality segment and document level assessments. It improves the
correlation between segment and document scores and increases inter-annotator
agreement for document scores but is considerably more time consuming for
annotators.

negative:
Term frequency normalization is a serious issue since lengths of documents
are various. Generally, documents become long due to two different reasons -
verbosity and multi-topicality. First, verbosity means that the same topic is
repeatedly mentioned by terms related to the topic, so that term frequency is
more increased than the well-summarized one. Second, multi-topicality indicates
that a document has a broad discussion of multi-topics, rather than single
topic. Although these document characteristics should be differently handled,
all previous methods of term frequency normalization have ignored these
differences and have used a simplified length-driven approach which decreases
the term frequency by only the length of a document, causing an unreasonable
penalization. To attack this problem, we propose a novel TF normalization
method which is a type of partially-axiomatic approach. We first formulate two
formal constraints that the retrieval model should satisfy for documents having
verbose and multi-topicality characteristic, respectively. Then, we modify
language modeling approaches to better satisfy these two constraints, and
derive novel smoothing methods. Experimental results show that the proposed
method increases significantly the precision for keyword queries, and
substantially improves MAP (Mean Average Precision) for verbose queries.