query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I am currently working on a project aimed at developing a chatbot capable of handling open-domain Q&A for technical queries within specific domains. To fine-tune a language model within this domain, I require a suitable dataset for domain adaptation and model fine-tuning. My interest lies in a dataset comprising real-world questions from users on technical forums, as opposed to artificially generated queries. The dataset should ideally be sufficiently large to train a model. Additionally, I am seeking an auxiliary resource for pretraining and learning representations of domain-specific terminologies. This resource could potentially be an external knowledge base, such as Wikipedia. Any academic papers that propose or utilize such a dataset would be of great assistance to my research, as they would aid in the development of a more effective and accurate chatbot system. Furthermore, I am interested in standard methodologies for creating and curating such a dataset.

positive:
With the rise of large-scale pre-trained language models, open-domain
question-answering (ODQA) has become an important research topic in NLP. Based
on the popular pre-training fine-tuning approach, we posit that an additional
in-domain pre-training stage using a large-scale, natural, and diverse
question-answering (QA) dataset can be beneficial for ODQA. Consequently, we
propose a novel QA dataset based on the Common Crawl project in this paper.
Using the readily available schema.org annotation, we extract around 130
million multilingual question-answer pairs, including about 60 million English
data-points. With this previously unseen number of natural QA pairs, we
pre-train popular language models to show the potential of large-scale
in-domain pre-training for the task of question-answering. In our experiments,
we find that pre-training question-answering models on our Common Crawl
Question Answering dataset (CCQA) achieves promising results in zero-shot, low
resource and fine-tuned settings across multiple tasks, models and benchmarks.

positive:
Domain adaptive pretraining, i.e. the continued unsupervised pretraining of a
language model on domain-specific text, improves the modelling of text for
downstream tasks within the domain. Numerous real-world applications are based
on domain-specific text, e.g. working with financial or biomedical documents,
and these applications often need to support multiple languages. However,
large-scale domain-specific multilingual pretraining data for such scenarios
can be difficult to obtain, due to regulations, legislation, or simply a lack
of language- and domain-specific text. One solution is to train a single
multilingual model, taking advantage of the data available in as many languages
as possible. In this work, we explore the benefits of domain adaptive
pretraining with a focus on adapting to multiple languages within a specific
domain. We propose different techniques to compose pretraining corpora that
enable a language model to both become domain-specific and multilingual.
Evaluation on nine domain-specific datasets-for biomedical named entity
recognition and financial sentence classification-covering seven different
languages show that a single multilingual domain-specific model can outperform
the general multilingual model, and performs close to its monolingual
counterpart. This finding holds across two different pretraining methods,
adapter-based pretraining and full model pretraining.

positive:
While self-supervised learning has made rapid advances in natural language
processing, it remains unclear when researchers should engage in
resource-intensive domain-specific pretraining (domain pretraining). The law,
puzzlingly, has yielded few documented instances of substantial gains to domain
pretraining in spite of the fact that legal language is widely seen to be
unique. We hypothesize that these existing results stem from the fact that
existing legal NLP tasks are too easy and fail to meet conditions for when
domain pretraining can help. To address this, we first present CaseHOLD (Case
Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple
choice questions to identify the relevant holding of a cited case. This dataset
presents a fundamental task to lawyers and is both legally meaningful and
difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second,
we assess performance gains on CaseHOLD and existing legal NLP datasets. While
a Transformer architecture (BERT) pretrained on a general corpus (Google Books
and Wikipedia) improves performance, domain pretraining (using corpus of
approximately 3.5M decisions across all courts in the U.S. that is larger than
BERT's) with a custom legal vocabulary exhibits the most substantial
performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12%
improvement on BERT) and consistent performance gains across two other legal
tasks. Third, we show that domain pretraining may be warranted when the task
exhibits sufficient similarity to the pretraining corpus: the level of
performance increase in three legal tasks was directly tied to the domain
specificity of the task. Our findings inform when researchers should engage
resource-intensive pretraining and show that Transformer-based architectures,
too, learn embeddings suggestive of distinct legal language.

positive:
The goal of this work is to build conversational Question Answering (QA)
interfaces for the large body of domain-specific information available in FAQ
sites. We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The
dialogues are collected from three Stack Exchange sites using the Wizard of Oz
method with crowdsourcing. Compared to previous work, DoQA comprises
well-defined information needs, leading to more coherent and natural
conversations with less factoid questions and is multi-domain. In addition, we
introduce a more realistic information retrieval(IR) scenario where the system
needs to find the answer in any of the FAQ documents. The results of an
existing, strong, system show that, thanks to transfer learning from a
Wikipedia QA dataset and fine tuning on a single FAQ domain, it is possible to
build high quality conversational QA systems for FAQs without in-domain
training data. The good results carry over into the more challenging IR
scenario. In both cases, there is still ample room for improvement, as
indicated by the higher human upperbound.

positive:
Question Answering (QA) is a natural language processing task that aims at
obtaining relevant answers to user questions. While some progress has been made
in this area, biomedical questions are still a challenge to most QA approaches,
due to the complexity of the domain and limited availability of training sets.
We present a method to automatically extract question-article pairs from Q\&A
web forums, which can be used for document retrieval, a crucial step of most QA
systems. The proposed framework extracts from selected forums the questions and
the respective answers that contain citations. This way, QA systems based on
document retrieval can be developed and evaluated using the question-article
pairs annotated by users of these forums. We generated the BiQA corpus by
applying our framework to three forums, obtaining 7,453 questions and 14,239
question-article pairs. We evaluated how the number of articles associated with
each question and the number of votes on each answer affects the performance of
baseline document retrieval approaches. Also, we demonstrated that the articles
given as answers are significantly similar to the questions and trained a
state-of-the-art deep learning model that obtained similar performance to using
a dataset manually annotated by experts. The proposed framework can be used to
update the BiQA corpus from the same forums as new posts are made, and from
other forums that support their answers with documents. The BiQA corpus and the
framework used to generate it are available at
\url{https://github.com/lasigeBioTM/BiQA}.

negative:
Domain-specific community question answering is becoming an integral part of
professions. Finding related questions and answers in these communities can
significantly improve the effectiveness and efficiency of information seeking.
Stack Overflow is one of the most popular communities that is being used by
millions of programmers. In this paper, we analyze the problem of predicting
knowledge unit (question thread) relatedness in Stack Overflow. In particular,
we formulate the question relatedness task as a multi-class classification
problem with four degrees of relatedness. We present a large-scale dataset with
more than 300K pairs. To the best of our knowledge, this dataset is the largest
domain-specific dataset for Question-Question relatedness. We present the steps
that we took to collect, clean, process, and assure the quality of the dataset.
The proposed dataset Stack Overflow is a useful resource to develop novel
solutions, specifically data-hungry neural network models, for the prediction
of relatedness in technical community question-answering forums. We adopt a
neural network architecture and a traditional model for this task that
effectively utilize information from different parts of knowledge units to
compute the relatedness between them. These models can be used to benchmark
novel models, as they perform well in our task and in a closely similar task.

negative:
The recent success of neural machine translation models relies on the
availability of high quality, in-domain data. Domain adaptation is required
when domain-specific data is scarce or nonexistent. Previous unsupervised
domain adaptation strategies include training the model with in-domain copied
monolingual or back-translated data. However, these methods use generic
representations for text regardless of domain shift, which makes it infeasible
for translation models to control outputs conditional on a specific domain. In
this work, we propose an approach that adapts models with domain-aware feature
embeddings, which are learned via an auxiliary language modeling task. Our
approach allows the model to assign domain-specific representations to words
and output sentences in the desired domain. Our empirical results demonstrate
the effectiveness of the proposed strategy, achieving consistent improvements
in multiple experimental settings. In addition, we show that combining our
method with back translation can further improve the performance of the model.

negative:
Question answering has seen significant advances in recent times, especially
with the introduction of increasingly bigger transformer-based models
pre-trained on massive amounts of data. While achieving impressive results on
many benchmarks, their performances appear to be proportional to the amount of
training data available in the target domain. In this work, we explore the
ability of current question-answering models to generalize - to both other
domains as well as with restricted training data. We find that large amounts of
training data are necessary, both for pre-training as well as fine-tuning to a
task, for the models to perform well on the designated task. We introduce a
novel abductive reasoning approach based on Grenander's Pattern Theory
framework to provide self-supervised domain adaptation cues or "pseudo-labels,"
which can be used instead of expensive human annotations. The proposed
self-supervised training regimen allows for effective domain adaptation without
losing performance compared to fully supervised baselines. Extensive
experiments on two publicly available benchmarks show the efficacy of the
proposed approach. We show that neural networks models trained using
self-labeled data can retain up to $75\%$ of the performance of models trained
on large amounts of human-annotated training data.

negative:
Question Answering (QA) is in increasing demand as the amount of information
available online and the desire for quick access to this content grows. A
common approach to QA has been to fine-tune a pretrained language model on a
task-specific labeled dataset. This paradigm, however, relies on scarce, and
costly to obtain, large-scale human-labeled data. We propose an unsupervised
approach to training QA models with generated pseudo-training data. We show
that generating questions for QA training by applying a simple template on a
related, retrieved sentence rather than the original context sentence improves
downstream QA performance by allowing the model to learn more complex
context-question relationships. Training a QA model on this data gives a
relative improvement over a previous unsupervised model in F1 score on the
SQuAD dataset by about 14%, and 20% when the answer is a named entity,
achieving state-of-the-art performance on SQuAD for unsupervised QA.

negative:
This paper introduces QAConv, a new question answering (QA) dataset that uses
conversations as a knowledge source. We focus on informative conversations,
including business emails, panel discussions, and work channels. Unlike
open-domain and task-oriented dialogues, these conversations are usually long,
complex, asynchronous, and involve strong domain knowledge. In total, we
collect 34,608 QA pairs from 10,259 selected conversations with both
human-written and machine-generated questions. We use a question generator and
a dialogue summarizer as auxiliary tools to collect and recommend questions.
The dataset has two testing scenarios: chunk mode and full mode, depending on
whether the grounded partial conversation is provided or retrieved.
Experimental results show that state-of-the-art pretrained QA systems have
limited zero-shot performance and tend to predict our questions as
unanswerable. Our dataset provides a new training and evaluation testbed to
facilitate QA on conversations research.

negative:
In a conversational question answering scenario, a questioner seeks to
extract information about a topic through a series of interdependent questions
and answers. As the conversation progresses, they may switch to related topics,
a phenomenon commonly observed in information-seeking search sessions. However,
current datasets for conversational question answering are limiting in two
ways: 1) they do not contain topic switches; and 2) they assume the reference
text for the conversation is given, i.e., the setting is not open-domain. We
introduce TopiOCQA (pronounced Tapioca), an open-domain conversational dataset
with topic switches on Wikipedia. TopiOCQA contains 3,920 conversations with
information-seeking questions and free-form answers. On average, a conversation
in our dataset spans 13 question-answer turns and involves four topics
(documents). TopiOCQA poses a challenging test-bed for models, where efficient
retrieval is required on multiple turns of the same conversation, in
conjunction with constructing valid responses using conversational history. We
evaluate several baselines, by combining state-of-the-art document retrieval
methods with neural reader models. Our best model achieves F1 of 55.8, falling
short of human performance by 14.2 points, indicating the difficulty of our
dataset. Our dataset and code is available at
https://mcgill-nlp.github.io/topiocqa

negative:
Most real world language problems require learning from heterogenous corpora,
raising the problem of learning robust models which generalise well to both
similar (in domain) and dissimilar (out of domain) instances to those seen in
training. This requires learning an underlying task, while not learning
irrelevant signals and biases specific to individual domains. We propose a
novel method to optimise both in- and out-of-domain accuracy based on joint
learning of a structured neural model with domain-specific and domain-general
components, coupled with adversarial training for domain. Evaluating on
multi-domain language identification and multi-domain sentiment analysis, we
show substantial improvements over standard domain adaptation techniques, and
domain-adversarial training.

negative:
Conversational modeling is an important task in natural language
understanding and machine intelligence. Although previous approaches exist,
they are often restricted to specific domains (e.g., booking an airline ticket)
and require hand-crafted rules. In this paper, we present a simple approach for
this task which uses the recently proposed sequence to sequence framework. Our
model converses by predicting the next sentence given the previous sentence or
sentences in a conversation. The strength of our model is that it can be
trained end-to-end and thus requires much fewer hand-crafted rules. We find
that this straightforward model can generate simple conversations given a large
conversational training dataset. Our preliminary results suggest that, despite
optimizing the wrong objective function, the model is able to converse well. It
is able extract knowledge from both a domain specific dataset, and from a
large, noisy, and general domain dataset of movie subtitles. On a
domain-specific IT helpdesk dataset, the model can find a solution to a
technical problem via conversations. On a noisy open-domain movie transcript
dataset, the model can perform simple forms of common sense reasoning. As
expected, we also find that the lack of consistency is a common failure mode of
our model.

negative:
Learning high-quality domain word embeddings is important for achieving good
performance in many NLP tasks. General-purpose embeddings trained on
large-scale corpora are often sub-optimal for domain-specific applications.
However, domain-specific tasks often do not have large in-domain corpora for
training high-quality domain embeddings. In this paper, we propose a novel
lifelong learning setting for domain embedding. That is, when performing the
new domain embedding, the system has seen many past domains, and it tries to
expand the new in-domain corpus by exploiting the corpora from the past domains
via meta-learning. The proposed meta-learner characterizes the similarities of
the contexts of the same word in many domain corpora, which helps retrieve
relevant data from the past domains to expand the new domain corpus.
Experimental results show that domain embeddings produced from such a process
improve the performance of the downstream tasks.

negative:
Language models pretrained on text from a wide variety of sources form the
foundation of today's NLP. In light of the success of these broad-coverage
models, we investigate whether it is still helpful to tailor a pretrained model
to the domain of a target task. We present a study across four domains
(biomedical and computer science publications, news, and reviews) and eight
classification tasks, showing that a second phase of pretraining in-domain
(domain-adaptive pretraining) leads to performance gains, under both high- and
low-resource settings. Moreover, adapting to the task's unlabeled data
(task-adaptive pretraining) improves performance even after domain-adaptive
pretraining. Finally, we show that adapting to a task corpus augmented using
simple data selection strategies is an effective alternative, especially when
resources for domain-adaptive pretraining might be unavailable. Overall, we
consistently find that multi-phase adaptive pretraining offers large gains in
task performance.

negative:
Alongside huge volumes of research on deep learning models in NLP in the
recent years, there has been also much work on benchmark datasets needed to
track modeling progress. Question answering and reading comprehension have been
particularly prolific in this regard, with over 80 new datasets appearing in
the past two years. This study is the largest survey of the field to date. We
provide an overview of the various formats and domains of the current
resources, highlighting the current lacunae for future work. We further discuss
the current classifications of "skills" that question answering/reading
comprehension systems are supposed to acquire, and propose a new taxonomy. The
supplementary materials survey the current multilingual resources and
monolingual resources for languages other than English, and we discuss the
implications of over-focusing on English. The study is aimed at both
practitioners looking for pointers to the wealth of existing data, and at
researchers working on new resources.

negative:
A long-term goal of machine learning is to build intelligent conversational
agents. One recent popular approach is to train end-to-end models on a large
amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals
& Le, 2015; Shang et al., 2015). However, this approach leaves many questions
unanswered as an understanding of the precise successes and shortcomings of
each model is hard to assess. A contrasting recent proposal are the bAbI tasks
(Weston et al., 2015b) which are synthetic data that measure the ability of
learning machines at various reasoning tasks over toy language. Unfortunately,
those tests are very small and hence may encourage methods that do not scale.
In this work, we propose a suite of new tasks of a much larger scale that
attempt to bridge the gap between the two regimes. Choosing the domain of
movies, we provide tasks that test the ability of models to answer factual
questions (utilizing OMDB), provide personalization (utilizing MovieLens),
carry short conversations about the two, and finally to perform on natural
dialogs from Reddit. We provide a dataset covering 75k movie entities and with
3.5M training examples. We present results of various models on these tasks,
and evaluate their performance.

negative:
Question answering and conversational systems are often baffled and need help
clarifying certain ambiguities. However, limitations of existing datasets
hinder the development of large-scale models capable of generating and
utilising clarification questions. In order to overcome these limitations, we
devise a novel bootstrapping framework (based on self-supervision) that assists
in the creation of a diverse, large-scale dataset of clarification questions
based on post-comment tuples extracted from stackexchange. The framework
utilises a neural network based architecture for classifying clarification
questions. It is a two-step method where the first aims to increase the
precision of the classifier and second aims to increase its recall. We
quantitatively demonstrate the utility of the newly created dataset by applying
it to the downstream task of question-answering. The final dataset, ClarQ,
consists of ~2M examples distributed across 173 domains of stackexchange. We
release this dataset in order to foster research into the field of
clarification question generation with the larger goal of enhancing dialog and
question answering systems.

negative:
In the last few years, open-domain question answering (ODQA) has advanced
rapidly due to the development of deep learning techniques and the availability
of large-scale QA datasets. However, the current datasets are essentially
designed for synchronic document collections (e.g., Wikipedia). Temporal news
collections such as long-term news archives spanning several decades, are
rarely used in training the models despite they are quite valuable for our
society. To foster the research in the field of ODQA on such historical
collections, we present ArchivalQA, a large question answering dataset
consisting of 532,444 question-answer pairs which is designed for temporal news
QA. We divide our dataset into four subparts based on the question difficulty
levels and the containment of temporal expressions, which we believe are useful
for training and testing ODQA systems characterized by different strengths and
abilities. The novel QA dataset-constructing framework that we introduce can be
also applied to generate non-ambiguous questions of good quality over other
types of temporal document collections.

negative:
Natural language dialogue systems raise great attention recently. As many
dialogue models are data-driven, high-quality datasets are essential to these
systems. In this paper, we introduce Pchatbot, a large-scale dialogue dataset
that contains two subsets collected from Weibo and Judicial forums
respectively. To adapt the raw dataset to dialogue systems, we elaborately
normalize the raw dataset via processes such as anonymization, deduplication,
segmentation, and filtering. The scale of Pchatbot is significantly larger than
existing Chinese datasets, which might benefit the data-driven models. Besides,
current dialogue datasets for personalized chatbot usually contain several
persona sentences or attributes. Different from existing datasets, Pchatbot
provides anonymized user IDs and timestamps for both posts and responses. This
enables the development of personalized dialogue models that directly learn
implicit user personality from the user's dialogue history. Our preliminary
experimental study benchmarks several state-of-the-art dialogue models to
provide a comparison for future work. The dataset can be publicly accessed at
Github.

negative:
Many users communicate with chatbots and AI assistants in order to help them
with various tasks. A key component of the assistant is the ability to
understand and answer a user's natural language questions for
question-answering (QA). Because data can be usually stored in a structured
manner, an essential step involves turning a natural language question into its
corresponding query language. However, in order to train most natural
language-to-query-language state-of-the-art models, a large amount of training
data is needed first. In most domains, this data is not available and
collecting such datasets for various domains can be tedious and time-consuming.
In this work, we propose a novel method for accelerating the training dataset
collection for developing the natural language-to-query-language machine
learning models. Our system allows one to generate conversational multi-term
data, where multiple turns define a dialogue session, enabling one to better
utilize chatbot interfaces. We train two current state-of-the-art NL-to-QL
models, on both an SQL and SPARQL-based datasets in order to showcase the
adaptability and efficacy of our created data.

negative:
Expert-layman text style transfer technologies have the potential to improve
communication between members of scientific communities and the general public.
High-quality information produced by experts is often filled with difficult
jargon laypeople struggle to understand. This is a particularly notable issue
in the medical domain, where layman are often confused by medical text online.
At present, two bottlenecks interfere with the goal of building high-quality
medical expert-layman style transfer systems: a dearth of pretrained
medical-domain language models spanning both expert and layman terminologies
and a lack of parallel corpora for training the transfer task itself. To
mitigate the first issue, we propose a novel language model (LM) pretraining
task, Knowledge Base Assimilation, to synthesize pretraining data from the
edges of a graph of expert- and layman-style medical terminology terms into an
LM during self-supervised learning. To mitigate the second issue, we build a
large-scale parallel corpus in the medical expert-layman domain using a
margin-based criterion. Our experiments show that transformer-based models
pretrained on knowledge base assimilation and other well-established
pretraining tasks fine-tuning on our new parallel corpus leads to considerable
improvement against expert-layman transfer benchmarks, gaining an average
relative improvement of our human evaluation, the Overall Success Rate (OSR),
by 106%. We release our code and parallel corpus for future research.

negative:
While most previous work has focused on different pretraining objectives and
architectures for transfer learning, we ask how to best adapt the pretrained
model to a given target task. We focus on the two most common forms of
adaptation, feature extraction (where the pretrained weights are frozen), and
directly fine-tuning the pretrained model. Our empirical results across diverse
NLP tasks with two state-of-the-art models show that the relative performance
of fine-tuning vs. feature extraction depends on the similarity of the
pretraining and target tasks. We explore possible explanations for this finding
and provide a set of adaptation guidelines for the NLP practitioner.

negative:
The meaning of a word often varies depending on its usage in different
domains. The standard word embedding models struggle to represent this
variation, as they learn a single global representation for a word. We propose
a method to learn domain-specific word embeddings, from text organized into
hierarchical domains, such as reviews in an e-commerce website, where products
follow a taxonomy. Our structured probabilistic model allows vector
representations for the same word to drift away from each other for distant
domains in the taxonomy, to accommodate its domain-specific meanings. By
learning sets of domain-specific word representations jointly, our model can
leverage domain relationships, and it scales well with the number of domains.
Using large real-world review datasets, we demonstrate the effectiveness of our
model compared to state-of-the-art approaches, in learning domain-specific word
embeddings that are both intuitive to humans and benefit downstream NLP tasks.

negative:
Neural machine translation often adopts the fine-tuning approach to adapt to
specific domains. However, nonrestricted fine-tuning can easily degrade on the
general domain and over-fit to the target domain. To mitigate the issue, we
propose Prune-Tune, a novel domain adaptation method via gradual pruning. It
learns tiny domain-specific sub-networks during fine-tuning on new domains.
Prune-Tune alleviates the over-fitting and the degradation problem without
model modification. Furthermore, Prune-Tune is able to sequentially learn a
single network with multiple disjoint domain-specific sub-networks for multiple
domains. Empirical experiment results show that Prune-Tune outperforms several
strong competitors in the target domain test set without sacrificing the
quality on the general domain in both single and multi-domain settings. The
source code and data are available at https://github.com/ohlionel/Prune-Tune.

negative:
Contextual embedding-based language models trained on large data sets, such
as BERT and RoBERTa, provide strong performance across a wide range of tasks
and are ubiquitous in modern NLP. It has been observed that fine-tuning these
models on tasks involving data from domains different from that on which they
were pretrained can lead to suboptimal performance. Recent work has explored
approaches to adapt pretrained language models to new domains by incorporating
additional pretraining using domain-specific corpora and task data. We propose
an alternative approach for transferring pretrained language models to new
domains by adapting their tokenizers. We show that domain-specific subword
sequences can be efficiently determined directly from divergences in the
conditional token distributions of the base and domain-specific corpora. In
datasets from four disparate domains, we find adaptive tokenization on a
pretrained RoBERTa model provides >97% of the performance benefits of domain
specific pretraining. Our approach produces smaller models and less training
and inference time than other approaches using tokenizer augmentation. While
adaptive tokenization incurs a 6% increase in model parameters in our
experimentation, due to the introduction of 10k new domain-specific tokens, our
approach, using 64 vCPUs, is 72x faster than further pretraining the language
model on domain-specific corpora on 8 TPUs.

negative:
Domain Adaptation arises when we aim at learning from source domain a model
that can per- form acceptably well on a different target domain. It is
especially crucial for Natural Language Generation (NLG) in Spoken Dialogue
Systems when there are sufficient annotated data in the source domain, but
there is a limited labeled data in the target domain. How to effectively
utilize as much of existing abilities from source domains is a crucial issue in
domain adaptation. In this paper, we propose an adversarial training procedure
to train a Variational encoder-decoder based language generator via multiple
adaptation steps. In this procedure, a model is first trained on a source
domain data and then fine-tuned on a small set of target domain utterances
under the guidance of two proposed critics. Experimental results show that the
proposed method can effec- tively leverage the existing knowledge in the source
domain to adapt to another related domain by using only a small amount of
in-domain data.

negative:
We propose a simple domain adaptation method for neural networks in a
supervised setting. Supervised domain adaptation is a way of improving the
generalization performance on the target domain by using the source domain
dataset, assuming that both of the datasets are labeled. Recently, recurrent
neural networks have been shown to be successful on a variety of NLP tasks such
as caption generation; however, the existing domain adaptation techniques are
limited to (1) tune the model parameters by the target dataset after the
training by the source dataset, or (2) design the network to have dual output,
one for the source domain and the other for the target domain. Reformulating
the idea of the domain adaptation technique proposed by Daume (2007), we
propose a simple domain adaptation method, which can be applied to neural
networks trained with a cross-entropy loss. On captioning datasets, we show
performance improvements over other domain adaptation methods.

negative:
This work presents biomedical and clinical language models for Spanish by
experimenting with different pretraining choices, such as masking at word and
subword level, varying the vocabulary size and testing with domain data,
looking for better language representations. Interestingly, in the absence of
enough clinical data to train a model from scratch, we applied mixed-domain
pretraining and cross-domain transfer approaches to generate a performant
bio-clinical model suitable for real-world clinical data. We evaluated our
models on Named Entity Recognition (NER) tasks for biomedical documents and
challenging hospital discharge reports. When compared against the competitive
mBERT and BETO models, we outperform them in all NER tasks by a significant
margin. Finally, we studied the impact of the model's vocabulary on the NER
performances by offering an interesting vocabulary-centric analysis. The
results confirm that domain-specific pretraining is fundamental to achieving
higher performances in downstream NER tasks, even within a mid-resource
scenario. To the best of our knowledge, we provide the first biomedical and
clinical transformer-based pretrained language models for Spanish, intending to
boost native Spanish NLP applications in biomedicine. Our best models are
freely available in the HuggingFace hub: https://huggingface.co/BSC-TeMU.

negative:
Building automatic technical support system is an important yet challenge
task. Conceptually, to answer a user question on a technical forum, a human
expert has to first retrieve relevant documents, and then read them carefully
to identify the answer snippet. Despite huge success the researchers have
achieved in coping with general domain question answering (QA), much less
attentions have been paid for investigating technical QA. Specifically,
existing methods suffer from several unique challenges (i) the question and
answer rarely overlaps substantially and (ii) very limited data size. In this
paper, we propose a novel framework of deep transfer learning to effectively
address technical QA across tasks and domains. To this end, we present an
adjustable joint learning approach for document retrieval and reading
comprehension tasks. Our experiments on the TechQA demonstrates superior
performance compared with state-of-the-art methods.

negative:
Research in question answering datasets and models has gained a lot of
attention in the research community. Many of them release their own question
answering datasets as well as the models. There is tremendous progress that we
have seen in this area of research. The aim of this survey is to recognize,
summarize and analyze the existing datasets that have been released by many
researchers, especially in non-English datasets as well as resources such as
research code, and evaluation metrics. In this paper, we review question
answering datasets that are available in common languages other than English
such as French, German, Japanese, Chinese, Arabic, Russian, as well as the
multilingual and cross-lingual question-answering datasets.

negative:
One of the difficulties in training dialogue systems is the lack of training
data. We explore the possibility of creating dialogue data through the
interaction between a dialogue system and a user simulator. Our goal is to
develop a modelling framework that can incorporate new dialogue scenarios
through self-play between the two agents. In this framework, we first pre-train
the two agents on a collection of source domain dialogues, which equips the
agents to converse with each other via natural language. With further
fine-tuning on a small amount of target domain data, the agents continue to
interact with the aim of improving their behaviors using reinforcement learning
with structured reward functions. In experiments on the MultiWOZ dataset, two
practical transfer learning problems are investigated: 1) domain adaptation and
2) single-to-multiple domain transfer. We demonstrate that the proposed
framework is highly effective in bootstrapping the performance of the two
agents in transfer learning. We also show that our method leads to improvements
in dialogue system performance on complete datasets.

negative:
Numerous new dialog domains are being created every day while collecting data
for these domains is extremely costly since it involves human interactions.
Therefore, it is essential to develop algorithms that can adapt to different
domains efficiently when building data-driven dialog models. The most recent
researches on domain adaption focus on giving the model a better
initialization, rather than optimizing the adaptation process. We propose an
efficient domain adaptive task-oriented dialog system model, which incorporates
a meta-teacher model to emphasize the different impacts between generated
tokens with respect to the context. We first train our base dialog model and
meta-teacher model adversarially in a meta-learning setting on rich-resource
domains. The meta-teacher learns to quantify the importance of tokens under
different contexts across different domains. During adaptation, the
meta-teacher guides the dialog model to focus on important tokens in order to
achieve better adaptation efficiency. We evaluate our model on two multi-domain
datasets, MultiWOZ and Google Schema-Guided Dialogue, and achieve
state-of-the-art performance.

negative:
To bridge the gap between the capabilities of the state-of-the-art in factoid
question answering (QA) and what users ask, we need large datasets of real user
questions that capture the various question phenomena users are interested in,
and the diverse ways in which these questions are formulated. We introduce
ComQA, a large dataset of real user questions that exhibit different
challenging aspects such as compositionality, temporal reasoning, and
comparisons. ComQA questions come from the WikiAnswers community QA platform,
which typically contains questions that are not satisfactorily answerable by
existing search engine technology. Through a large crowdsourcing effort, we
clean the question dataset, group questions into paraphrase clusters, and
annotate clusters with their answers. ComQA contains 11,214 questions grouped
into 4,834 paraphrase clusters. We detail the process of constructing ComQA,
including the measures taken to ensure its high quality while making effective
use of crowdsourcing. We also present an extensive analysis of the dataset and
the results achieved by state-of-the-art systems on ComQA, demonstrating that
our dataset can be a driver of future research on QA.

negative:
Recently multi-domain recommender systems have received much attention from
researchers because they can solve cold-start problem as well as support for
cross-selling. However, when applying into multi-domain items, although
algorithms specifically addressing a single domain have many difficulties in
capturing the specific characteristics of each domain, multi-domain algorithms
have less opportunity to obtain similar features among domains. Because both
similarities and differences exist among domains, multi-domain models must
capture both to achieve good performance. Other studies of multi-domain systems
merely transfer knowledge from the source domain to the target domain, so the
source domain usually comes from external factors such as the search query or
social network, which is sometimes impossible to obtain. To handle the two
problems, we propose a model that can extract both homogeneous and divergent
features among domains and extract data in a domain can support for other
domain equally: a so-called Domain-to-Domain Translation Model (D2D-TM). It is
based on generative adversarial networks (GANs), Variational Autoencoders
(VAEs), and Cycle-Consistency (CC) for weight-sharing. We use the user
interaction history of each domain as input and extract latent features through
a VAE-GAN-CC network. Experiments underscore the effectiveness of the proposed
system over state-of-the-art methods by a large margin.

negative:
Contextualized word embeddings such as ELMo and BERT provide a foundation for
strong performance across a wide range of natural language processing tasks by
pretraining on large corpora of unlabeled text. However, the applicability of
this approach is unknown when the target domain varies substantially from the
pretraining corpus. We are specifically interested in the scenario in which
labeled data is available in only a canonical source domain such as newstext,
and the target domain is distinct from both the labeled and pretraining texts.
To address this scenario, we propose domain-adaptive fine-tuning, in which the
contextualized embeddings are adapted by masked language modeling on text from
the target domain. We test this approach on sequence labeling in two
challenging domains: Early Modern English and Twitter. Both domains differ
substantially from existing pretraining corpora, and domain-adaptive
fine-tuning yields substantial improvements over strong BERT baselines, with
particularly impressive results on out-of-vocabulary words. We conclude that
domain-adaptive fine-tuning offers a simple and effective approach for the
unsupervised adaptation of sequence labeling to difficult new domains.

negative:
Domain adaptation of neural networks commonly relies on three training
phases: pretraining, selected data training and then fine tuning. Data
selection improves target domain generalization by training further on
pretraining data identified by relying on a small sample of target domain data.
This work examines the benefit of data selection for language modeling and
machine translation. Our experiments assess the complementarity of selection
with fine tuning and result in practical recommendations: (i) selected data
must be similar to the fine-tuning domain but not so much as to erode the
complementary effect of fine-tuning; (ii) there is a trade-off between
selecting little data for fast but limited progress or much data for slow but
long lasting progress; (iii) data selection can be applied early during
pretraining, with performance gains comparable to long pretraining session;
(iv) data selection from domain classifiers is often more effective than the
popular contrastive data selection method.

negative:
The key challenge of multi-domain translation lies in simultaneously encoding
both the general knowledge shared across domains and the particular knowledge
distinctive to each domain in a unified model. Previous work shows that the
standard neural machine translation (NMT) model, trained on mixed-domain data,
generally captures the general knowledge, but misses the domain-specific
knowledge. In response to this problem, we augment NMT model with additional
domain transformation networks to transform the general representations to
domain-specific representations, which are subsequently fed to the NMT decoder.
To guarantee the knowledge transformation, we also propose two complementary
supervision signals by leveraging the power of knowledge distillation and
adversarial learning. Experimental results on several language pairs, covering
both balanced and unbalanced multi-domain translation, demonstrate the
effectiveness and universality of the proposed approach. Encouragingly, the
proposed unified model achieves comparable results with the fine-tuning
approach that requires multiple models to preserve the particular knowledge.
Further analyses reveal that the domain transformation networks successfully
capture the domain-specific knowledge as expected.

negative:
Named entity recognition (NER) for identifying proper nouns in unstructured
text is one of the most important and fundamental tasks in natural language
processing. However, despite the widespread use of NER models, they still
require a large-scale labeled data set, which incurs a heavy burden due to
manual annotation. Domain adaptation is one of the most promising solutions to
this problem, where rich labeled data from the relevant source domain are
utilized to strengthen the generalizability of a model based on the target
domain. However, the mainstream cross-domain NER models are still affected by
the following two challenges (1) Extracting domain-invariant information such
as syntactic information for cross-domain transfer. (2) Integrating
domain-specific information such as semantic information into the model to
improve the performance of NER. In this study, we present a semi-supervised
framework for transferable NER, which disentangles the domain-invariant latent
variables and domain-specific latent variables. In the proposed framework, the
domain-specific information is integrated with the domain-specific latent
variables by using a domain predictor. The domain-specific and domain-invariant
latent variables are disentangled using three mutual information regularization
terms, i.e., maximizing the mutual information between the domain-specific
latent variables and the original embedding, maximizing the mutual information
between the domain-invariant latent variables and the original embedding, and
minimizing the mutual information between the domain-specific and
domain-invariant latent variables. Extensive experiments demonstrated that our
model can obtain state-of-the-art performance with cross-domain and
cross-lingual NER benchmark data sets.

negative:
One weakness of machine-learned NLP models is that they typically perform
poorly on out-of-domain data. In this work, we study the task of identifying
products being bought and sold in online cybercrime forums, which exhibits
particularly challenging cross-domain effects. We formulate a task that
represents a hybrid of slot-filling information extraction and named entity
recognition and annotate data from four different forums. Each of these forums
constitutes its own "fine-grained domain" in that the forums cover different
market sectors with different properties, even though all forums are in the
broad domain of cybercrime. We characterize these domain differences in the
context of a learning-based system: supervised models see decreased accuracy
when applied to new forums, and standard techniques for semi-supervised
learning and domain adaptation have limited effectiveness on this data, which
suggests the need to improve these techniques. We release a dataset of 1,938
annotated posts from across the four forums.

negative:
Linking clinical narratives to standardized vocabularies and coding systems
is a key component of unlocking the information in medical text for analysis.
However, many domains of medical concepts lack well-developed terminologies
that can support effective coding of medical text. We present a framework for
developing natural language processing (NLP) technologies for automated coding
of under-studied types of medical information, and demonstrate its
applicability via a case study on physical mobility function. Mobility is a
component of many health measures, from post-acute care and surgical outcomes
to chronic frailty and disability, and is coded in the International
Classification of Functioning, Disability, and Health (ICF). However, mobility
and other types of functional activity remain under-studied in medical
informatics, and neither the ICF nor commonly-used medical terminologies
capture functional status terminology in practice. We investigated two
data-driven paradigms, classification and candidate selection, to link
narrative observations of mobility to standardized ICF codes, using a dataset
of clinical narratives from physical therapy encounters. Recent advances in
language modeling and word embedding were used as features for established
machine learning models and a novel deep learning approach, achieving a macro
F-1 score of 84% on linking mobility activity reports to ICF codes. Both
classification and candidate selection approaches present distinct strengths
for automated coding in under-studied domains, and we highlight that the
combination of (i) a small annotated data set; (ii) expert definitions of codes
of interest; and (iii) a representative text corpus is sufficient to produce
high-performing automated coding systems. This study has implications for the
ongoing growth of NLP tools for a variety of specialized applications in
clinical care and research.

negative:
Neural rankers based on deep pretrained language models (LMs) have been shown
to improve many information retrieval benchmarks. However, these methods are
affected by their the correlation between pretraining domain and target domain
and rely on massive fine-tuning relevance labels. Directly applying pretraining
methods to specific domains may result in suboptimal search quality because
specific domains may have domain adaption problems, such as the COVID domain.
This paper presents a search system to alleviate the special domain adaption
problem. The system utilizes the domain-adaptive pretraining and few-shot
learning technologies to help neural rankers mitigate the domain discrepancy
and label scarcity problems. Besides, we also integrate dense retrieval to
alleviate traditional sparse retrieval's vocabulary mismatch obstacle. Our
system performs the best among the non-manual runs in Round 2 of the TREC-COVID
task, which aims to retrieve useful information from scientific literature
related to COVID-19. Our code is publicly available at
https://github.com/thunlp/OpenMatch.

negative:
Numerous recent works have proposed pretraining generic visio-linguistic
representations and then finetuning them for downstream vision and language
tasks. While architecture and objective function design choices have received
attention, the choice of pretraining datasets has received little attention. In
this work, we question some of the default choices made in literature. For
instance, we systematically study how varying similarity between the
pretraining dataset domain (textual and visual) and the downstream domain
affects performance. Surprisingly, we show that automatically generated data in
a domain closer to the downstream task (e.g., VQA v2) is a better choice for
pretraining than "natural" data but of a slightly different domain (e.g.,
Conceptual Captions). On the other hand, some seemingly reasonable choices of
pretraining datasets were found to be entirely ineffective for some downstream
tasks. This suggests that despite the numerous recent efforts, vision &
language pretraining does not quite work "out of the box" yet. Overall, as a
by-product of our study, we find that simple design choices in pretraining can
help us achieve close to state-of-art results on downstream tasks without any
architectural changes.

negative:
In this paper, we propose a novel domain adaptation method named "mixed fine
tuning" for neural machine translation (NMT). We combine two existing
approaches namely fine tuning and multi domain NMT. We first train an NMT model
on an out-of-domain parallel corpus, and then fine tune it on a parallel corpus
which is a mix of the in-domain and out-of-domain corpora. All corpora are
augmented with artificial tags to indicate specific domains. We empirically
compare our proposed method against fine tuning and multi domain methods and
discuss its benefits and shortcomings.

negative:
Spoken language understanding has been addressed as a supervised learning
problem, where a set of training data is available for each domain. However,
annotating data for each domain is both financially costly and non-scalable so
we should fully utilize information across all domains. One existing approach
solves the problem by conducting multi-domain learning, using shared parameters
for joint training across domains. We propose to improve the parameterization
of this method by using domain-specific and task-specific model parameters to
improve knowledge learning and transfer. Experiments on 5 domains show that our
model is more effective for multi-domain SLU and obtain the best results. In
addition, we show its transferability by outperforming the prior best model by
12.4\% when adapting to a new domain with little data.

negative:
In this paper we present the results of our experiments in training and
deploying a self-supervised retrieval-based chatbot trained with contrastive
learning for assisting customer support agents. In contrast to most existing
research papers in this area where the focus is on solving just one component
of a deployable chatbot, we present an end-to-end set of solutions to take the
reader from an unlabelled chatlogs to a deployed chatbot. This set of solutions
includes creating a self-supervised dataset and a weakly labelled dataset from
chatlogs, as well as a systematic approach to selecting a fixed list of canned
responses. We present a hierarchical-based RNN architecture for the response
selection model, chosen for its ability to cache intermediate utterance
embeddings, which helped to meet deployment inference speed requirements. We
compare the performance of this architecture across 3 different learning
objectives: self-supervised contrastive learning, binary classification, and
multi-class classification. We find that using a self-supervised contrastive
learning model outperforms training the binary and multi-class classification
models on a weakly labelled dataset. Our results validate that the
self-supervised contrastive learning approach can be effectively used for a
real-world chatbot scenario.

negative:
Neural network methods exhibit strong performance only in a few resource-rich
domains. Practitioners, therefore, employ domain adaptation from resource-rich
domains that are, in most cases, distant from the target domain. Domain
adaptation between distant domains (e.g., movie subtitles and research papers),
however, cannot be performed effectively due to mismatches in vocabulary; it
will encounter many domain-specific words (e.g., "angstrom") and words whose
meanings shift across domains(e.g., "conductor"). In this study, aiming to
solve these vocabulary mismatches in domain adaptation for neural machine
translation (NMT), we propose vocabulary adaptation, a simple method for
effective fine-tuning that adapts embedding layers in a given pre-trained NMT
model to the target domain. Prior to fine-tuning, our method replaces the
embedding layers of the NMT model by projecting general word embeddings induced
from monolingual data in a target domain onto a source-domain embedding space.
Experimental results indicate that our method improves the performance of
conventional fine-tuning by 3.86 and 3.28 BLEU points in En-Ja and De-En
translation, respectively.

negative:
Frequently asked question (FAQ) retrieval, with the purpose of providing
information on frequent questions or concerns, has far-reaching applications in
many areas, where a collection of question-answer (Q-A) pairs compiled a priori
can be employed to retrieve an appropriate answer in response to a user\u2019s
query that is likely to reoccur frequently. To this end, predominant approaches
to FAQ retrieval typically rank question-answer pairs by considering either the
similarity between the query and a question (q-Q), the relevance between the
query and the associated answer of a question (q-A), or combining the clues
gathered from the q-Q similarity measure and the q-A relevance measure. In this
paper, we extend this line of research by combining the clues gathered from the
q-Q similarity measure and the q-A relevance measure and meanwhile injecting
extra word interaction information, distilled from a generic (open domain)
knowledge base, into a contextual language model for inferring the q-A
relevance. Furthermore, we also explore to capitalize on domain-specific
topically-relevant relations between words in an unsupervised manner, acting as
a surrogate to the supervised domain-specific knowledge base information. As
such, it enables the model to equip sentence representations with the knowledge
about domain-specific and topically-relevant relations among words, thereby
providing a better q-A relevance measure. We evaluate variants of our approach
on a publicly-available Chinese FAQ dataset, and further apply and
contextualize it to a large-scale question-matching task, which aims to search
questions from a QA dataset that have a similar intent as an input query.
Extensive experimental results on these two datasets confirm the promising
performance of the proposed approach in relation to some state-of-the-art ones.

negative:
We develop a chatbot using Deep Bidirectional Transformer models (BERT) to
handle client questions in financial investment customer service. The bot can
recognize 381 intents, and decides when to say "I don't know" and escalates
irrelevant/uncertain questions to human operators. Our main novel contribution
is the discussion about uncertainty measure for BERT, where three different
approaches are systematically compared on real problems. We investigated two
uncertainty metrics, information entropy and variance of dropout sampling in
BERT, followed by mixed-integer programming to optimize decision thresholds.
Another novel contribution is the usage of BERT as a language model in
automatic spelling correction. Inputs with accidental spelling errors can
significantly decrease intent classification performance. The proposed approach
combines probabilities from masked language model and word edit distances to
find the best corrections for misspelled words. The chatbot and the entire
conversational AI system are developed using open-source tools, and deployed
within our company's intranet. The proposed approach can be useful for
industries seeking similar in-house solutions in their specific business
domains. We share all our code and a sample chatbot built on a public dataset
on Github.

negative:
With social media becoming increasingly pop-ular on which lots of news and
real-time eventsare reported, developing automated questionanswering systems is
critical to the effective-ness of many applications that rely on real-time
knowledge. While previous datasets haveconcentrated on question answering (QA)
forformal text like news and Wikipedia, wepresent the first large-scale dataset
for QA oversocial media data. To ensure that the tweetswe collected are useful,
we only gather tweetsused by journalists to write news articles. Wethen ask
human annotators to write questionsand answers upon these tweets. Unlike
otherQA datasets like SQuAD in which the answersare extractive, we allow the
answers to be ab-stractive. We show that two recently proposedneural models
that perform well on formaltexts are limited in their performance when ap-plied
to our dataset. In addition, even the fine-tuned BERT model is still lagging
behind hu-man performance with a large margin. Our re-sults thus point to the
need of improved QAsystems targeting social media text.

negative:
Much of NLP research has focused on crowdsourced static datasets and the
supervised learning paradigm of training once and then evaluating test
performance. As argued in de Vries et al. (2020), crowdsourced data has the
issues of lack of naturalness and relevance to real-world use cases, while the
static dataset paradigm does not allow for a model to learn from its
experiences of using language (Silver et al., 2013). In contrast, one might
hope for machine learning systems that become more useful as they interact with
people. In this work, we build and deploy a role-playing game, whereby human
players converse with learning agents situated in an open-domain fantasy world.
We show that by training models on the conversations they have with humans in
the game the models progressively improve, as measured by automatic metrics and
online engagement scores. This learning is shown to be more efficient than
crowdsourced data when applied to conversations with real users, as well as
being far cheaper to collect.

negative:
Continuously-growing data volumes lead to larger generic models. Specific
use-cases are usually left out, since generic models tend to perform poorly in
domain-specific cases. Our work addresses this gap with a method for selecting
in-domain data from generic-domain (parallel text) corpora, for the task of
machine translation. The proposed method ranks sentences in parallel
general-domain data according to their cosine similarity with a monolingual
domain-specific data set. We then select the top K sentences with the highest
similarity score to train a new machine translation system tuned to the
specific in-domain data. Our experimental results show that models trained on
this in-domain data outperform models trained on generic or a mixture of
generic and domain data. That is, our method selects high-quality
domain-specific training instances at low computational cost and data size.

negative:
Chatbot is a technology that is used to mimic human behavior using natural
language. There are different types of Chatbot that can be used as
conversational agent in various business domains in order to increase the
customer service and satisfaction. For any business domain, it requires a
knowledge base to be built for that domain and design an information retrieval
based system that can respond the user with a piece of documentation or
generated sentences. The core component of a Chatbot is Natural Language
Understanding (NLU) which has been impressively improved by deep learning
methods. But we often lack such properly built NLU modules and requires more
time to build it from scratch for high quality conversations. This may
encourage fresh learners to build a Chatbot from scratch with simple
architecture and using small dataset, although it may have reduced
functionality, rather than building high quality data driven methods. This
research focuses on Named Entity Recognition (NER) and Intent Classification
models which can be integrated into NLU service of a Chatbot. Named entities
will be inserted manually in the knowledge base and automatically detected in a
given sentence. The NER model in the proposed architecture is based on
artificial neural network which is trained on manually created entities and
evaluated using CoNLL-2003 dataset.

negative:
Aspect-based summarization is the task of generating focused summaries based
on specific points of interest. Such summaries aid efficient analysis of text,
such as quickly understanding reviews or opinions from different angles.
However, due to large differences in the type of aspects for different domains
(e.g., sentiment, product features), the development of previous models has
tended to be domain-specific. In this paper, we propose WikiAsp, a large-scale
dataset for multi-domain aspect-based summarization that attempts to spur
research in the direction of open-domain aspect-based summarization.
Specifically, we build the dataset using Wikipedia articles from 20 different
domains, using the section titles and boundaries of each article as a proxy for
aspect annotation. We propose several straightforward baseline models for this
task and conduct experiments on the dataset. Results highlight key challenges
that existing summarization models face in this setting, such as proper pronoun
handling of quoted sources and consistent explanation of time-sensitive events.

negative:
Fine-tuning pre-trained Neural Machine Translation (NMT) models is the
dominant approach for adapting to new languages and domains. However,
fine-tuning requires adapting and maintaining a separate model for each target
task. We propose a simple yet efficient approach for adaptation in NMT. Our
proposed approach consists of injecting tiny task specific adapter layers into
a pre-trained model. These lightweight adapters, with just a small fraction of
the original model size, adapt the model to multiple individual tasks
simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation
and (ii) Massively Multilingual NMT. Experiments on domain adaptation
demonstrate that our proposed approach is on par with full fine-tuning on
various domains, dataset sizes and model capacities. On a massively
multilingual dataset of 103 languages, our adaptation approach bridges the gap
between individual bilingual models and one massively multilingual model for
most language pairs, paving the way towards universal machine translation.

negative:
Given appropriate representations of the semantic relations between carpenter
and wood and between mason and stone (for example, vectors in a vector space
model), a suitable algorithm should be able to recognize that these relations
are highly similar (carpenter is to wood as mason is to stone; the relations
are analogous). Likewise, with representations of dog, house, and kennel, an
algorithm should be able to recognize that the semantic composition of dog and
house, dog house, is highly similar to kennel (dog house and kennel are
synonymous). It seems that these two tasks, recognizing relations and
compositions, are closely connected. However, up to now, the best models for
relations are significantly different from the best models for compositions. In
this paper, we introduce a dual-space model that unifies these two tasks. This
model matches the performance of the best previous models for relations and
compositions. The dual-space model consists of a space for measuring domain
similarity and a space for measuring function similarity. Carpenter and wood
share the same domain, the domain of carpentry. Mason and stone share the same
domain, the domain of masonry. Carpenter and mason share the same function, the
function of artisans. Wood and stone share the same function, the function of
materials. In the composition dog house, kennel has some domain overlap with
both dog and house (the domains of pets and buildings). The function of kennel
is similar to the function of house (the function of shelters). By combining
domain and function similarities in various ways, we can model relations,
compositions, and other aspects of semantics.

negative:
GNNs and chemical fingerprints are the predominant approaches to representing
molecules for property prediction. However, in NLP, transformers have become
the de-facto standard for representation learning thanks to their strong
downstream task transfer. In parallel, the software ecosystem around
transformers is maturing rapidly, with libraries like HuggingFace and BertViz
enabling streamlined training and introspection. In this work, we make one of
the first attempts to systematically evaluate transformers on molecular
property prediction tasks via our ChemBERTa model. ChemBERTa scales well with
pretraining dataset size, offering competitive downstream performance on
MoleculeNet and useful attention-based visualization modalities. Our results
suggest that transformers offer a promising avenue of future work for molecular
representation learning and property prediction. To facilitate these efforts,
we release a curated dataset of 77M SMILES from PubChem suitable for
large-scale self-supervised pretraining.

negative:
We present BART, a denoising autoencoder for pretraining sequence-to-sequence
models. BART is trained by (1) corrupting text with an arbitrary noising
function, and (2) learning a model to reconstruct the original text. It uses a
standard Tranformer-based neural machine translation architecture which,
despite its simplicity, can be seen as generalizing BERT (due to the
bidirectional encoder), GPT (with the left-to-right decoder), and many other
more recent pretraining schemes. We evaluate a number of noising approaches,
finding the best performance by both randomly shuffling the order of the
original sentences and using a novel in-filling scheme, where spans of text are
replaced with a single mask token. BART is particularly effective when fine
tuned for text generation but also works well for comprehension tasks. It
matches the performance of RoBERTa with comparable training resources on GLUE
and SQuAD, achieves new state-of-the-art results on a range of abstractive
dialogue, question answering, and summarization tasks, with gains of up to 6
ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system
for machine translation, with only target language pretraining. We also report
ablation experiments that replicate other pretraining schemes within the BART
framework, to better measure which factors most influence end-task performance.

negative:
The goal of semi-supervised learning is to utilize the unlabeled, in-domain
dataset U to improve models trained on the labeled dataset D. Under the context
of large-scale language-model (LM) pretraining, how we can make the best use of
U is poorly understood: is semi-supervised learning still beneficial with the
presence of large-scale pretraining? should U be used for in-domain LM
pretraining or pseudo-label generation? how should the pseudo-label based
semi-supervised model be actually implemented? how different semi-supervised
strategies affect performances regarding D of different sizes, U of different
sizes, etc. In this paper, we conduct comprehensive studies on semi-supervised
learning in the task of text classification under the context of large-scale LM
pretraining. Our studies shed important lights on the behavior of
semi-supervised learning methods: (1) with the presence of in-domain
pretraining LM on U, open-domain LM pretraining is unnecessary; (2) both the
in-domain pretraining strategy and the pseudo-label based strategy introduce
significant performance boosts, with the former performing better with larger
U, the latter performing better with smaller U, and the combination leading to
the largest performance boost; (3) self-training (pretraining first on pseudo
labels D' and then fine-tuning on D) yields better performances when D is
small, while joint training on the combination of pseudo labels D' and the
original dataset D yields better performances when D is large. Using
semi-supervised learning strategies, we are able to achieve a performance of
around 93.8% accuracy with only 50 training data points on the IMDB dataset,
and a competitive performance of 96.6% with the full IMDB dataset. Our work
marks an initial step in understanding the behavior of semi-supervised learning
models under the context of large-scale pretraining.

negative:
We study the problem of recommending relevant products to users in relatively
resource-scarce markets by leveraging data from similar, richer in resource
auxiliary markets. We hypothesize that data from one market can be used to
improve performance in another. Only a few studies have been conducted in this
area, partly due to the lack of publicly available experimental data. To this
end, we collect and release XMarket, a large dataset covering 18 local markets
on 16 different product categories, featuring 52.5 million user-item
interactions. We introduce and formalize the problem of cross-market product
recommendation, i.e., market adaptation. We explore different market-adaptation
techniques inspired by state-of-the-art domain-adaptation and meta-learning
approaches and propose a novel neural approach for market adaptation, named
FOREC. Our model follows a three-step procedure -- pre-training, forking, and
fine-tuning -- in order to fully utilize the data from an auxiliary market as
well as the target market. We conduct extensive experiments studying the impact
of market adaptation on different pairs of markets. Our proposed approach
demonstrates robust effectiveness, consistently improving the performance on
target markets compared to competitive baselines selected for our analysis. In
particular, FOREC improves on average 24% and up to 50% in terms of nDCG@10,
compared to the NMF baseline. Our analysis and experiments suggest specific
future directions in this research area. We release our data and code for
academic purposes.

negative:
Cross-domain sentiment classification has drawn much attention in recent
years. Most existing approaches focus on learning domain-invariant
representations in both the source and target domains, while few of them pay
attention to the domain-specific information. Despite the non-transferability
of the domain-specific information, simultaneously learning domain-dependent
representations can facilitate the learning of domain-invariant
representations. In this paper, we focus on aspect-level cross-domain sentiment
classification, and propose to distill the domain-invariant sentiment features
with the help of an orthogonal domain-dependent task, i.e. aspect detection,
which is built on the aspects varying widely in different domains. We conduct
extensive experiments on three public datasets and the experimental results
demonstrate the effectiveness of our method.

negative:
The aim of this work is to create a framework for synthetically generating
question/query pairs with as little human input as possible. These datasets can
be used to train machine translation systems to convert natural language
questions into queries, a useful tool that could allow for more natural access
to database information. Existing methods of dataset generation require human
input that scales linearly with the size of the dataset, resulting in small
datasets. Aside from a short initial configuration task, no human input is
required during the query generation process of our system. We leverage
WikiData, a knowledge base of RDF triples, as a source for generating the main
content of questions and queries. Using multiple layers of question templating
we are able to sidestep some of the most challenging parts of query generation
that have been handled by humans in previous methods; humans never have to
modify, aggregate, inspect, annotate, or generate any questions or queries at
any step in the process. Our system is easily configurable to multiple domains
and can be modified to generate queries in natural languages other than
English. We also present an example dataset of 110,000 question/query pairs
across four WikiData domains. We then present a baseline model that we train
using the dataset which shows promise in a commercial QA setting.

negative:
Recently, semantic parsing has attracted much attention in the community.
Although many neural modeling efforts have greatly improved the performance, it
still suffers from the data scarcity issue. In this paper, we propose a novel
semantic parser for domain adaptation, where we have much fewer annotated data
in the target domain compared to the source domain. Our semantic parser
benefits from a two-stage coarse-to-fine framework, thus can provide different
and accurate treatments for the two stages, i.e., focusing on domain invariant
and domain specific information, respectively. In the coarse stage, our novel
domain discrimination component and domain relevance attention encourage the
model to learn transferable domain general structures. In the fine stage, the
model is guided to concentrate on domain related details. Experiments on a
benchmark dataset show that our method consistently outperforms several popular
domain adaptation strategies. Additionally, we show that our model can well
exploit limited target data to capture the difference between the source and
target domain, even when the target domain has far fewer training instances.

negative:
Previous studies on the domain adaptation for neural machine translation
(NMT) mainly focus on the one-pass transferring out-of-domain translation
knowledge to in-domain NMT model. In this paper, we argue that such a strategy
fails to fully extract the domain-shared translation knowledge, and repeatedly
utilizing corpora of different domains can lead to better distillation of
domain-shared translation knowledge. To this end, we propose an iterative dual
domain adaptation framework for NMT. Specifically, we first pre-train in-domain
and out-of-domain NMT models using their own training corpora respectively, and
then iteratively perform bidirectional translation knowledge transfer (from
in-domain to out-of-domain and then vice versa) based on knowledge distillation
until the in-domain NMT model convergences. Furthermore, we extend the proposed
framework to the scenario of multiple out-of-domain training corpora, where the
above-mentioned transfer is performed sequentially between the in-domain and
each out-of-domain NMT models in the ascending order of their domain
similarities. Empirical results on Chinese-English and English-German
translation tasks demonstrate the effectiveness of our framework.

negative:
Large, human-annotated datasets are central to the development of natural
language processing models. Collecting these datasets can be the most
challenging part of the development process. We address this problem by
introducing a general purpose technique for ``simulation-to-real'' transfer in
language understanding problems with a delimited set of target behaviors,
making it possible to develop models that can interpret natural utterances
without natural training data. We begin with a synthetic data generation
procedure, and train a model that can accurately interpret utterances produced
by the data generator. To generalize to natural utterances, we automatically
find projections of natural language utterances onto the support of the
synthetic language, using learned sentence embeddings to define a distance
metric. With only synthetic training data, our approach matches or outperforms
state-of-the-art models trained on natural language data in several domains.
These results suggest that simulation-to-real transfer is a practical framework
for developing NLP applications, and that improved models for transfer might
provide wide-ranging improvements in downstream tasks.

negative:
With a lot of work about context-free question answering systems, there is an
emerging trend of conversational question answering models in the natural
language processing field. Thanks to the recently collected datasets, including
QuAC and CoQA, there has been more work on conversational question answering,
and recent work has achieved competitive performance on both datasets. However,
to best of our knowledge, two important questions for conversational
comprehension research have not been well studied: 1) How well can the
benchmark dataset reflect models' content understanding? 2) Do the models well
utilize the conversation content when answering questions? To investigate these
questions, we design different training settings, testing settings, as well as
an attack to verify the models' capability of content understanding on QuAC and
CoQA. The experimental results indicate some potential hazards in the benchmark
datasets, QuAC and CoQA, for conversational comprehension research. Our
analysis also sheds light on both what models may learn and how datasets may
bias the models. With deep investigation of the task, it is believed that this
work can benefit the future progress of conversation comprehension. The source
code is available at https://github.com/MiuLab/CQA-Study.

negative:
Making accurate recommendations for cold-start users has been a longstanding
and critical challenge for recommender systems (RS). Cross-domain
recommendations (CDR) offer a solution to tackle such a cold-start problem when
there is no sufficient data for the users who have rarely used the system. An
effective approach in CDR is to leverage the knowledge (e.g., user
representations) learned from a related but different domain and transfer it to
the target domain. Fine-tuning works as an effective transfer learning
technique for this objective, which adapts the parameters of a pre-trained
model from the source domain to the target domain. However, current methods are
mainly based on the global fine-tuning strategy: the decision of which layers
of the pre-trained model to freeze or fine-tune is taken for all users in the
target domain. In this paper, we argue that users in RS are personalized and
should have their own fine-tuning policies for better preference transfer
learning. As such, we propose a novel User-specific Adaptive Fine-tuning method
(UAF), selecting which layers of the pre-trained network to fine-tune, on a
per-user basis. Specifically, we devise a policy network with three alternative
strategies to automatically decide which layers to be fine-tuned and which
layers to have their parameters frozen for each user. Extensive experiments
show that the proposed UAF exhibits significantly better and more robust
performance for user cold-start recommendation.

negative:
In this paper, we study transfer learning for the PI and NLI problems, aiming
to propose a general framework, which can effectively and efficiently adapt the
shared knowledge learned from a resource-rich source domain to a resource- poor
target domain. Specifically, since most existing transfer learning methods only
focus on learning a shared feature space across domains while ignoring the
relationship between the source and target domains, we propose to
simultaneously learn shared representations and domain relationships in a
unified framework. Furthermore, we propose an efficient and effective hybrid
model by combining a sentence encoding- based method and a sentence
interaction-based method as our base model. Extensive experiments on both
paraphrase identification and natural language inference demonstrate that our
base model is efficient and has promising performance compared to the competing
models, and our transfer learning method can help to significantly boost the
performance. Further analysis shows that the inter-domain and intra-domain
relationship captured by our model are insightful. Last but not least, we
deploy our transfer learning model for PI into our online chatbot system, which
can bring in significant improvements over our existing system. Finally, we
launch our new system on the chatbot platform Eva in our E-commerce site
AliExpress.

negative:
Answering questions on scholarly knowledge comprising text and other
artifacts is a vital part of any research life cycle. Querying scholarly
knowledge and retrieving suitable answers is currently hardly possible due to
the following primary reason: machine inactionable, ambiguous and unstructured
content in publications. We present JarvisQA, a BERT based system to answer
questions on tabular views of scholarly knowledge graphs. Such tables can be
found in a variety of shapes in the scholarly literature (e.g., surveys,
comparisons or results). Our system can retrieve direct answers to a variety of
different questions asked on tabular data in articles. Furthermore, we present
a preliminary dataset of related tables and a corresponding set of natural
language questions. This dataset is used as a benchmark for our system and can
be reused by others. Additionally, JarvisQA is evaluated on two datasets
against other baselines and shows an improvement of two to three folds in
performance compared to related methods.

negative:
Multi-modal pretraining for learning high-level multi-modal representation is
a further step towards deep learning and artificial intelligence. In this work,
we propose a novel model, namely InterBERT (BERT for Interaction), which is the
first model of our series of multimodal pretraining methods M6
(MultiModality-to-MultiModality Multitask Mega-transformer). The model owns
strong capability of modeling interaction between the information flows of
different modalities. The single-stream interaction module is capable of
effectively processing information of multiple modalilties, and the two-stream
module on top preserves the independence of each modality to avoid performance
downgrade in single-modal tasks. We pretrain the model with three pretraining
tasks, including masked segment modeling (MSM), masked region modeling (MRM)
and image-text matching (ITM); and finetune the model on a series of
vision-and-language downstream tasks. Experimental results demonstrate that
InterBERT outperforms a series of strong baselines, including the most recent
multi-modal pretraining methods, and the analysis shows that MSM and MRM are
effective for pretraining and our method can achieve performances comparable to
BERT in single-modal tasks. Besides, we propose a large-scale dataset for
multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which
is the first Chinese multi-modal pretrained model. We pretrain the Chinese
InterBERT on our proposed dataset of 3.1M image-text pairs from the mobile
Taobao, the largest Chinese e-commerce platform. We finetune the model for
text-based image retrieval, and recently we deployed the model online for
topic-based recommendation.

negative:
Many domain adaptation approaches rely on learning cross domain shared
representations to transfer the knowledge learned in one domain to other
domains. Traditional domain adaptation only considers adapting for one task. In
this paper, we explore multi-task representation learning under the domain
adaptation scenario. We propose a neural network framework that supports domain
adaptation for multiple tasks simultaneously, and learns shared representations
that better generalize for domain adaptation. We apply the proposed framework
to domain adaptation for sequence tagging problems considering two tasks:
Chinese word segmentation and named entity recognition. Experiments show that
multi-task domain adaptation works better than disjoint domain adaptation for
each task, and achieves the state-of-the-art results for both tasks in the
social media domain.

negative:
Various forums and question answering (Q&A) sites are available online that
allow Ubuntu users to find results similar to their queries. However, searching
for a result is often time consuming as it requires the user to find a specific
problem instance relevant to his/her query from a large set of questions. In
this paper, we present an automated question answering system for Ubuntu users
called Dr. Tux that is designed to answer user's queries by selecting the most
similar question from an online database. The prototype was implemented in
Python and uses NLTK and CoreNLP tools for Natural Language Processing. The
data for the prototype was taken from the AskUbuntu website which contains
about 150k questions. The results obtained from the manual evaluation of the
prototype were promising while also presenting some interesting opportunities
for improvement.

negative:
Neural network pretraining is gaining attention due to its outstanding
performance in natural language processing applications. However, pretraining
usually leverages predefined task sequences to learn general linguistic clues.
The lack of mechanisms in choosing proper tasks during pretraining makes the
learning and knowledge encoding inefficient. We thus propose using
meta-learning to select tasks that provide the most informative learning
signals in each episode of pretraining. With the proposed method, we aim to
achieve better efficiency in computation and memory usage for the pretraining
process and resulting networks while maintaining the performance. In this
preliminary work, we discuss the algorithm of the method and its two variants,
downstream-aware and downstream-agnostic pretraining. Our experiment plan is
also summarized, while empirical results will be shared in our future works.

negative:
In sequence labeling, previous domain adaptation methods focus on the
adaptation from the source domain to the entire target domain without
considering the diversity of individual target domain samples, which may lead
to negative transfer results for certain samples. Besides, an important
characteristic of sequence labeling tasks is that different elements within a
given sample may also have diverse domain relevance, which requires further
consideration. To take the multi-level domain relevance discrepancy into
account, in this paper, we propose a fine-grained knowledge fusion model with
the domain relevance modeling scheme to control the balance between learning
from the target domain data and learning from the source domain model.
Experiments on three sequence labeling tasks show that our fine-grained
knowledge fusion model outperforms strong baselines and other state-of-the-art
sequence labeling domain adaptation methods.

negative:
Supervised models of NLP rely on large collections of text which closely
resemble the intended testing setting. Unfortunately matching text is often not
available in sufficient quantity, and moreover, within any domain of text, data
is often highly heterogenous. In this paper we propose a method to distill the
important domain signal as part of a multi-domain learning system, using a
latent variable model in which parts of a neural model are stochastically gated
based on the inferred domain. We compare the use of discrete versus continuous
latent variables, operating in a domain-supervised or a domain semi-supervised
setting, where the domain is known only for a subset of training inputs. We
show that our model leads to substantial performance improvements over
competitive benchmark domain adaptation methods, including methods using
adversarial learning.

negative:
Named entity recognition (NER) is a fundamental task of natural language
processing (NLP). However, most state-of-the-art research is mainly oriented to
high-resource languages such as English and has not been widely applied to
low-resource languages. In Malay language, relevant NER resources are limited.
In this work, we propose a dataset construction framework, which is based on
labeled datasets of homologous languages and iterative optimization, to build a
Malay NER dataset (MYNER) comprising 28,991 sentences (over 384 thousand
tokens). Additionally, to better integrate boundary information for NER, we
propose a multi-task (MT) model with a bidirectional revision (Bi-revision)
mechanism for Malay NER task. Specifically, an auxiliary task, boundary
detection, is introduced to improve NER training in both explicit and implicit
ways. Furthermore, a gated ignoring mechanism is proposed to conduct
conditional label transfer and alleviate error propagation by the auxiliary
task. Experimental results demonstrate that our model achieves comparable
results over baselines on MYNER. The dataset and the model in this paper would
be publicly released as a benchmark dataset.

negative:
The volume of academic paper submissions and publications is growing at an
ever increasing rate. While this flood of research promises progress in various
fields, the sheer volume of output inherently increases the amount of noise. We
present a system to automatically separate papers with a high from those with a
low likelihood of gaining citations as a means to quickly find high impact,
high quality research. Our system uses both a visual classifier, useful for
surmising a document's overall appearance, and a text classifier, for making
content-informed decisions. Current work in the field focuses on small datasets
composed of papers from individual conferences. Attempts to use similar
techniques on larger datasets generally only considers excerpts of the
documents such as the abstract, potentially throwing away valuable data. We
rectify these issues by providing a dataset composed of PDF documents and
citation counts spanning a decade of output within two separate academic
domains: computer science and medicine. This new dataset allows us to expand on
current work in the field by generalizing across time and academic domain.
Moreover, we explore inter-domain prediction models - evaluating a classifier's
performance on a domain it was not trained on - to shed further insight on this
important problem.

negative:
We study the problem of domain adaptation for neural abstractive
summarization. We make initial efforts in investigating what information can be
transferred to a new domain. Experimental results on news stories and opinion
articles indicate that neural summarization model benefits from pre-training
based on extractive summaries. We also find that the combination of in-domain
and out-of-domain setup yields better summaries when in-domain data is
insufficient. Further analysis shows that, the model is capable to select
salient content even trained on out-of-domain data, but requires in-domain data
to capture the style for a target domain.

negative:
Wikipedia is the largest online encyclopedia, used by algorithms and web
users as a central hub of reliable information on the web. The quality and
reliability of Wikipedia content is maintained by a community of volunteer
editors. Machine learning and information retrieval algorithms could help scale
up editors' manual efforts around Wikipedia content reliability. However, there
is a lack of large-scale data to support the development of such research. To
fill this gap, in this paper, we propose Wiki-Reliability, the first dataset of
English Wikipedia articles annotated with a wide set of content reliability
issues. To build this dataset, we rely on Wikipedia "templates". Templates are
tags used by expert Wikipedia editors to indicate content issues, such as the
presence of "non-neutral point of view" or "contradictory articles", and serve
as a strong signal for detecting reliability issues in a revision. We select
the 10 most popular reliability-related templates on Wikipedia, and propose an
effective method to label almost 1M samples of Wikipedia article revisions as
positive or negative with respect to each template. Each positive/negative
example in the dataset comes with the full article text and 20 features from
the revision's metadata. We provide an overview of the possible downstream
tasks enabled by such data, and show that Wiki-Reliability can be used to train
large-scale models for content reliability prediction. We release all data and
code for public use.

negative:
As open-ended human-chatbot interaction becomes commonplace, sensitive
content detection gains importance. In this work, we propose a two stage
semi-supervised approach to bootstrap large-scale data for automatic sensitive
language detection from publicly available web resources. We explore various
data selection methods including 1) using a blacklist to rank online discussion
forums by the level of their sensitiveness followed by randomly sampling
utterances and 2) training a weakly supervised model in conjunction with the
blacklist for scoring sentences from online discussion forums to curate a
dataset. Our data collection strategy is flexible and allows the models to
detect implicit sensitive content for which manual annotations may be
difficult. We train models using publicly available annotated datasets as well
as using the proposed large-scale semi-supervised datasets. We evaluate the
performance of all the models on Twitter and Toxic Wikipedia comments testsets
as well as on a manually annotated spoken language dataset collected during a
large scale chatbot competition. Results show that a model trained on this
collected data outperforms the baseline models by a large margin on both
in-domain and out-of-domain testsets, achieving an F1 score of 95.5% on an
out-of-domain testset compared to a score of 75% for models trained on public
datasets. We also showcase that large scale two stage semi-supervision
generalizes well across multiple classes of sensitivities such as hate speech,
racism, sexual and pornographic content, etc. without even providing explicit
labels for these classes, leading to an average recall of 95.5% versus the
models trained using annotated public datasets which achieve an average recall
of 73.2% across seven sensitive classes on out-of-domain testsets.

negative:
Large datasets are essential for neural modeling of many NLP tasks. Current
publicly available open-domain dialogue datasets offer a trade-off between
quality (e.g., DailyDialog) and size (e.g., Opensubtitles). We narrow this gap
by building a high-quality dataset of 14.8M utterances in English, and smaller
datasets in German, Dutch, Spanish, Portuguese, Italian, and Hungarian. We
extract and process dialogues from public-domain books made available by
Project Gutenberg. We describe our dialogue extraction pipeline, analyze the
effects of the various heuristics used, and present an error analysis of
extracted dialogues. Finally, we conduct experiments showing that better
response quality can be achieved in zero-shot and finetuning settings by
training on our data than on the larger but much noisier Opensubtitles dataset.
Our open-source pipeline (https://github.com/ricsinaruto/gutenberg-dialog) can
be extended to further languages with little additional effort. Researchers can
also build their versions of existing datasets by adjusting various trade-off
parameters. We also built a web demo for interacting with our models:
https://ricsinaruto.github.io/chatbot.html.

negative:
We propose a new architecture for adapting a sentence-level
sequence-to-sequence transformer by incorporating multiple pretrained document
context signals and assess the impact on translation performance of (1)
different pretraining approaches for generating these signals, (2) the quantity
of parallel data for which document context is available, and (3) conditioning
on source, target, or source and target contexts. Experiments on the NIST
Chinese-English, and IWSLT and WMT English-German tasks support four general
conclusions: that using pretrained context representations markedly improves
sample efficiency, that adequate parallel data resources are crucial for
learning to use document context, that jointly conditioning on multiple context
representations outperforms any single representation, and that source context
is more valuable for translation performance than target side context. Our best
multi-context model consistently outperforms the best existing context-aware
transformers.

negative:
Using chatbots to deliver recommendations is increasingly popular. The design
of recommendation chatbots has primarily been taking an information-centric
approach by focusing on the recommended content per se. Limited attention is on
how social connection and relational strategies, such as self-disclosure from a
chatbot, may influence users' perception and acceptance of the recommendation.
In this work, we designed, implemented, and evaluated a social chatbot capable
of performing three different levels of self-disclosure: factual information
(low), cognitive opinions (medium), and emotions (high). In the evaluation, we
recruited 372 participants to converse with the chatbot on two topics: movies
and COVID-19 experiences. In each topic, the chatbot performed small talks and
made recommendations relevant to the topic. Participants were randomly assigned
to four experimental conditions where the chatbot used factual, cognitive,
emotional, and adaptive strategies to perform self-disclosures. By training a
text classifier to identify users' level of self-disclosure in real-time, the
adaptive chatbot can dynamically match its self-disclosure to the level of
disclosure exhibited by the users. Our results show that users reciprocate with
higher-level self-disclosure when a recommendation chatbot consistently
displays emotions throughout the conversation. Chatbot's emotional disclosure
also led to increased interactional enjoyment and more positive interpersonal
perception towards the bot, fostering a stronger human-chatbot relationship and
thus leading to increased recommendation effectiveness, including a higher
tendency to accept the recommendation. We discuss the understandings obtained
and implications to future design.

negative:
General purpose Search Engines (SEs) crawl all domains (e.g., Sports, News,
Entertainment) of the Web, but sometimes the informational need of a query is
restricted to a particular domain (e.g., Medical). We leverage the work of SEs
as part of our effort to route domain specific queries to local Digital
Libraries (DLs). SEs are often used even if they are not the "best" source for
certain types of queries. Rather than tell users to "use this DL for this kind
of query", we intend to automatically detect when a query could be better
served by a local DL (such as a private, access-controlled DL that is not
crawlable via SEs). This is not an easy task because Web queries are short,
ambiguous, and there is lack of quality labeled training data (or it is
expensive to create). To detect queries that should be routed to local,
specialized DLs, we first send the queries to Google and then examine the
features in the resulting Search Engine Result Pages (SERPs), and then classify
the query as belonging to either the scholar or non-scholar domain. Using
400,000 AOL queries for the non-scholar domain and 400,000 queries from the
NASA Technical Report Server (NTRS) for the scholar domain, our classifier
achieved a precision of 0.809 and F-measure of 0.805.

negative:
We investigate neural techniques for end-to-end computational argumentation
mining (AM). We frame AM both as a token-based dependency parsing and as a
token-based sequence tagging problem, including a multi-task learning setup.
Contrary to models that operate on the argument component level, we find that
framing AM as dependency parsing leads to subpar performance results. In
contrast, less complex (local) tagging models based on BiLSTMs perform robustly
across classification scenarios, being able to catch long-range dependencies
inherent to the AM problem. Moreover, we find that jointly learning 'natural'
subtasks, in a multi-task learning setup, improves performance.

negative:
This paper examines various unsupervised pretraining objectives for learning
dialog context representations. Two novel methods of pretraining dialog context
encoders are proposed, and a total of four methods are examined. Each
pretraining objective is fine-tuned and evaluated on a set of downstream dialog
tasks using the MultiWoz dataset and strong performance improvement is
observed. Further evaluation shows that our pretraining objectives result in
not only better performance, but also better convergence, models that are less
data hungry and have better domain generalizability.

negative:
Creating open-domain chatbots requires large amounts of conversational data
and related benchmark tasks to evaluate them. Standardized evaluation tasks are
crucial for creating automatic evaluation metrics for model development;
otherwise, comparing the models would require resource-expensive human
evaluation. While chatbot challenges have recently managed to provide a
plethora of such resources for English, resources in other languages are not
yet available. In this work, we provide a starting point for Finnish
open-domain chatbot research. We describe our collection efforts to create the
Finnish chat conversation corpus FinChat, which is made available publicly.
FinChat includes unscripted conversations on seven topics from people of
different ages. Using this corpus, we also construct a retrieval-based
evaluation task for Finnish chatbot development. We observe that off-the-shelf
chatbot models trained on conversational corpora do not perform better than
chance at choosing the right answer based on automatic metrics, while humans
can do the same task almost perfectly. Similarly, in a human evaluation,
responses to questions from the evaluation set generated by the chatbots are
predominantly marked as incoherent. Thus, FinChat provides a challenging
evaluation set, meant to encourage chatbot development in Finnish.

negative:
We propose a simple method for automatic speech recognition (ASR) by
fine-tuning BERT, which is a language model (LM) trained on large-scale
unlabeled text data and can generate rich contextual representations. Our
assumption is that given a history context sequence, a powerful LM can narrow
the range of possible choices and the speech signal can be used as a simple
clue. Hence, comparing to conventional ASR systems that train a powerful
acoustic model (AM) from scratch, we believe that speech recognition is
possible by simply fine-tuning a BERT model. As an initial study, we
demonstrate the effectiveness of the proposed idea on the AISHELL dataset and
show that stacking a very simple AM on top of BERT can yield reasonable
performance.

negative:
In this paper, we explore the ways to improve POS-tagging using various types
of auxiliary losses and different word representations. As a baseline, we
utilized a BiLSTM tagger, which is able to achieve state-of-the-art results on
the sequence labelling tasks. We developed a new method for character-level
word representation using feedforward neural network. Such representation gave
us better results in terms of speed and performance of the model. We also
applied a novel technique of pretraining such word representations with
existing word vectors. Finally, we designed a new variant of auxiliary loss for
sequence labelling tasks: an additional prediction of the neighbour labels.
Such loss forces a model to learn the dependencies in-side a sequence of labels
and accelerates the process of training. We test these methods on English and
Russian languages.

negative:
In this paper, we propose two novel methods for domain adaptation for the
attention-only neural machine translation (NMT) model, i.e., the Transformer.
Our methods focus on training a single translation model for multiple domains
by either learning domain specialized hidden state representations or predictor
biases for each domain. We combine our methods with a previously proposed
black-box method called mixed fine tuning, which is known to be highly
effective for domain adaptation. In addition, we incorporate multilingualism
into the domain adaptation framework. Experiments show that multilingual
multi-domain adaptation can significantly improve both resource-poor in-domain
and resource-rich out-of-domain translations, and the combination of our
methods with mixed fine tuning achieves the best performance.

negative:
Pretraining Neural Language Models (NLMs) over a large corpus involves
chunking the text into training examples, which are contiguous text segments of
sizes processable by the neural architecture. We highlight a bias introduced by
this common practice: we prove that the pretrained NLM can model much stronger
dependencies between text segments that appeared in the same training example,
than it can between text segments that appeared in different training examples.
This intuitive result has a twofold role. First, it formalizes the motivation
behind a broad line of recent successful NLM training heuristics, proposed for
the pretraining and fine-tuning stages, which do not necessarily appear related
at first glance. Second, our result clearly indicates further improvements to
be made in NLM pretraining for the benefit of Natural Language Understanding
tasks. As an example, we propose "kNN-Pretraining": we show that including
semantically related non-neighboring sentences in the same pretraining example
yields improved sentence representations and open domain question answering
abilities. This theoretically motivated degree of freedom for pretraining
example design indicates new training schemes for self-improving
representations.

negative:
In this work, we propose BertGCN, a model that combines large scale
pretraining and transductive learning for text classification. BertGCN
constructs a heterogeneous graph over the dataset and represents documents as
nodes using BERT representations. By jointly training the BERT and GCN modules
within BertGCN, the proposed model is able to leverage the advantages of both
worlds: large-scale pretraining which takes the advantage of the massive amount
of raw data and transductive learning which jointly learns representations for
both training data and unlabeled test data by propagating label influence
through graph convolution. Experiments show that BertGCN achieves SOTA
performances on a wide range of text classification datasets. Code is available
at https://github.com/ZeroRin/BertGCN.

negative:
Recent work using auxiliary prediction task classifiers to investigate the
properties of LSTM representations has begun to shed light on why pretrained
representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al.,
2017), are so beneficial for neural language understanding models. We still,
though, do not yet have a clear understanding of how the choice of pretraining
objective affects the type of linguistic information that models learn. With
this in mind, we compare four objectives---language modeling, translation,
skip-thought, and autoencoding---on their ability to induce syntactic and
part-of-speech information. We make a fair comparison between the tasks by
holding constant the quantity and genre of the training data, as well as the
LSTM architecture. We find that representations from language models
consistently perform best on our syntactic auxiliary prediction tasks, even
when trained on relatively small amounts of data. These results suggest that
language modeling may be the best data-rich pretraining task for transfer
learning applications requiring syntactic information. We also find that the
representations from randomly-initialized, frozen LSTMs perform strikingly well
on our syntactic auxiliary tasks, but this effect disappears when the amount of
training data for the auxiliary tasks is reduced.

negative:
We present a CLSRIL-23, a self supervised learning based audio pre-trained
model which learns cross lingual speech representations from raw audio across
23 Indic languages. It is built on top of wav2vec 2.0 which is solved by
training a contrastive task over masked latent speech representations and
jointly learns the quantization of latents shared across all languages. We
compare the language wise loss during pretraining to compare effects of
monolingual and multilingual pretraining. Performance on some downstream
fine-tuning tasks for speech recognition is also compared and our experiments
show that multilingual pretraining outperforms monolingual training, in terms
of learning speech representations which encodes phonetic similarity of
languages and also in terms of performance on down stream tasks. A decrease of
5% is observed in WER and 9.5% in CER when a multilingual pretrained model is
used for finetuning in Hindi. All the code models are also open sourced.
CLSRIL-23 is a model trained on $23$ languages and almost 10,000 hours of audio
data to facilitate research in speech recognition for Indic languages. We hope
that new state of the art systems will be created using the self supervised
approach, especially for low resources Indic languages.

negative:
State-of-the-art speech recognition systems rely heavily on three basic
components: an acoustic model, a pronunciation lexicon and a language model. To
build these components, a researcher needs linguistic as well as technical
expertise, which is a barrier in low-resource domains. Techniques to construct
these three components without having expert domain knowledge are in great
demand. Urdu, despite having millions of speakers all over the world, is a
low-resource language in terms of standard publically available linguistic
resources. In this paper, we present a grapheme-to-phoneme conversion tool for
Urdu that generates a pronunciation lexicon in a form suitable for use with
speech recognition systems from a list of Urdu words. The tool predicts the
pronunciation of words using a LSTM-based model trained on a handcrafted expert
lexicon of around 39,000 words and shows an accuracy of 64% upon internal
evaluation. For external evaluation on a speech recognition task, we obtain a
word error rate comparable to one achieved using a fully handcrafted expert
lexicon.

negative:
Self-supervised pretraining for Automated Speech Recognition (ASR) has shown
varied degrees of success. In this paper, we propose to jointly learn
representations during pretraining from two different modalities: speech and
text. The proposed method, tts4pretrain complements the power of contrastive
learning in self-supervision with linguistic/lexical representations derived
from synthesized speech, effectively learning from untranscribed speech and
unspoken text. Lexical learning in the speech encoder is enforced through an
additional sequence loss term that is coupled with contrastive loss during
pretraining. We demonstrate that this novel pretraining method yields Word
Error Rate (WER) reductions of 10% relative on the well-benchmarked,
Librispeech task over a state-of-the-art baseline pretrained with wav2vec2.0
only. The proposed method also serves as an effective strategy to compensate
for the lack of transcribed speech, effectively matching the performance of
5000 hours of transcribed speech with just 100 hours of transcribed speech on
the AMI meeting transcription task. Finally, we demonstrate WER reductions of
up to 15% on an in-house Voice Search task over traditional pretraining.
Incorporating text into encoder pretraining is complimentary to rescoring with
a larger or in-domain language model, resulting in additional 6% relative
reduction in WER.

negative:
The continuous growth of scientific literature brings innovations and, at the
same time, raises new challenges. One of them is related to the fact that its
analysis has become difficult due to the high volume of published papers for
which manual effort for annotations and management is required. Novel
technological infrastructures are needed to help researchers, research policy
makers, and companies to time-efficiently browse, analyse, and forecast
scientific research. Knowledge graphs i.e., large networks of entities and
relationships, have proved to be effective solution in this space. Scientific
knowledge graphs focus on the scholarly domain and typically contain metadata
describing research publications such as authors, venues, organizations,
research topics, and citations. However, the current generation of knowledge
graphs lacks of an explicit representation of the knowledge presented in the
research papers. As such, in this paper, we present a new architecture that
takes advantage of Natural Language Processing and Machine Learning methods for
extracting entities and relationships from research publications and integrates
them in a large-scale knowledge graph. Within this research work, we i) tackle
the challenge of knowledge extraction by employing several state-of-the-art
Natural Language Processing and Text Mining tools, ii) describe an approach for
integrating entities and relationships generated by these tools, iii) show the
advantage of such an hybrid system over alternative approaches, and vi) as a
chosen use case, we generated a scientific knowledge graph including 109,105
triples, extracted from 26,827 abstracts of papers within the Semantic Web
domain. As our approach is general and can be applied to any domain, we expect
that it can facilitate the management, analysis, dissemination, and processing
of scientific knowledge.

negative:
As the wide adoption of intelligent chatbot in human daily life, user demands
for such systems evolve from basic task-solving conversations to more casual
and friend-like communication. To meet the user needs and build emotional bond
with users, it is essential for social chatbots to incorporate more human-like
and advanced linguistic features. In this paper, we investigate the usage of a
commonly used rhetorical device by human -- metaphor for social chatbot. Our
work first designs a metaphor generation framework, which generates topic-aware
and novel figurative sentences. By embedding the framework into a chatbot
system, we then enables the chatbot to communicate with users using figurative
language. Human annotators validate the novelty and properness of the generated
metaphors. More importantly, we evaluate the effects of employing metaphors in
human-chatbot conversations. Experiments indicate that our system effectively
arouses user interests in communicating with our chatbot, resulting in
significantly longer human-chatbot conversations.

negative:
Email responses often contain items-such as a file or a hyperlink to an
external document-that are attached to or included inline in the body of the
message. Analysis of an enterprise email corpus reveals that 35% of the time
when users include these items as part of their response, the attachable item
is already present in their inbox or sent folder. A modern email client can
proactively retrieve relevant attachable items from the user's past emails
based on the context of the current conversation, and recommend them for
inclusion, to reduce the time and effort involved in composing the response. In
this paper, we propose a weakly supervised learning framework for recommending
attachable items to the user. As email search systems are commonly available,
we constrain the recommendation task to formulating effective search queries
from the context of the conversations. The query is submitted to an existing IR
system to retrieve relevant items for attachment. We also present a novel
strategy for generating labels from an email corpus---without the need for
manual annotations---that can be used to train and evaluate the query
formulation model. In addition, we describe a deep convolutional neural network
that demonstrates satisfactory performance on this query formulation task when
evaluated on the publicly available Avocado dataset and a proprietary dataset
of internal emails obtained through an employee participation program.

negative:
Contextual word representations derived from large-scale neural language
models are successful across a diverse set of NLP tasks, suggesting that they
encode useful and transferable features of language. To shed light on the
linguistic knowledge they capture, we study the representations produced by
several recent pretrained contextualizers (variants of ELMo, the OpenAI
transformer language model, and BERT) with a suite of seventeen diverse probing
tasks. We find that linear models trained on top of frozen contextual
representations are competitive with state-of-the-art task-specific models in
many cases, but fail on tasks requiring fine-grained linguistic knowledge
(e.g., conjunct identification). To investigate the transferability of
contextual word representations, we quantify differences in the transferability
of individual layers within contextualizers, especially between recurrent
neural networks (RNNs) and transformers. For instance, higher layers of RNNs
are more task-specific, while transformer layers do not exhibit the same
monotonic trend. In addition, to better understand what makes contextual word
representations transferable, we compare language model pretraining with eleven
supervised pretraining tasks. For any given task, pretraining on a closely
related task yields better performance than language model pretraining (which
is better on average) when the pretraining dataset is fixed. However, language
model pretraining on more data gives the best results.

negative:
This work explores the application of textual entailment in news claim
verification and stance prediction using a new corpus in Arabic. The publicly
available corpus comes in two perspectives: a version consisting of 4,547 true
and false claims and a version consisting of 3,786 pairs (claim, evidence). We
describe the methodology for creating the corpus and the annotation process.
Using the introduced corpus, we also develop two machine learning baselines for
two proposed tasks: claim verification and stance prediction. Our best model
utilizes pretraining (BERT) and achieves 76.7 F1 on the stance prediction task
and 64.3 F1 on the claim verification task. Our preliminary experiments shed
some light on the limits of automatic claim verification that relies on claims
text only. Results hint that while the linguistic features and world knowledge
learned during pretraining are useful for stance prediction, such learned
representations from pretraining are insufficient for verifying claims without
access to context or evidence.

negative:
We introduce a set of nine challenge tasks that test for the understanding of
function words. These tasks are created by structurally mutating sentences from
existing datasets to target the comprehension of specific types of function
words (e.g., prepositions, wh-words). Using these probing tasks, we explore the
effects of various pretraining objectives for sentence encoders (e.g., language
modeling, CCG supertagging and natural language inference (NLI)) on the learned
representations. Our results show that pretraining on language modeling
performs the best on average across our probing tasks, supporting its
widespread use for pretraining state-of-the-art NLP models, and CCG
supertagging and NLI pretraining perform comparably. Overall, no pretraining
objective dominates across the board, and our function word probing tasks
highlight several intuitive differences between pretraining objectives, e.g.,
that NLI helps the comprehension of negation.

negative:
Language Models like ELMo and BERT have provided robust representations of
natural language, which serve as the language understanding component for a
diverse range of downstream tasks.Curriculum learning is a method that employs
a structured training regime instead, which has been leveraged in computer
vision and machine translation to improve model training speed and model
performance. While language models have proven transformational for the natural
language processing community, these models have proven expensive,
energy-intensive, and challenging to train. In this work, we explore the effect
of curriculum learning on language model pretraining using various
linguistically motivated curricula and evaluate transfer performance on the
GLUE Benchmark. Despite a broad variety of training methodologies and
experiments we do not find compelling evidence that curriculum learning methods
improve language model training.

negative:
Users of machine translation (MT) may want to ensure the use of specific
lexical terminologies. While there exist techniques for incorporating
terminology constraints during inference for MT, current APE approaches cannot
ensure that they will appear in the final translation. In this paper, we
present both autoregressive and non-autoregressive models for lexically
constrained APE, demonstrating that our approach enables preservation of 95% of
the terminologies and also improves translation quality on English-German
benchmarks. Even when applied to lexically constrained MT output, our approach
is able to improve preservation of the terminologies. However, we show that our
models do not learn to copy constraints systematically and suggest a simple
data augmentation technique that leads to improved performance and robustness.