query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I aim to assist actors in better comprehending their roles in movies or TV shows by creating a system that can comprehend and analyze the mood and personality of characters within a narrative context. My plan is to utilize transformer-based models in conjunction with a reinforcement learning (RL) framework to accomplish this. The transformer-based model should be capable of inferring from the context and dialogue to predict a character's potential mood and anticipate their next line. The RL framework will then compare the predicted conversation with the actual conversation in the reward formula to provide weak self-supervision during the training process. I am aware of the concept of a warm start in training, but I am unsure how to adapt this to my current project. Furthermore, the target narrative contexts, such as scripts, differ from typical texts, so I would appreciate advice on the appropriate length for the context window in this type of text. Finally, the model should be able to map mood or personality to facial expressions or character appearances to provide guidance to the actors.

positive:
Recent neural models of dialogue generation offer great promise for
generating responses for conversational agents, but tend to be shortsighted,
predicting utterances one at a time while ignoring their influence on future
outcomes. Modeling the future direction of a dialogue is crucial to generating
coherent, interesting dialogues, a need which led traditional NLP models of
dialogue to draw on reinforcement learning. In this paper, we show how to
integrate these goals, applying deep reinforcement learning to model future
reward in chatbot dialogue. The model simulates dialogues between two virtual
agents, using policy gradient methods to reward sequences that display three
useful conversational properties: informativity (non-repetitive turns),
coherence, and ease of answering (related to forward-looking function). We
evaluate our model on diversity, length as well as with human judges, showing
that the proposed algorithm generates more interactive responses and manages to
foster a more sustained conversation in dialogue simulation. This work marks a
first step towards learning a neural conversational model based on the
long-term success of dialogues.

positive:
A conversational agent (chatbot) is a piece of software that is able to
communicate with humans using natural language. Modeling conversation is an
important task in natural language processing and artificial intelligence.
While chatbots can be used for various tasks, in general they have to
understand users' utterances and provide responses that are relevant to the
problem at hand.
  In my work, I conduct an in-depth survey of recent literature, examining over
70 publications related to chatbots published in the last 3 years. Then, I
proceed to make the argument that the very nature of the general conversation
domain demands approaches that are different from current state-of-of-the-art
architectures. Based on several examples from the literature I show why current
chatbot models fail to take into account enough priors when generating
responses and how this affects the quality of the conversation. In the case of
chatbots, these priors can be outside sources of information that the
conversation is conditioned on like the persona or mood of the conversers. In
addition to presenting the reasons behind this problem, I propose several ideas
on how it could be remedied.
  The next section focuses on adapting the very recent Transformer model to the
chatbot domain, which is currently state-of-the-art in neural machine
translation. I first present experiments with the vanilla model, using
conversations extracted from the Cornell Movie-Dialog Corpus. Secondly, I
augment the model with some of my ideas regarding the issues of encoder-decoder
architectures. More specifically, I feed additional features into the model
like mood or persona together with the raw conversation data. Finally, I
conduct a detailed analysis of how the vanilla model performs on conversational
data by comparing it to previous chatbot models and how the additional features
affect the quality of the generated responses.

positive:
Neural models for response generation produce responses that are semantically
plausible but not necessarily factually consistent with facts describing the
speaker's persona. These models are trained with fully supervised learning
where the objective function barely captures factual consistency. We propose to
fine-tune these models by reinforcement learning and an efficient reward
function that explicitly captures the consistency between a response and
persona facts as well as semantic plausibility. Our automatic and human
evaluations on the PersonaChat corpus confirm that our approach increases the
rate of responses that are factually consistent with persona facts over its
supervised counterpart while retaining the language quality of responses.

negative:
Standard neural machine translation (NMT) is on the assumption of
document-level context independent. Most existing document-level NMT methods
are satisfied with a smattering sense of brief document-level information,
while this work focuses on exploiting detailed document-level context in terms
of multiple forms of document embeddings, which is capable of sufficiently
modeling deeper and richer document-level context. The proposed document-aware
NMT is implemented to enhance the Transformer baseline by introducing both
global and local document-level clues on the source end. Experiments show that
the proposed method significantly improves the translation performance over
strong baselines and other related studies.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
The successful emotional conversation system depends on sufficient perception
and appropriate expression of emotions. In a real-life conversation, humans
firstly instinctively perceive emotions from multi-source information,
including the emotion flow hidden in dialogue history, facial expressions,
audio, and personalities of speakers. Then, they convey suitable emotions
according to their personalities, but these multiple types of information are
insufficiently exploited in emotional conversation fields. To address this
issue, in this paper, we propose a heterogeneous graph-based model for
emotional conversation generation. Firstly, we design a Heterogeneous
Graph-Based Encoder to represent the conversation content (i.e., the dialogue
history, its emotion flow, facial expressions, audio, and speakers'
personalities) with a heterogeneous graph neural network, and then predict
suitable emotions for feedback. Secondly, we employ an
Emotion-Personality-Aware Decoder to generate a response relevant to the
conversation context as well as with appropriate emotions, through taking the
encoded graph representations, the predicted emotions by the encoder and the
personality of the current speaker as inputs. Experiments on both automatic and
human evaluation show that our method can effectively perceive emotions from
multi-source knowledge and generate a satisfactory response. Furthermore, based
on the up-to-date text generator BART, our model still can achieve consistent
improvement, which significantly outperforms some existing state-of-the-art
models.

negative:
Neural networks have recently become good at engaging in dialog. However,
current approaches are based solely on verbal text, lacking the richness of a
real face-to-face conversation. We propose a neural conversation model that
aims to read and generate facial gestures alongside with text. This allows our
model to adapt its response based on the "mood" of the conversation. In
particular, we introduce an RNN encoder-decoder that exploits the movement of
facial muscles, as well as the verbal conversation. The decoder consists of two
layers, where the lower layer aims at generating the verbal response and coarse
facial expressions, while the second layer fills in the subtle gestures, making
the generated output more smooth and natural. We train our neural network by
having it "watch" 250 movies. We showcase our joint face-text model in
generating more natural conversations through automatic metrics and a human
study. We demonstrate an example application with a face-to-face chatting
avatar.

negative:
An important aspect of human conversation difficult for machines is
conversing with empathy, which is to understand the user's emotion and respond
appropriately. Recent neural conversation models that attempted to generate
empathetic responses either focused on conditioning the output to a given
emotion, or incorporating the current user emotional state. However, these
approaches do not factor in how the user would feel towards the generated
response. Hence, in this paper, we propose Sentiment Look-ahead, which is a
novel perspective for empathy that models the future user emotional state. In
short, Sentiment Look-ahead is a reward function under a reinforcement learning
framework that provides a higher reward to the generative model when the
generated utterance improves the user's sentiment. We implement and evaluate
three different possible implementations of sentiment look-ahead and
empirically show that our proposed approach can generate significantly more
empathetic, relevant, and fluent responses than other competitive baselines
such as multitask learning.

negative:
People speak at different levels of specificity in different situations.
Depending on their knowledge, interlocutors, mood, etc.} A conversational agent
should have this ability and know when to be specific and when to be general.
We propose an approach that gives a neural network--based conversational agent
this ability. Our approach involves alternating between \emph{data
distillation} and model training : removing training examples that are closest
to the responses most commonly produced by the model trained from the last
round and then retrain the model on the remaining dataset. Dialogue generation
models trained with different degrees of data distillation manifest different
levels of specificity.
  We then train a reinforcement learning system for selecting among this pool
of generation models, to choose the best level of specificity for a given
input. Compared to the original generative model trained without distillation,
the proposed system is capable of generating more interesting and
higher-quality responses, in addition to appropriately adjusting specificity
depending on the context.
  Our research constitutes a specific case of a broader approach involving
training multiple subsystems from a single dataset distinguished by differences
in a specific property one wishes to model. We show that from such a set of
subsystems, one can use reinforcement learning to build a system that tailors
its output to different input contexts at test time.

negative:
To provide consistent emotional interaction with users, dialog systems should
be capable to automatically select appropriate emotions for responses like
humans. However, most existing works focus on rendering specified emotions in
responses or empathetically respond to the emotion of users, yet the individual
difference in emotion expression is overlooked. This may lead to inconsistent
emotional expressions and disinterest users. To tackle this issue, we propose
to equip the dialog system with personality and enable it to automatically
select emotions in responses by simulating the emotion transition of humans in
conversation. In detail, the emotion of the dialog system is transitioned from
its preceding emotion in context. The transition is triggered by the preceding
dialog context and affected by the specified personality trait. To achieve
this, we first model the emotion transition in the dialog system as the
variation between the preceding emotion and the response emotion in the
Valence-Arousal-Dominance (VAD) emotion space. Then, we design neural networks
to encode the preceding dialog context and the specified personality traits to
compose the variation. Finally, the emotion for response is selected from the
sum of the preceding emotion and the variation. We construct a dialog dataset
with emotion and personality labels and conduct emotion prediction tasks for
evaluation. Experimental results validate the effectiveness of the
personality-affected emotion transition.

negative:
For conversational AI and virtual assistants to communicate with humans in a
realistic way, they must exhibit human characteristics such as expression of
emotion and personality. Current attempts toward constructing human-like
dialogue agents have presented significant difficulties. We propose Human Level
Attributes (HLAs) based on tropes as the basis of a method for learning
dialogue agents that can imitate the personalities of fictional characters.
Tropes are characteristics of fictional personalities that are observed
recurrently and determined by viewers' impressions. By combining detailed HLA
data with dialogue data for specific characters, we present a dataset,
HLA-Chat, that models character profiles and gives dialogue agents the ability
to learn characters' language styles through their HLAs. We then introduce a
three-component system, ALOHA (which stands for Artificial Learning of Human
Attributes), that combines character space mapping, character community
detection, and language style retrieval to build a character (or personality)
specific language model. Our preliminary experiments demonstrate that two
variations of ALOHA, combined with our proposed dataset, can outperform
baseline models at identifying the correct dialogue responses of chosen target
characters, and are stable regardless of the character's identity, the genre of
the show, and the context of the dialogue.

negative:
Language systems have been of great interest to the research community and
have recently reached the mass market through various assistant platforms on
the web. Reinforcement Learning methods that optimize dialogue policies have
seen successes in past years and have recently been extended into methods that
personalize the dialogue, e.g. take the personal context of users into account.
These works, however, are limited to personalization to a single user with whom
they require multiple interactions and do not generalize the usage of context
across users. This work introduces a problem where a generalized usage of
context is relevant and proposes two Reinforcement Learning (RL)-based
approaches to this problem. The first approach uses a single learner and
extends the traditional POMDP formulation of dialogue state with features that
describe the user context. The second approach segments users by context and
then employs a learner per context. We compare these approaches in a benchmark
of existing non-RL and RL-based methods in three established and one novel
application domain of financial product recommendation. We compare the
influence of context and training experiences on performance and find that
learning approaches generally outperform a handcrafted gold standard.

negative:
Understanding narrative text requires capturing characters' motivations,
goals, and mental states. This paper proposes an Entity-based Narrative Graph
(ENG) to model the internal-states of characters in a story. We explicitly
model entities, their interactions and the context in which they appear, and
learn rich representations for them. We experiment with different task-adaptive
pre-training objectives, in-domain training, and symbolic inference to capture
dependencies between different decisions in the output space. We evaluate our
model on two narrative understanding tasks: predicting character mental states,
and desire fulfillment, and conduct a qualitative analysis.

negative:
Open-domain dialog generation is a challenging problem; maximum likelihood
training can lead to repetitive outputs, models have difficulty tracking
long-term conversational goals, and training on standard movie or online
datasets may lead to the generation of inappropriate, biased, or offensive
text. Reinforcement Learning (RL) is a powerful framework that could
potentially address these issues, for example by allowing a dialog model to
optimize for reducing toxicity and repetitiveness. However, previous approaches
which apply RL to open-domain dialog generation do so at the word level, making
it difficult for the model to learn proper credit assignment for long-term
conversational rewards. In this paper, we propose a novel approach to
hierarchical reinforcement learning, VHRL, which uses policy gradients to tune
the utterance-level embedding of a variational sequence model. This
hierarchical approach provides greater flexibility for learning long-term,
conversational rewards. We use self-play and RL to optimize for a set of
human-centered conversation metrics, and show that our approach provides
significant improvements -- in terms of both human evaluation and automatic
metrics -- over state-of-the-art dialog models, including Transformers.

negative:
The use of virtual agents in social coaching has increased rapidly in the
last decade. In order to train the user in different situations than can occur
in real life, the virtual agent should be able to express different social
attitudes. In this paper, we propose a model of social attitudes that enables a
virtual agent to reason on the appropriate social attitude to express during
the interaction with a user given the course of the interaction, but also the
emotions, mood and personality of the agent. Moreover, the model enables the
virtual agent to display its social attitude through its non-verbal behaviour.
The proposed model has been developed in the context of job interview
simulation. The methodology used to develop such a model combined a theoretical
and an empirical approach. Indeed, the model is based both on the literature in
Human and Social Sciences on social attitudes but also on the analysis of an
audiovisual corpus of job interviews and on post-hoc interviews with the
recruiters on their expressed attitudes during the job interview.

negative:
Explainable AI techniques that describe agent reward functions can enhance
human-robot collaboration in a variety of settings. One context where human
understanding of agent reward functions is particularly beneficial is in the
value alignment setting. In the value alignment context, an agent aims to infer
a human's reward function through interaction so that it can assist the human
with their tasks. If the human can understand where gaps exist in the agent's
reward understanding, they will be able to teach more efficiently and
effectively, leading to quicker human-agent team performance improvements. In
order to support human collaborators in the value alignment setting and similar
contexts, it is first important to understand the effectiveness of different
reward explanation techniques in a variety of domains. In this paper, we
introduce a categorization of information modalities for reward explanation
techniques, suggest a suite of assessment techniques for human reward
understanding, and introduce four axes of domain complexity. We then propose an
experiment to study the relative efficacy of a broad set of reward explanation
techniques covering multiple modalities of information in a set of domains of
varying complexity.

negative:
One obstacle to applying reinforcement learning algorithms to real-world
problems is the lack of suitable reward functions. Designing such reward
functions is difficult in part because the user only has an implicit
understanding of the task objective. This gives rise to the agent alignment
problem: how do we create agents that behave in accordance with the user's
intentions? We outline a high-level research direction to solve the agent
alignment problem centered around reward modeling: learning a reward function
from interaction with the user and optimizing the learned reward function with
reinforcement learning. We discuss the key challenges we expect to face when
scaling reward modeling to complex and general domains, concrete approaches to
mitigate these challenges, and ways to establish trust in the resulting agents.

negative:
Automated prediction of valence, one key feature of a person's emotional
state, from individuals' personal narratives may provide crucial information
for mental healthcare (e.g. early diagnosis of mental diseases, supervision of
disease course, etc.). In the Interspeech 2018 ComParE Self-Assessed Affect
challenge, the task of valence prediction was framed as a three-class
classification problem using 8 seconds fragments from individuals' narratives.
As such, the task did not allow for exploring contextual information of the
narratives. In this work, we investigate the intrinsic information from
multiple narratives recounted by the same individual in order to predict their
current state-of-mind. Furthermore, with generalizability in mind, we decided
to focus our experiments exclusively on textual information as the public
availability of audio narratives is limited compared to text. Our hypothesis
is, that context modeling might provide insights about emotion triggering
concepts (e.g. events, people, places) mentioned in the narratives that are
linked to an individual's state of mind. We explore multiple machine learning
techniques to model narratives. We find that the models are able to capture
inter-individual differences, leading to more accurate predictions of an
individual's emotional state, as compared to single narratives.

negative:
The high request for autonomous and flexible HRI implies the necessity of
deploying Machine Learning (ML) mechanisms in the robot control. Indeed, the
use of ML techniques, such as Reinforcement Learning (RL), makes the robot
behaviour, during the learning process, not transparent to the observing user.
In this work, we proposed an emotional model to improve the transparency in RL
tasks for human-robot collaborative scenarios. The architecture we propose
supports the RL algorithm with an emotional model able to both receive human
feedback and exhibit emotional responses based on the learning process. The
model is entirely based on the Temporal Difference (TD) error. The architecture
was tested in an isolated laboratory with a simple setup. The results highlight
that showing its internal state through an emotional response is enough to make
a robot transparent to its human teacher. People also prefer to interact with a
responsive robot because they are used to understand their intentions via
emotions and social signals.

negative:
Human communication is often executed in the form of a narrative, an account
of connected events composed of characters, actions, and settings. A coherent
narrative structure is therefore a requisite for a well-formulated narrative --
be it fictional or nonfictional -- for informative and effective communication,
opening up the possibility of a deeper understanding of a narrative by studying
its structural properties. In this paper we present a network-based framework
for modeling and analyzing the structure of a narrative, which is further
expanded by incorporating methods from computational linguistics to utilize the
narrative text. Modeling a narrative as a dynamically unfolding system, we
characterize its progression via the growth patterns of the character network,
and use sentiment analysis and topic modeling to represent the actual content
of the narrative in the form of interaction maps between characters with
associated sentiment values and keywords. This is a network framework advanced
beyond the simple occurrence-based one most often used until now, allowing one
to utilize the unique characteristics of a given narrative to a high degree.
Given the ubiquity and importance of narratives, such advanced network-based
representation and analysis framework may lead to a more systematic modeling
and understanding of narratives for social interactions, expression of human
sentiments, and communication.

negative:
Grounding human-machine conversation in a document is an effective way to
improve the performance of retrieval-based chatbots. However, only a part of
the document content may be relevant to help select the appropriate response at
a round. It is thus crucial to select the part of document content relevant to
the current conversation context. In this paper, we propose a document content
selection network (CSN) to perform explicit selection of relevant document
contents, and filter out the irrelevant parts. We show in experiments on two
public document-grounded conversation datasets that CSN can effectively help
select the relevant document contents to the conversation context, and it
produces better results than the state-of-the-art approaches. Our code and
datasets are available at https://github.com/DaoD/CSN.

negative:
An emerging recipe for achieving state-of-the-art effectiveness in neural
document re-ranking involves utilizing large pre-trained language models -
e.g., BERT - to evaluate all individual passages in the document and then
aggregating the outputs by pooling or additional Transformer layers. A major
drawback of this approach is high query latency due to the cost of evaluating
every passage in the document with BERT. To make matters worse, this high
inference cost and latency varies based on the length of the document, with
longer documents requiring more time and computation. To address this
challenge, we adopt an intra-document cascading strategy, which prunes passages
of a candidate document using a less expensive model, called ESM, before
running a scoring model that is more expensive and effective, called ETM. We
found it best to train ESM (short for Efficient Student Model) via knowledge
distillation from the ETM (short for Effective Teacher Model) e.g., BERT. This
pruning allows us to only run the ETM model on a smaller set of passages whose
size does not vary by document length. Our experiments on the MS MARCO and TREC
Deep Learning Track benchmarks suggest that the proposed Intra-Document
Cascaded Ranking Model (IDCM) leads to over 400% lower query latency by
providing essentially the same effectiveness as the state-of-the-art BERT-based
document ranking models.

negative:
End-to-End intelligent neural dialogue systems suffer from the problems of
generating inconsistent and repetitive responses. Existing dialogue models pay
attention to unilaterally incorporating personal knowledge into the dialog
while ignoring the fact that incorporating the personality-related conversation
information into personal knowledge taken as the bilateral information flow
boosts the quality of the subsequent conversation. Besides, it is indispensable
to control personal knowledge utilization over the conversation level. In this
paper, we propose a conversation-adaption multi-view persona aware response
generation model that aims at enhancing conversation consistency and
alleviating the repetition from two folds. First, we consider conversation
consistency from multiple views. From the view of the persona profile, we
design a novel interaction module that not only iteratively incorporates
personalized knowledge into each turn conversation but also captures the
personality-related information from conversation to enhance personalized
knowledge semantic representation. From the view of speaking style, we
introduce the speaking style vector and feed it into the decoder to keep the
speaking style consistency. To avoid conversation repetition, we devise a
coverage mechanism to keep track of the activation of personal knowledge
utilization. Experiments on both automatic and human evaluation verify the
superiority of our model over previous models.

negative:
Mental State Transition Network (MSTN) is a basic concept of approximating to
human psychological and mental responses. A stimulus calculated by Emotion
Generating Calculations (EGC) method can cause the transition of mood from an
emotional state to others. In this paper, the agent can interact with human to
realize smooth communication by an adaptive learning method of the user's
personality trait based mood. The learning method consists of the profit
sharing (PS) method and the recurrent neural network (RNN). An emotion for
sensor inputs to MSTN is calculated by EGC and the variance of emotion leads to
the change of mental state, and then the sequence of states forms an episode.
In order to learn the tendency of personality trait effectively, the
ineffective rules should be removed from the episode. PS method finds out a
detour in episode and should be deleted. Furthermore, RNN works to realize the
variance of user's mood. Some experimental results were shown the success of
representing a various human's delicate emotion.

negative:
Hierarchical neural architectures are often used to capture long-distance
dependencies and have been applied to many document-level tasks such as
summarization, document segmentation, and sentiment analysis. However,
effective usage of such a large context can be difficult to learn, especially
in the case where there is limited labeled data available. Building on the
recent success of language model pretraining methods for learning flat
representations of text, we propose algorithms for pre-training hierarchical
document representations from unlabeled data. Unlike prior work, which has
focused on pre-training contextual token representations or context-independent
{sentence/paragraph} representations, our hierarchical document representations
include fixed-length sentence/paragraph representations which integrate
contextual information from the entire documents. Experiments on document
segmentation, document-level question answering, and extractive document
summarization demonstrate the effectiveness of the proposed pre-training
algorithms.

negative:
World models improve a learning agent's ability to efficiently operate in
interactive and situated environments. This work focuses on the task of
building world models of text-based game environments. Text-based games, or
interactive narratives, are reinforcement learning environments in which agents
perceive and interact with the world using textual natural language. These
environments contain long, multi-step puzzles or quests woven through a world
that is filled with hundreds of characters, locations, and objects. Our world
model learns to simultaneously: (1) predict changes in the world caused by an
agent's actions when representing the world as a knowledge graph; and (2)
generate the set of contextually relevant natural language actions required to
operate in the world. We frame this task as a Set of Sequences generation
problem by exploiting the inherent structure of knowledge graphs and actions
and introduce both a transformer-based multi-task architecture and a loss
function to train it. A zero-shot ablation study on never-before-seen textual
worlds shows that our methodology significantly outperforms existing textual
world modeling techniques as well as the importance of each of our
contributions.

negative:
To build an open-domain multi-turn conversation system is one of the most
interesting and challenging tasks in Artificial Intelligence. Many research
efforts have been dedicated to building such dialogue systems, yet few shed
light on modeling the conversation flow in an ongoing dialogue. Besides, it is
common for people to talk about highly relevant aspects during a conversation.
And the topics are coherent and drift naturally, which demonstrates the
necessity of dialogue flow modeling. To this end, we present the multi-turn
cue-words driven conversation system with reinforcement learning method (RLCw),
which strives to select an adaptive cue word with the greatest future credit,
and therefore improve the quality of generated responses. We introduce a new
reward to measure the quality of cue words in terms of effectiveness and
relevance. To further optimize the model for long-term conversations, a
reinforcement approach is adopted in this paper. Experiments on real-life
dataset demonstrate that our model consistently outperforms a set of
competitive baselines in terms of simulated turns, diversity and human
evaluation.

negative:
In this paper we consider the problem of how a reinforcement learning agent
that is tasked with solving a sequence of reinforcement learning problems (a
sequence of Markov decision processes) can use knowledge acquired early in its
lifetime to improve its ability to solve new problems. We argue that previous
experience with similar problems can provide an agent with information about
how it should explore when facing a new but related problem. We show that the
search for an optimal exploration strategy can be formulated as a reinforcement
learning problem itself and demonstrate that such strategy can leverage
patterns found in the structure of related problems. We conclude with
experiments that show the benefits of optimizing an exploration strategy using
our proposed approach.

negative:
How to leverage cross-document interactions to improve ranking performance is
an important topic in information retrieval (IR) research. However, this topic
has not been well-studied in the learning-to-rank setting and most of the
existing work still treats each document independently while scoring. The
recent development of deep learning shows strength in modeling complex
relationships across sequences and sets. It thus motivates us to study how to
leverage cross-document interactions for learning-to-rank in the deep learning
framework. In this paper, we formally define the permutation-equivariance
requirement for a scoring function that captures cross-document interactions.
We then propose a self-attention based document interaction network and show
that it satisfies the permutation-equivariant requirement, and can generate
scores for document sets of varying sizes. Our proposed methods can
automatically learn to capture document interactions without any auxiliary
information, and can scale across large document sets. We conduct experiments
on three ranking datasets: the benchmark Web30k, a Gmail search, and a Google
Drive Quick Access dataset. Experimental results show that our proposed methods
are both more effective and efficient than baselines.

negative:
We present two deep learning approaches to narrative text understanding for
character relationship modelling. The temporal evolution of these relations is
described by dynamic word embeddings, that are designed to learn semantic
changes over time. An empirical analysis of the corresponding character
trajectories shows that such approaches are effective in depicting dynamic
evolution. A supervised learning approach based on the state-of-the-art
transformer model BERT is used instead to detect static relations between
characters. The empirical validation shows that such events (e.g., two
characters belonging to the same family) might be spotted with good accuracy,
even when using automatically annotated data. This provides a deeper
understanding of narrative plots based on the identification of key facts.
Standard clustering techniques are finally used for character de-aliasing, a
necessary pre-processing step for both approaches. Overall, deep learning
models appear to be suitable for narrative text understanding, while also
providing a challenging and unexploited benchmark for general natural language
understanding.

negative:
Dialogue authoring in large games requires not only content creation but the
subtlety of its delivery, which can vary from character to character. Manually
authoring this dialogue can be tedious, time-consuming, or even altogether
infeasible. This paper utilizes a rich narrative representation for modeling
dialogue and an expressive natural language generation engine for realizing it,
and expands upon a translation tool that bridges the two. We add functionality
to the translator to allow direct speech to be modeled by the narrative
representation, whereas the original translator supports only narratives told
by a third person narrator. We show that we can perform character substitution
in dialogues. We implement and evaluate a potential application to dialogue
implementation: generating dialogue for games with big, dynamic, or
procedurally-generated open worlds. We present a pilot study on human
perceptions of the personalities of characters using direct speech, assuming
unknown personality types at the time of authoring.

negative:
We propose Object-oriented Neural Programming (OONP), a framework for
semantically parsing documents in specific domains. Basically, OONP reads a
document and parses it into a predesigned object-oriented data structure
(referred to as ontology in this paper) that reflects the domain-specific
semantics of the document. An OONP parser models semantic parsing as a decision
process: a neural net-based Reader sequentially goes through the document, and
during the process it builds and updates an intermediate ontology to summarize
its partial understanding of the text it covers. OONP supports a rich family of
operations (both symbolic and differentiable) for composing the ontology, and a
big variety of forms (both symbolic and differentiable) for representing the
state and the document. An OONP parser can be trained with supervision of
different forms and strength, including supervised learning (SL) ,
reinforcement learning (RL) and hybrid of the two. Our experiments on both
synthetic and real-world document parsing tasks have shown that OONP can learn
to handle fairly complicated ontology with training data of modest sizes.

negative:
It is appealing to have a system that generates a story or scripts
automatically from a story-line, even though this is still out of our reach. In
dialogue systems, it would also be useful to drive dialogues by a dialogue
plan. In this paper, we address a key problem involved in these applications --
guiding a dialogue by a narrative. The proposed model ScriptWriter selects the
best response among the candidates that fit the context as well as the given
narrative. It keeps track of what in the narrative has been said and what is to
be said. A narrative plays a different role than the context (i.e., previous
utterances), which is generally used in current dialogue systems. Due to the
unavailability of data for this new application, we construct a new large-scale
data collection GraphMovie from a movie website where end-users can upload
their narratives freely when watching a movie. Experimental results on the
dataset show that our proposed approach based on narratives significantly
outperforms the baselines that simply use the narrative as a kind of context.

negative:
Neural summarization models suffer from the fixed-size input limitation: if
text length surpasses the model's maximal number of input tokens, some document
content (possibly summary-relevant) gets truncated Independently summarizing
windows of maximal input size disallows for information flow between windows
and leads to incoherent summaries. We propose windowing models for neural
abstractive summarization of (arbitrarily) long texts. We extend the
sequence-to-sequence model augmented with pointer generator network by (1)
allowing the encoder to slide over different windows of the input document and
(2) sharing the decoder and retaining its state across different input windows.
We explore two windowing variants: Static Windowing precomputes the number of
tokens the decoder should generate from each window (based on training corpus
statistics); in Dynamic Windowing the decoder learns to emit a token that
signals encoder's shift to the next input window. Empirical results render our
models effective in their intended use-case: summarizing long texts with
relevant content not bound to the very document beginning.

negative:
The use of conversational assistants to search for information is becoming
increasingly more popular among the general public, pushing the research
towards more advanced and sophisticated techniques. In the last few years, in
particular, the interest in conversational search is increasing, not only
because of the generalization of conversational assistants but also because
conversational search is a step forward in allowing a more natural interaction
with the system.
  In this work, the focus is on exploring the context present of the
conversation via the historical utterances and respective embeddings with the
aim of developing a conversational search system that helps people search for
information in a natural way. In particular, this system must be able to
understand the context where the question is posed, tracking the current state
of the conversation and detecting mentions to previous questions and answers.
We achieve this by using a context-tracking component based on neural
query-rewriting models. Another crucial aspect of the system is to provide the
most relevant answers given the question and the conversational history. To
achieve this objective, we used a Transformer-based re-ranking method and
expanded this architecture to use the conversational context.
  The results obtained with the system developed showed the advantages of using
the context present in the natural language utterances and in the neural
embeddings generated throughout the conversation.

negative:
Reinforcement learning optimizes policies for expected cumulative reward.
Need the supervision be so narrow? Reward is delayed and sparse for many tasks,
making it a difficult and impoverished signal for end-to-end optimization. To
augment reward, we consider a range of self-supervised tasks that incorporate
states, actions, and successors to provide auxiliary losses. These losses offer
ubiquitous and instantaneous supervision for representation learning even in
the absence of reward. While current results show that learning from reward
alone is feasible, pure reinforcement learning methods are constrained by
computational and data efficiency issues that can be remedied by auxiliary
losses. Self-supervised pre-training and joint optimization improve the data
efficiency and policy returns of end-to-end reinforcement learning.

negative:
Recent progress in pretrained Transformer-based language models has shown
great success in learning contextual representation of text. However, due to
the quadratic self-attention complexity, most of the pretrained Transformers
models can only handle relatively short text. It is still a challenge when it
comes to modeling very long documents. In this work, we propose to use a graph
attention network on top of the available pretrained Transformers model to
learn document embeddings. This graph attention network allows us to leverage
the high-level semantic structure of the document. In addition, based on our
graph document model, we design a simple contrastive learning strategy to
pretrain our models on a large amount of unlabeled corpus. Empirically, we
demonstrate the effectiveness of our approaches in document classification and
document retrieval tasks.

negative:
Neural conversation models are known to generate appropriate but
non-informative responses in general. A scenario where informativeness can be
significantly enhanced is Conversing by Reading (CbR), where conversations take
place with respect to a given external document. In previous work, the external
document is utilized by (1) creating a context-aware document memory that
integrates information from the document and the conversational context, and
then (2) generating responses referring to the memory. In this paper, we
propose to create the document memory with some anticipated responses in mind.
This is achieved using a teacher-student framework. The teacher is given the
external document, the context, and the ground-truth response, and learns how
to build a response-aware document memory from three sources of information.
The student learns to construct a response-anticipated document memory from the
first two sources, and the teacher's insight on memory creation. Empirical
results show that our model outperforms the previous state-of-the-art for the
CbR task.

negative:
In this work, we propose a novel method for training neural networks to
perform single-document extractive summarization without
heuristically-generated extractive labels. We call our approach BanditSum as it
treats extractive summarization as a contextual bandit (CB) problem, where the
model receives a document to summarize (the context), and chooses a sequence of
sentences to include in the summary (the action). A policy gradient
reinforcement learning algorithm is used to train the model to select sequences
of sentences that maximize ROUGE score. We perform a series of experiments
demonstrating that BanditSum is able to achieve ROUGE scores that are better
than or comparable to the state-of-the-art for extractive summarization, and
converges using significantly fewer update steps than competing approaches. In
addition, we show empirically that BanditSum performs significantly better than
competing approaches when good summary sentences appear late in the source
document.

negative:
No real-world reward function is perfect. Sensory errors and software bugs
may result in RL agents observing higher (or lower) rewards than they should.
For example, a reinforcement learning agent may prefer states where a sensory
error gives it the maximum reward, but where the true reward is actually small.
We formalise this problem as a generalised Markov Decision Problem called
Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under
strong simplifying assumptions and when trying to compensate for the possibly
corrupt rewards. Two ways around the problem are investigated. First, by giving
the agent richer data, such as in inverse reinforcement learning and
semi-supervised reinforcement learning, reward corruption stemming from
systematic sensory errors may sometimes be completely managed. Second, by using
randomisation to blunt the agent's optimisation, reward corruption can be
partially managed under some assumptions.

negative:
Bayesian reward learning from demonstrations enables rigorous safety and
uncertainty analysis when performing imitation learning. However, Bayesian
reward learning methods are typically computationally intractable for complex
control problems. We propose Bayesian Reward Extrapolation (Bayesian REX), a
highly efficient Bayesian reward learning algorithm that scales to
high-dimensional imitation learning problems by pre-training a low-dimensional
feature encoding via self-supervised tasks and then leveraging preferences over
demonstrations to perform fast Bayesian inference. Bayesian REX can learn to
play Atari games from demonstrations, without access to the game score and can
generate 100,000 samples from the posterior over reward functions in only 5
minutes on a personal laptop. Bayesian REX also results in imitation learning
performance that is competitive with or better than state-of-the-art methods
that only learn point estimates of the reward function. Finally, Bayesian REX
enables efficient high-confidence policy evaluation without having access to
samples of the reward function. These high-confidence performance bounds can be
used to rank the performance and risk of a variety of evaluation policies and
provide a way to detect reward hacking behaviors.

negative:
When a person is not satisfied with how a robot performs a task, they can
intervene to correct it. Reward learning methods enable the robot to adapt its
reward function online based on such human input, but they rely on handcrafted
features. When the correction cannot be explained by these features, recent
work in deep Inverse Reinforcement Learning (IRL) suggests that the robot could
ask for task demonstrations and recover a reward defined over the raw state
space. Our insight is that rather than implicitly learning about the missing
feature(s) from demonstrations, the robot should instead ask for data that
explicitly teaches it about what it is missing. We introduce a new type of
human input in which the person guides the robot from states where the feature
being taught is highly expressed to states where it is not. We propose an
algorithm for learning the feature from the raw state space and integrating it
into the reward function. By focusing the human input on the missing feature,
our method decreases sample complexity and improves generalization of the
learned reward over the above deep IRL baseline. We show this in experiments
with a physical 7DOF robot manipulator, as well as in a user study conducted in
a simulated environment.

negative:
In dialogue systems, utterances with similar semantics may have distinctive
emotions under different contexts. Therefore, modeling long-range contextual
emotional relationships with speaker dependency plays a crucial part in
dialogue emotion recognition. Meanwhile, distinguishing the different emotion
categories is non-trivial since they usually have semantically similar
sentiments. To this end, we adopt supervised contrastive learning to make
different emotions mutually exclusive to identify similar emotions better.
Meanwhile, we utilize an auxiliary response generation task to enhance the
model's ability of handling context information, thereby forcing the model to
recognize emotions with similar semantics in diverse contexts. To achieve these
objectives, we use the pre-trained encoder-decoder model BART as our backbone
model since it is very suitable for both understanding and generation tasks.
The experiments on four datasets demonstrate that our proposed model obtains
significantly more favorable results than the state-of-the-art model in
dialogue emotion recognition. The ablation study further demonstrates the
effectiveness of supervised contrastive loss and generative loss.

negative:
While there have been significant advances in de-tecting emotions in text, in
the field of utter-ance-level emotion recognition (ULER), there are still many
problems to be solved. In this paper, we address some challenges in ULER in
dialog sys-tems. (1) The same utterance can deliver different emotions when it
is in different contexts or from different speakers. (2) Long-range contextual
in-formation is hard to effectively capture. (3) Unlike the traditional text
classification problem, this task is supported by a limited number of datasets,
among which most contain inadequate conversa-tions or speech. To address these
problems, we propose a hierarchical transformer framework (apart from the
description of other studies, the "transformer" in this paper usually refers to
the encoder part of the transformer) with a lower-level transformer to model
the word-level input and an upper-level transformer to capture the context of
utterance-level embeddings. We use a pretrained language model bidirectional
encoder representa-tions from transformers (BERT) as the lower-level
transformer, which is equivalent to introducing external data into the model
and solve the problem of data shortage to some extent. In addition, we add
speaker embeddings to the model for the first time, which enables our model to
capture the in-teraction between speakers. Experiments on three dialog emotion
datasets, Friends, EmotionPush, and EmoryNLP, demonstrate that our proposed
hierarchical transformer network models achieve 1.98%, 2.83%, and 3.94%
improvement, respec-tively, over the state-of-the-art methods on each dataset
in terms of macro-F1.

negative:
Emotion dynamics modeling is a significant task in emotion recognition in
conversation. It aims to predict conversational emotions when building
empathetic dialogue systems. Existing studies mainly develop models based on
Recurrent Neural Networks (RNNs). They cannot benefit from the power of the
recently-developed pre-training strategies for better token representation
learning in conversations. More seriously, it is hard to distinguish the
dependency of interlocutors and the emotional influence among interlocutors by
simply assembling the features on top of RNNs. In this paper, we develop a
series of BERT-based models to specifically capture the inter-interlocutor and
intra-interlocutor dependencies of the conversational emotion dynamics.
Concretely, we first substitute BERT for RNNs to enrich the token
representations. Then, a Flat-structured BERT (F-BERT) is applied to link up
utterances in a conversation directly, and a Hierarchically-structured BERT
(H-BERT) is employed to distinguish the interlocutors when linking up
utterances. More importantly, a Spatial-Temporal-structured BERT, namely
ST-BERT, is proposed to further determine the emotional influence among
interlocutors. Finally, we conduct extensive experiments on two popular emotion
recognition in conversation benchmark datasets and demonstrate that our
proposed models can attain around 5\% and 10\% improvement over the
state-of-the-art baselines, respectively.

negative:
While large-scale pretrained language models have been shown to learn
effective linguistic representations for many NLP tasks, there remain many
real-world contextual aspects of language that current approaches do not
capture. For instance, consider a cloze-test "I enjoyed the ____ game this
weekend": the correct answer depends heavily on where the speaker is from, when
the utterance occurred, and the speaker's broader social milieu and
preferences. Although language depends heavily on the geographical, temporal,
and other social contexts of the speaker, these elements have not been
incorporated into modern transformer-based language models. We propose a simple
but effective approach to incorporate speaker social context into the learned
representations of large-scale language models. Our method first learns dense
representations of social contexts using graph representation learning
algorithms and then primes language model pretraining with these social context
representations. We evaluate our approach on geographically-sensitive
language-modeling tasks and show a substantial improvement (more than 100%
relative lift on MRR) compared to baselines.

negative:
Most extractive summarization methods focus on the main body of the document
from which sentences need to be extracted. However, the gist of the document
may lie in side information, such as the title and image captions which are
often available for newswire articles. We propose to explore side information
in the context of single-document extractive summarization. We develop a
framework for single-document summarization composed of a hierarchical document
encoder and an attention-based extractor with attention over side information.
We evaluate our model on a large scale news dataset. We show that extractive
summarization with side information consistently outperforms its counterpart
that does not use any side information, in terms of both informativeness and
fluency.

negative:
Document-level machine translation manages to outperform sentence level
models by a small margin, but have failed to be widely adopted. We argue that
previous research did not make a clear use of the global context, and propose a
new document-level NMT framework that deliberately models the local context of
each sentence with the awareness of the global context of the document in both
source and target languages. We specifically design the model to be able to
deal with documents containing any number of sentences, including single
sentences. This unified approach allows our model to be trained elegantly on
standard datasets without needing to train on sentence and document level data
separately. Experimental results demonstrate that our model outperforms
Transformer baselines and previous document-level NMT models with substantial
margins of up to 2.1 BLEU on state-of-the-art baselines. We also provide
analyses which show the benefit of context far beyond the neighboring two or
three sentences, which previous studies have typically incorporated.

negative:
We introduce a framework that abstracts Reinforcement Learning (RL) as a
sequence modeling problem. This allows us to draw upon the simplicity and
scalability of the Transformer architecture, and associated advances in
language modeling such as GPT-x and BERT. In particular, we present Decision
Transformer, an architecture that casts the problem of RL as conditional
sequence modeling. Unlike prior approaches to RL that fit value functions or
compute policy gradients, Decision Transformer simply outputs the optimal
actions by leveraging a causally masked Transformer. By conditioning an
autoregressive model on the desired return (reward), past states, and actions,
our Decision Transformer model can generate future actions that achieve the
desired return. Despite its simplicity, Decision Transformer matches or exceeds
the performance of state-of-the-art model-free offline RL baselines on Atari,
OpenAI Gym, and Key-to-Door tasks.

negative:
In continuing tasks, average-reward reinforcement learning may be a more
appropriate problem formulation than the more common discounted reward
formulation. As usual, learning an optimal policy in this setting typically
requires a large amount of training experiences. Reward shaping is a common
approach for incorporating domain knowledge into reinforcement learning in
order to speed up convergence to an optimal policy. However, to the best of our
knowledge, the theoretical properties of reward shaping have thus far only been
established in the discounted setting. This paper presents the first reward
shaping framework for average-reward learning and proves that, under standard
assumptions, the optimal policy under the original reward function can be
recovered. In order to avoid the need for manual construction of the shaping
function, we introduce a method for utilizing domain knowledge expressed as a
temporal logic formula. The formula is automatically translated to a shaping
function that provides additional reward throughout the learning process. We
evaluate the proposed method on three continuing tasks. In all cases, shaping
speeds up the average-reward learning rate without any reduction in the
performance of the learned policy compared to relevant baselines.

negative:
Narrative, and in particular storytelling, is an important part of the human
experience. Consequently, computational systems that can reason about narrative
can be more effective communicators, entertainers, educators, and trainers. One
of the central challenges in computational narrative reasoning is narrative
generation, the automated creation of meaningful event sequences. There are
many factors -- logical and aesthetic -- that contribute to the success of a
narrative artifact. Central to this success is its understandability. We argue
that the following two attributes of narratives are universal: (a) the logical
causal progression of plot, and (b) character believability. Character
believability is the perception by the audience that the actions performed by
characters do not negatively impact the audiences suspension of disbelief.
Specifically, characters must be perceived by the audience to be intentional
agents. In this article, we explore the use of refinement search as a technique
for solving the narrative generation problem -- to find a sound and believable
sequence of character actions that transforms an initial world state into a
world state in which goal propositions hold. We describe a novel refinement
search planning algorithm -- the Intent-based Partial Order Causal Link (IPOCL)
planner -- that, in addition to creating causally sound plot progression,
reasons about character intentionality by identifying possible character goals
that explain their actions and creating plan structures that explain why those
characters commit to their goals. We present the results of an empirical
evaluation that demonstrates that narrative plans generated by the IPOCL
algorithm support audience comprehension of character intentions better than
plans generated by conventional partial-order planners.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
Transformer is important for text modeling. However, it has difficulty in
handling long documents due to the quadratic complexity with input text length.
In order to handle this problem, we propose a hierarchical interactive
Transformer (Hi-Transformer) for efficient and effective long document
modeling. Hi-Transformer models documents in a hierarchical way, i.e., first
learns sentence representations and then learns document representations. It
can effectively reduce the complexity and meanwhile capture global document
context in the modeling of each sentence. More specifically, we first use a
sentence Transformer to learn the representations of each sentence. Then we use
a document Transformer to model the global document context from these sentence
representations. Next, we use another sentence Transformer to enhance sentence
modeling using the global document context. Finally, we use hierarchical
pooling method to obtain document embedding. Extensive experiments on three
benchmark datasets validate the efficiency and effectiveness of Hi-Transformer
in long document modeling.

negative:
With the improvements in speech recognition and voice generation technologies
over the last years, a lot of companies have sought to develop conversation
understanding systems that run on mobile phones or smart home devices through
natural language interfaces. Conversational assistants, such as Google
Assistant and Microsoft Cortana, can help users to complete various types of
tasks. This requires an accurate understanding of the user's information need
as the conversation evolves into multiple turns. Finding relevant context in a
conversation's history is challenging because of the complexity of natural
language and the evolution of a user's information need. In this work, we
present an extensive analysis of language, relevance, dependency of user
utterances in a multi-turn information-seeking conversation. To this aim, we
have annotated relevant utterances in the conversations released by the TREC
CaST 2019 track. The annotation labels determine which of the previous
utterances in a conversation can be used to improve the current one.
Furthermore, we propose a neural utterance relevance model based on BERT
fine-tuning, outperforming competitive baselines. We study and compare the
performance of multiple retrieval models, utilizing different strategies to
incorporate the user's context. The experimental results on both classification
and retrieval tasks show that our proposed approach can effectively identify
and incorporate the conversation context. We show that processing the current
utterance using the predicted relevant utterance leads to a 38% relative
improvement in terms of nDCG@20. Finally, to foster research in this area, we
have released the dataset of the annotations.

negative:
The problem of inverse reinforcement learning (IRL) is relevant to a variety
of tasks including value alignment and robot learning from demonstration.
Despite significant algorithmic contributions in recent years, IRL remains an
ill-posed problem at its core; multiple reward functions coincide with the
observed behavior and the actual reward function is not identifiable without
prior knowledge or supplementary information. This paper presents an IRL
framework called Bayesian optimization-IRL (BO-IRL) which identifies multiple
solutions that are consistent with the expert demonstrations by efficiently
exploring the reward function space. BO-IRL achieves this by utilizing Bayesian
Optimization along with our newly proposed kernel that (a) projects the
parameters of policy invariant reward functions to a single point in a latent
space and (b) ensures nearby points in the latent space correspond to reward
functions yielding similar likelihoods. This projection allows the use of
standard stationary kernels in the latent space to capture the correlations
present across the reward function space. Empirical results on synthetic and
real-world environments (model-free and model-based) show that BO-IRL discovers
multiple reward functions while minimizing the number of expensive exact policy
optimizations.

negative:
Human-in-the-loop machine learning is widely used in artificial intelligence
(AI) to elicit labels for data points from experts or to provide feedback on
how close the predicted results are to the target. This simplifies away all the
details of the decision-making process of the expert. In this work, we allow
the experts to additionally produce decision rules describing their
decision-making; the rules are expected to be imperfect but to give additional
information. In particular, the rules can extend to new distributions, and
hence enable significantly improving performance for cases where the training
and testing distributions differ, such as in domain adaptation. We apply the
proposed method to lifelong learning and domain adaptation problems and discuss
applications in other branches of AI, such as knowledge acquisition problems in
expert systems. In simulated and real-user studies, we show that decision rule
elicitation improves domain adaptation of the algorithm and helps to propagate
expert's knowledge to the AI model.

negative:
Humans are able to understand and perform complex tasks by strategically
structuring the tasks into incremental steps or subgoals. For a robot
attempting to learn to perform a sequential task with critical subgoal states,
such states can provide a natural opportunity for interaction with a human
expert. This paper analyzes the benefit of incorporating a notion of subgoals
into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL)
framework. The learning process is interactive, with a human expert first
providing input in the form of full demonstrations along with some subgoal
states. These subgoal states define a set of subtasks for the learning agent to
complete in order to achieve the final goal. The learning agent queries for
partial demonstrations corresponding to each subtask as needed when the agent
struggles with the subtask. The proposed Human Interactive IRL (HI-IRL)
framework is evaluated on several discrete path-planning tasks. We demonstrate
that subgoal-based interactive structuring of the learning task results in
significantly more efficient learning, requiring only a fraction of the
demonstration data needed for learning the underlying reward function with the
baseline IRL model.

negative:
Deep reinforcement learning is the combination of reinforcement learning (RL)
and deep learning. This field of research has been able to solve a wide range
of complex decision-making tasks that were previously out of reach for a
machine. Thus, deep RL opens up many new applications in domains such as
healthcare, robotics, smart grids, finance, and many more. This manuscript
provides an introduction to deep reinforcement learning models, algorithms and
techniques. Particular focus is on the aspects related to generalization and
how deep RL can be used for practical applications. We assume the reader is
familiar with basic machine learning concepts.

negative:
Recognizing emotions in conversations is a challenging task due to the
presence of contextual dependencies governed by self- and inter-personal
influences. Recent approaches have focused on modeling these dependencies
primarily via supervised learning. However, purely supervised strategies demand
large amounts of annotated data, which is lacking in most of the available
corpora in this task. To tackle this challenge, we look at transfer learning
approaches as a viable alternative. Given the large amount of available
conversational data, we investigate whether generative conversational models
can be leveraged to transfer affective knowledge for detecting emotions in
context. We propose an approach, TL-ERC, where we pre-train a hierarchical
dialogue model on multi-turn conversations (source) and then transfer its
parameters to a conversational emotion classifier (target). In addition to the
popular practice of using pre-trained sentence encoders, our approach also
incorporates recurrent parameters that model inter-sentential context across
the whole conversation. Based on this idea, we perform several experiments
across multiple datasets and find improvement in performance and robustness
against limited training data. TL-ERC also achieves better validation
performances in significantly fewer epochs. Overall, we infer that knowledge
acquired from dialogue generators can indeed help recognize emotions in
conversations.

negative:
Recently, text world games have been proposed to enable artificial agents to
understand and reason about real-world scenarios. These text-based games are
challenging for artificial agents, as it requires an understanding of and
interaction using natural language in a partially observable environment.
Agents observe the environment via textual descriptions designed to be
challenging enough for even human players. Past approaches have not paid enough
attention to the language understanding capability of the proposed agents.
Typically, these approaches train from scratch, an agent that learns both
textual representations and the gameplay online during training using a
temporal loss function. Given the sample-inefficiency of RL approaches, it is
inefficient to learn rich enough textual representations to be able to
understand and reason using the textual observation in such a complicated game
environment setting. In this paper, we improve the semantic understanding of
the agent by proposing a simple RL with LM framework where we use
transformer-based language models with Deep RL models. We perform a detailed
study of our framework to demonstrate how our model outperforms all existing
agents on the popular game, Zork1, to achieve a score of 44.7, which is 1.6
higher than the state-of-the-art model. Overall, our proposed approach
outperforms 4 games out of the 14 text-based games, while performing comparable
to the state-of-the-art models on the remaining games.

negative:
Standard neural machine translation (NMT) is on the assumption that the
document-level context is independent. Most existing document-level NMT
approaches are satisfied with a smattering sense of global document-level
information, while this work focuses on exploiting detailed document-level
context in terms of a memory network. The capacity of the memory network that
detecting the most relevant part of the current sentence from memory renders a
natural solution to model the rich document-level context. In this work, the
proposed document-aware memory network is implemented to enhance the
Transformer NMT baseline. Experiments on several tasks show that the proposed
method significantly improves the NMT performance over strong Transformer
baselines and other related studies.

negative:
In this paper, we propose a model-free reinforcement learning method to
synthesize control policies for motion planning problems with continuous states
and actions. The robot is modelled as a labeled discrete-time Markov decision
process (MDP) with continuous state and action spaces. Linear temporal logics
(LTL) are used to specify high-level tasks. We then train deep neural networks
to approximate the value function and policy using an actor-critic
reinforcement learning method. The LTL specification is converted into an
annotated limit-deterministic B\"uchi automaton (LDBA) for continuously shaping
the reward so that dense rewards are available during training. A na\"ive way
of solving a motion planning problem with LTL specifications using
reinforcement learning is to sample a trajectory and then assign a high reward
for training if the trajectory satisfies the entire LTL formula. However, the
sampling complexity needed to find such a trajectory is too high when we have a
complex LTL formula for continuous state and action spaces. As a result, it is
very unlikely that we get enough reward for training if all sample trajectories
start from the initial state in the automata. In this paper, we propose a
method that samples not only an initial state from the state space, but also an
arbitrary state in the automata at the beginning of each training episode. We
test our algorithm in simulation using a car-like robot and find out that our
method can learn policies for different working configurations and LTL
specifications successfully.

negative:
Previous researchers have considered sentiment analysis as a document
classification task, in which input documents are classified into predefined
sentiment classes. Although there are sentences in a document that support
important evidences for sentiment analysis and sentences that do not, they have
treated the document as a bag of sentences. In other words, they have not
considered the importance of each sentence in the document. To effectively
determine polarity of a document, each sentence in the document should be dealt
with different degrees of importance. To address this problem, we propose a
document-level sentence classification model based on deep neural networks, in
which the importance degrees of sentences in documents are automatically
determined through gate mechanisms. To verify our new sentiment analysis model,
we conducted experiments using the sentiment datasets in the four different
domains such as movie reviews, hotel reviews, restaurant reviews, and music
reviews. In the experiments, the proposed model outperformed previous
state-of-the-art models that do not consider importance differences of
sentences in a document. The experimental results show that the importance of
sentences should be considered in a document-level sentiment classification
task.

negative:
Mental health conditions remain under-diagnosed even in countries with common
access to advanced medical care. The ability to accurately and efficiently
predict mood from easily collectible data has several important implications
towards the early detection and intervention of mental health disorders. One
promising data source to help monitor human behavior is from daily smartphone
usage. However, care must be taken to summarize behaviors without identifying
the user through personal (e.g., personally identifiable information) or
protected attributes (e.g., race, gender). In this paper, we study behavioral
markers or daily mood using a recent dataset of mobile behaviors from high-risk
adolescent populations. Using computational models, we find that multimodal
modeling of both text and app usage features is highly predictive of daily mood
over each modality alone. Furthermore, we evaluate approaches that reliably
obfuscate user identity while remaining predictive of daily mood. By combining
multimodal representations with privacy-preserving learning, we are able to
push forward the performance-privacy frontier as compared to unimodal
approaches.

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
In some agent designs like inverse reinforcement learning an agent needs to
learn its own reward function. Learning the reward function and optimising for
it are typically two different processes, usually performed at different
stages. We consider a continual (``one life'') learning approach where the
agent both learns the reward function and optimises for it at the same time. We
show that this comes with a number of pitfalls, such as deliberately
manipulating the learning process in one direction, refusing to learn,
``learning'' facts already known to the agent, and making decisions that are
strictly dominated (for all relevant reward functions). We formally introduce
two desirable properties: the first is `unriggability', which prevents the
agent from steering the learning process in the direction of a reward function
that is easier to optimise. The second is `uninfluenceability', whereby the
reward-function learning process operates by learning facts about the
environment. We show that an uninfluenceable process is automatically
unriggable, and if the set of possible environments is sufficiently rich, the
converse is true too.

negative:
Real-world robotic tasks require complex reward functions. When we define the
problem the robot needs to solve, we pretend that a designer specifies this
complex reward exactly, and it is set in stone from then on. In practice,
however, reward design is an iterative process: the designer chooses a reward,
eventually encounters an "edge-case" environment where the reward incentivizes
the wrong behavior, revises the reward, and repeats. What would it mean to
rethink robotics problems to formally account for this iterative nature of
reward design? We propose that the robot not take the specified reward for
granted, but rather have uncertainty about it, and account for the future
design iterations as future evidence. We contribute an Assisted Reward Design
method that speeds up the design process by anticipating and influencing this
future evidence: rather than letting the designer eventually encounter failure
cases and revise the reward then, the method actively exposes the designer to
such environments during the development phase. We test this method in a
simplified autonomous driving task and find that it more quickly improves the
car's behavior in held-out environments by proposing environments that are
"edge cases" for the current reward.

negative:
We study the problem of designing autonomous agents that can learn to
cooperate effectively with a potentially suboptimal partner while having no
access to the joint reward function. This problem is modeled as a cooperative
episodic two-agent Markov decision process. We assume control over only the
first of the two agents in a Stackelberg formulation of the game, where the
second agent is acting so as to maximise expected utility given the first
agent's policy. How should the first agent act in order to learn the joint
reward function as quickly as possible and so that the joint policy is as close
to optimal as possible? We analyse how knowledge about the reward function can
be gained in this interactive two-agent scenario. We show that when the
learning agent's policies have a significant effect on the transition function,
the reward function can be learned efficiently.

negative:
The ability to infer persona from dialogue can have applications in areas
ranging from computational narrative analysis to personalized dialogue
generation. We introduce neural models to learn persona embeddings in a
supervised character trope classification task. The models encode dialogue
snippets from IMDB into representations that can capture the various categories
of film characters. The best-performing models use a multi-level attention
mechanism over a set of utterances. We also utilize prior knowledge in the form
of textual descriptions of the different tropes. We apply the learned
embeddings to find similar characters across different movies, and cluster
movies according to the distribution of the embeddings. The use of short
conversational text as input, and the ability to learn from prior knowledge
using memory, suggests these methods could be applied to other domains.

negative:
Recently, the automatic prediction of personality traits has received a lot
of attention. Specifically, personality trait prediction from multimodal data
has emerged as a hot topic within the field of affective computing. In this
paper, we review significant machine learning models which have been employed
for personality detection, with an emphasis on deep learning-based methods.
This review paper provides an overview of the most popular approaches to
automated personality detection, various computational datasets, its industrial
applications, and state-of-the-art machine learning models for personality
detection with specific focus on multimodal approaches. Personality detection
is a very broad and diverse topic: this survey only focuses on computational
approaches and leaves out psychological studies on personality detection.

negative:
Document grounded generation is the task of using the information provided in
a document to improve text generation. This work focuses on two different
document grounded generation tasks: Wikipedia Update Generation task and
Dialogue response generation. Our work introduces two novel adaptations of
large scale pre-trained encoder-decoder models focusing on building context
driven representation of the document and enabling specific attention to the
information in the document. Additionally, we provide a stronger BART baseline
for these tasks. Our proposed techniques outperform existing methods on both
automated (at least 48% increase in BLEU-4 points) and human evaluation for
closeness to reference and relevance to the document. Furthermore, we perform
comprehensive manual inspection of the generated output and categorize errors
to provide insights into future directions in modeling these tasks.

negative:
Question Answering with NLP has progressed through the evolution of advanced
model architectures like BERT and BiDAF and earlier word, character, and
context-based embeddings. As BERT has leapfrogged the accuracy of models, an
element of the next frontier can be the introduction of deep networks and an
effective way to train them. In this context, I explored the effectiveness of
deep networks focussing on the model encoder layer of BiDAF. BiDAF with its
heterogeneous layers provides the opportunity not only to explore the
effectiveness of deep networks but also to evaluate whether the refinements
made in lower layers are additive to the refinements made in the upper layers
of the model architecture. I believe the next greatest model in NLP will in
fact fold in a solid language modeling like BERT with a composite architecture
which will bring in refinements in addition to generic language modeling and
will have a more extensive layered architecture. I experimented with the Bypass
network, Residual Highway network, and DenseNet architectures. In addition, I
evaluated the effectiveness of ensembling the last few layers of the network. I
also studied the difference character embeddings make in adding them to the
word embeddings, and whether the effects are additive with deep networks. My
studies indicate that deep networks are in fact effective in giving a boost.
Also, the refinements in the lower layers like embeddings are passed on
additively to the gains made through deep networks.

negative:
We present a contrasting learning approach with data augmentation techniques
to learn document representations in an unsupervised manner. Inspired by recent
contrastive self-supervised learning algorithms used for image and NLP
pretraining, we hypothesize that high-quality document embedding should be
invariant to diverse paraphrases that preserve the semantics of the original
document. With different backbones and contrastive learning frameworks, our
study reveals the enormous benefits of contrastive augmentation for document
representation learning with two additional insights: 1) including data
augmentation in a contrastive way can substantially improve the embedding
quality in unsupervised document representation learning, and 2) in general,
stochastic augmentations generated by simple word-level manipulation work much
better than sentence-level and document-level ones. We plug our method into a
classifier and compare it with a broad range of baseline methods on six
benchmark datasets. Our method can decrease the classification error rate by up
to 6.4% over the SOTA approaches on the document classification task, matching
or even surpassing fully-supervised methods.

negative:
We propose a framework for verifiable and compositional reinforcement
learning (RL) in which a collection of RL subsystems, each of which learns to
accomplish a separate subtask, are composed to achieve an overall task. The
framework consists of a high-level model, represented as a parametric Markov
decision process (pMDP) which is used to plan and to analyze compositions of
subsystems, and of the collection of low-level subsystems themselves. By
defining interfaces between the subsystems, the framework enables automatic
decompositions of task specifications, e.g., reach a target set of states with
a probability of at least 0.95, into individual subtask specifications, i.e.
achieve the subsystem's exit conditions with at least some minimum probability,
given that its entry conditions are met. This in turn allows for the
independent training and testing of the subsystems; if they each learn a policy
satisfying the appropriate subtask specification, then their composition is
guaranteed to satisfy the overall task specification. Conversely, if the
subtask specifications cannot all be satisfied by the learned policies, we
present a method, formulated as the problem of finding an optimal set of
parameters in the pMDP, to automatically update the subtask specifications to
account for the observed shortcomings. The result is an iterative procedure for
defining subtask specifications, and for training the subsystems to meet them.
As an additional benefit, this procedure allows for particularly challenging or
important components of an overall task to be determined automatically, and
focused on, during training. Experimental results demonstrate the presented
framework's novel capabilities.

negative:
Reinforcement learning (RL) algorithms have been around for decades and
employed to solve various sequential decision-making problems. These algorithms
however have faced great challenges when dealing with high-dimensional
environments. The recent development of deep learning has enabled RL methods to
drive optimal policies for sophisticated and capable agents, which can perform
efficiently in these challenging environments. This paper addresses an
important aspect of deep RL related to situations that require multiple agents
to communicate and cooperate to solve complex tasks. A survey of different
approaches to problems related to multi-agent deep RL (MADRL) is presented,
including non-stationarity, partial observability, continuous state and action
spaces, multi-agent training schemes, multi-agent transfer learning. The merits
and demerits of the reviewed methods will be analyzed and discussed, with their
corresponding applications explored. It is envisaged that this review provides
insights about various MADRL methods and can lead to future development of more
robust and highly useful multi-agent learning methods for solving real-world
problems.

negative:
We examine a new commonsense reasoning task: given a narrative describing a
social interaction that centers on two protagonists, systems make inferences
about the underlying relationship trajectory. Specifically, we propose two
evaluation tasks: Relationship Outlook Prediction MCQ and Resolution Prediction
MCQ. In Relationship Outlook Prediction, a system maps an interaction to a
relationship outlook that captures how the interaction is expected to change
the relationship. In Resolution Prediction, a system attributes a given
relationship outlook to a particular resolution that explains the outcome.
These two tasks parallel two real-life questions that people frequently ponder
upon as they navigate different social situations: "where is this relationship
going?" and "how did we end up here?". To facilitate the investigation of human
social relationships through these two tasks, we construct a new dataset,
Social Narrative Tree, which consists of 1250 stories documenting a variety of
daily social interactions. The narratives encode a multitude of social elements
that interweave to give rise to rich commonsense knowledge of how relationships
evolve with respect to social interactions. We establish baseline performances
using language models and the accuracies are significantly lower than human
performance. The results demonstrate that models need to look beyond syntactic
and semantic signals to comprehend complex human relationships.

negative:
Policy distillation, which transfers a teacher policy to a student policy has
achieved great success in challenging tasks of deep reinforcement learning.
This teacher-student framework requires a well-trained teacher model which is
computationally expensive. Moreover, the performance of the student model could
be limited by the teacher model if the teacher model is not optimal. In the
light of collaborative learning, we study the feasibility of involving joint
intellectual efforts from diverse perspectives of student models. In this work,
we introduce dual policy distillation(DPD), a student-student framework in
which two learners operate on the same environment to explore different
perspectives of the environment and extract knowledge from each other to
enhance their learning. The key challenge in developing this dual learning
framework is to identify the beneficial knowledge from the peer learner for
contemporary learning-based reinforcement learning algorithms, since it is
unclear whether the knowledge distilled from an imperfect and noisy peer
learner would be helpful. To address the challenge, we theoretically justify
that distilling knowledge from a peer learner will lead to policy improvement
and propose a disadvantageous distillation strategy based on the theoretical
results. The conducted experiments on several continuous control tasks show
that the proposed framework achieves superior performance with a learning-based
agent and function approximation without the use of expensive teacher models.

negative:
As intelligent systems gain autonomy and capability, it becomes vital to
ensure that their objectives match those of their human users; this is known as
the value-alignment problem. In robotics, value alignment is key to the design
of collaborative robots that can integrate into human workflows, successfully
inferring and adapting to their users' objectives as they go. We argue that a
meaningful solution to value alignment must combine multi-agent decision theory
with rich mathematical models of human cognition, enabling robots to tap into
people's natural collaborative capabilities. We present a solution to the
cooperative inverse reinforcement learning (CIRL) dynamic game based on
well-established cognitive models of decision making and theory of mind. The
solution captures a key reciprocity relation: the human will not plan her
actions in isolation, but rather reason pedagogically about how the robot might
learn from them; the robot, in turn, can anticipate this and interpret the
human's actions pragmatically. To our knowledge, this work constitutes the
first formal analysis of value alignment grounded in empirically validated
cognitive models.

negative:
We present an approach to training neural networks to generate sequences
using actor-critic methods from reinforcement learning (RL). Current
log-likelihood training methods are limited by the discrepancy between their
training and testing modes, as models must generate tokens conditioned on their
previous guesses rather than the ground-truth tokens. We address this problem
by introducing a \textit{critic} network that is trained to predict the value
of an output token, given the policy of an \textit{actor} network. This results
in a training procedure that is much closer to the test phase, and allows us to
directly optimize for a task-specific score such as BLEU. Crucially, since we
leverage these techniques in the supervised learning setting rather than the
traditional RL setting, we condition the critic network on the ground-truth
output. We show that our method leads to improved performance on both a
synthetic task, and for German-English machine translation. Our analysis paves
the way for such methods to be applied in natural language generation tasks,
such as machine translation, caption generation, and dialogue modelling.

negative:
Deep reading models for question-answering have demonstrated promising
performance over the last couple of years. However current systems tend to
learn how to cleverly extract a span of the source document, based on its
similarity with the question, instead of seeking for the appropriate answer.
Indeed, a reading machine should be able to detect relevant passages in a
document regarding a question, but more importantly, it should be able to
reason over the important pieces of the document in order to produce an answer
when it is required. To motivate this purpose, we present ReviewQA, a
question-answering dataset based on hotel reviews. The questions of this
dataset are linked to a set of relational understanding competencies that we
expect a model to master. Indeed, each question comes with an associated type
that characterizes the required competency. With this framework, it is possible
to benchmark the main families of models and to get an overview of what are the
strengths and the weaknesses of a given model on the set of tasks evaluated in
this dataset. Our corpus contains more than 500.000 questions in natural
language over 100.000 hotel reviews. Our setup is projective, the answer of a
question does not need to be extracted from a document, like in most of the
recent datasets, but selected among a set of candidates that contains all the
possible answers to the questions of the dataset. Finally, we present several
baselines over this dataset.

negative:
We explore using multilingual document embeddings for nearest neighbor mining
of parallel data. Three document-level representations are investigated: (i)
document embeddings generated by simply averaging multilingual sentence
embeddings; (ii) a neural bag-of-words (BoW) document encoding model; (iii) a
hierarchical multilingual document encoder (HiDE) that builds on our
sentence-level model. The results show document embeddings derived from
sentence-level averaging are surprisingly effective for clean datasets, but
suggest models trained hierarchically at the document-level are more effective
on noisy data. Analysis experiments demonstrate our hierarchical models are
very robust to variations in the underlying sentence embedding quality. Using
document embeddings trained with HiDE achieves state-of-the-art performance on
United Nations (UN) parallel document mining, 94.9% P@1 for en-fr and 97.3% P@1
for en-es.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
Bipolar disorder (BPD) is a chronic mental illness characterized by extreme
mood and energy changes from mania to depression. These changes drive behaviors
that often lead to devastating personal or social consequences. BPD is managed
clinically with regular interactions with care providers, who assess mood,
energy levels, and the form and content of speech. Recent work has proposed
smartphones for monitoring mood using speech. However, these works do not
predict when to intervene. Predicting when to intervene is challenging because
there is not a single measure that is relevant for every person: different
individuals may have different levels of symptom severity considered typical.
Additionally, this typical mood, or baseline, may change over time, making a
single symptom threshold insufficient. This work presents an innovative
approach that expands clinical mood monitoring to predict when interventions
are necessary using an anomaly detection framework, which we call Temporal
Normalization. We first validate the model using a dataset annotated for
clinical interventions and then incorporate this method in a deep learning
framework to predict mood anomalies from natural, unstructured, telephone
speech data. The combination of these approaches provides a framework to enable
real-world speech-focused mood monitoring.

negative:
In many real-world deployments of machine learning systems, data arrive
piecemeal. These learning scenarios may be passive, where data arrive
incrementally due to structural properties of the problem (e.g., daily
financial data) or active, where samples are selected according to a measure of
their quality (e.g., experimental design). In both of these cases, we are
building a sequence of models that incorporate an increasing amount of data. We
would like each of these models in the sequence to be performant and take
advantage of all the data that are available to that point. Conventional
intuition suggests that when solving a sequence of related optimization
problems of this form, it should be possible to initialize using the solution
of the previous iterate -- to "warm start" the optimization rather than
initialize from scratch -- and see reductions in wall-clock time. However, in
practice this warm-starting seems to yield poorer generalization performance
than models that have fresh random initializations, even though the final
training losses are similar. While it appears that some hyperparameter settings
allow a practitioner to close this generalization gap, they seem to only do so
in regimes that damage the wall-clock gains of the warm start. Nevertheless, it
is highly desirable to be able to warm-start neural network training, as it
would dramatically reduce the resource usage associated with the construction
of performant deep learning systems. In this work, we take a closer look at
this empirical phenomenon and try to understand when and how it occurs. We also
provide a surprisingly simple trick that overcomes this pathology in several
important situations, and present experiments that elucidate some of its
properties.

negative:
Information Retrieval (IR) models need to deal with two difficult issues,
vocabulary mismatch and term dependencies. Vocabulary mismatch corresponds to
the difficulty of retrieving relevant documents that do not contain exact query
terms but semantically related terms. Term dependencies refers to the need of
considering the relationship between the words of the query when estimating the
relevance of a document. A multitude of solutions has been proposed to solve
each of these two problems, but no principled model solve both. In parallel, in
the last few years, language models based on neural networks have been used to
cope with complex natural language processing tasks like emotion and paraphrase
detection. Although they present good abilities to cope with both term
dependencies and vocabulary mismatch problems, thanks to the distributed
representation of words they are based upon, such models could not be used
readily in IR, where the estimation of one language model per document (or
query) is required. This is both computationally unfeasible and prone to
over-fitting. Based on a recent work that proposed to learn a generic language
model that can be modified through a set of document-specific parameters, we
explore use of new neural network models that are adapted to ad-hoc IR tasks.
Within the language model IR framework, we propose and study the use of a
generic language model as well as a document-specific language model. Both can
be used as a smoothing component, but the latter is more adapted to the
document at hand and has the potential of being used as a full document
language model. We experiment with such models and analyze their results on
TREC-1 to 8 datasets.

negative:
This work explores the ability of collective matrix factorization models in
recommender systems to make predictions about users and items for which there
is side information available but no feedback or interactions data, and
proposes a new formulation with a faster cold-start prediction formula that can
be used in real-time systems. While these cold-start recommendations are not as
good as warm-start ones, they were found to be of better quality than
non-personalized recommendations, and predictions about new users were found to
be more reliable than those about new items. The formulation proposed here
resulted in improved cold-start recommendations in many scenarios, at the
expense of worse warm-start ones.

negative:
Different people have different habits of describing their intents in
conversations. Some people tend to deliberate their intents in several
successive utterances, i.e., they use several consistent messages for
readability instead of a long sentence to express their question. This creates
a predicament faced by the application of dialogue systems, especially in
real-world industry scenarios, in which the dialogue system is unsure whether
it should answer the query of user immediately or wait for further
supplementary input. Motivated by such an interesting predicament, we define a
novel Wait-or-Answer task for dialogue systems. We shed light on a new research
topic about how the dialogue system can be more intelligent to behave in this
Wait-or-Answer quandary. Further, we propose a predictive approach named
Predict-then-Decide (PTD) to tackle this Wait-or-Answer task. More
specifically, we take advantage of a decision model to help the dialogue system
decide whether to wait or answer. The decision of decision model is made with
the assistance of two ancillary prediction models: a user prediction and an
agent prediction. The user prediction model tries to predict what the user
would supplement and uses its prediction to persuade the decision model that
the user has some information to add, so the dialogue system should wait. The
agent prediction model tries to predict the answer of the dialogue system and
convince the decision model that it is a superior choice to answer the query of
user immediately since the input of user has come to an end. We conduct our
experiments on two real-life scenarios and three public datasets. Experimental
results on five datasets show our proposed PTD approach significantly
outperforms the existing models in solving this Wait-or-Answer problem.

negative:
The recent advances introduced by neural machine translation (NMT) are
rapidly expanding the application fields of machine translation, as well as
reshaping the quality level to be targeted. In particular, if translations have
to fit some given layout, quality should not only be measured in terms of
adequacy and fluency, but also length. Exemplary cases are the translation of
document files, subtitles, and scripts for dubbing, where the output length
should ideally be as close as possible to the length of the input text. This
paper addresses for the first time, to the best of our knowledge, the problem
of controlling the output length in NMT. We investigate two methods for biasing
the output length with a transformer architecture: i) conditioning the output
to a given target-source length-ratio class and ii) enriching the transformer
positional embedding with length information. Our experiments show that both
methods can induce the network to generate shorter translations, as well as
acquiring interpretable linguistic skills.

negative:
How can humans remain in control of artificial intelligence (AI)-based
systems designed to perform tasks autonomously? Such systems are increasingly
ubiquitous, creating benefits - but also undesirable situations where moral
responsibility for their actions cannot be properly attributed to any
particular person or group. The concept of meaningful human control has been
proposed to address responsibility gaps and mitigate them by establishing
conditions that enable a proper attribution of responsibility for humans;
however, clear requirements for researchers, designers, and engineers are yet
inexistent, making the development of AI-based systems that remain under
meaningful human control challenging. In this paper, we address the gap between
philosophical theory and engineering practice by identifying, through an
iterative process of abductive thinking, four actionable properties for
AI-based systems under meaningful human control, which we discuss making use of
two applications scenarios: automated vehicles and AI-based hiring. First, a
system in which humans and AI algorithms interact should have an explicitly
defined domain of morally loaded situations within which the system ought to
operate. Second, humans and AI agents within the system should have appropriate
and mutually compatible representations. Third, responsibility attributed to a
human should be commensurate with that human's ability and authority to control
the system. Fourth, there should be explicit links between the actions of the
AI agents and actions of humans who are aware of their moral responsibility. We
argue that these four properties will support practically-minded professionals
to take concrete steps toward designing and engineering for AI systems that
facilitate meaningful human control.

negative:
We study a security threat to reinforcement learning where an attacker
poisons the learning environment to force the agent into executing a target
policy chosen by the attacker. As a victim, we consider RL agents whose
objective is to find a policy that maximizes average reward in undiscounted
infinite-horizon problem settings. The attacker can manipulate the rewards or
the transition dynamics in the learning environment at training-time and is
interested in doing so in a stealthy manner. We propose an optimization
framework for finding an \emph{optimal stealthy attack} for different measures
of attack cost. We provide sufficient technical conditions under which the
attack is feasible and provide lower/upper bounds on the attack cost. We
instantiate our attacks in two settings: (i) an \emph{offline} setting where
the agent is doing planning in the poisoned environment, and (ii) an
\emph{online} setting where the agent is learning a policy using a
regret-minimization framework with poisoned feedback. Our results show that the
attacker can easily succeed in teaching any target policy to the victim under
mild conditions and highlight a significant security threat to reinforcement
learning agents in practice.

negative:
The ability to comprehend wishes or desires and their fulfillment is
important to Natural Language Understanding. This paper introduces the task of
identifying if a desire expressed by a subject in a given short piece of text
was fulfilled. We propose various unstructured and structured models that
capture fulfillment cues such as the subject's emotional state and actions. Our
experiments with two different datasets demonstrate the importance of
understanding the narrative and discourse structure to address this task.

negative:
In many real-world tasks, it is not possible to procedurally specify an RL
agent's reward function. In such cases, a reward function must instead be
learned from interacting with and observing humans. However, current techniques
for reward learning may fail to produce reward functions which accurately
reflect user preferences. Absent significant advances in reward learning, it is
thus important to be able to audit learned reward functions to verify whether
they truly capture user preferences. In this paper, we investigate techniques
for interpreting learned reward functions. In particular, we apply saliency
methods to identify failure modes and predict the robustness of reward
functions. We find that learned reward functions often implement surprising
algorithms that rely on contingent aspects of the environment. We also discover
that existing interpretability techniques often attend to irrelevant changes in
reward output, suggesting that reward interpretability may need significantly
different methods from policy interpretability.

negative:
A long-term goal of machine learning is to build intelligent conversational
agents. One recent popular approach is to train end-to-end models on a large
amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals
& Le, 2015; Shang et al., 2015). However, this approach leaves many questions
unanswered as an understanding of the precise successes and shortcomings of
each model is hard to assess. A contrasting recent proposal are the bAbI tasks
(Weston et al., 2015b) which are synthetic data that measure the ability of
learning machines at various reasoning tasks over toy language. Unfortunately,
those tests are very small and hence may encourage methods that do not scale.
In this work, we propose a suite of new tasks of a much larger scale that
attempt to bridge the gap between the two regimes. Choosing the domain of
movies, we provide tasks that test the ability of models to answer factual
questions (utilizing OMDB), provide personalization (utilizing MovieLens),
carry short conversations about the two, and finally to perform on natural
dialogs from Reddit. We provide a dataset covering 75k movie entities and with
3.5M training examples. We present results of various models on these tasks,
and evaluate their performance.

negative:
Determinantal point processes (DPPs) are well known models for diverse subset
selection problems, including recommendation tasks, document summarization and
image search. In this paper, we discuss a greedy deterministic adaptation of
k-DPP. Deterministic algorithms are interesting for many applications, as they
provide interpretability to the user by having no failure probability and
always returning the same results. First, the ability of the method to yield
low-rank approximations of kernel matrices is evaluated by comparing the
accuracy of the Nystr\"om approximation on multiple datasets. Afterwards, we
demonstrate the usefulness of the model on an image search task.

negative:
Along with the advance of opinion mining techniques, public mood has been
found to be a key element for stock market prediction. However, how market
participants' behavior is affected by public mood has been rarely discussed.
Consequently, there has been little progress in leveraging public mood for the
asset allocation problem, which is preferred in a trusted and interpretable
way. In order to address the issue of incorporating public mood analyzed from
social media, we propose to formalize public mood into market views, because
market views can be integrated into the modern portfolio theory. In our
framework, the optimal market views will maximize returns in each period with a
Bayesian asset allocation model. We train two neural models to generate the
market views, and benchmark the model performance on other popular asset
allocation strategies. Our experimental results suggest that the formalization
of market views significantly increases the profitability (5% to 10% annually)
of the simulated portfolio at a given risk level.

negative:
The primary goal of ad-hoc retrieval (document retrieval in the context of
question answering) is to find relevant documents satisfied the information
need posted in a natural language query. It requires a good understanding of
the query and all the documents in a corpus, which is difficult because the
meaning of natural language texts depends on the context, syntax,and semantics.
Recently deep neural networks have been used to rank search results in response
to a query. In this paper, we devise a multi-resolution neural network(MRNN) to
leverage the whole hierarchy of representations for document retrieval. The
proposed MRNN model is capable of deriving a representation that integrates
representations of different levels of abstraction from all the layers of the
learned hierarchical representation.Moreover, a duplex attention component is
designed to refinethe multi-resolution representation so that an optimal
contextfor matching the query and document can be determined. More
specifically, the first attention mechanism determines optimal context from the
learned multi-resolution representation for the query and document. The latter
attention mechanism aims to fine-tune the representation so that the query and
the relevant document are closer in proximity. The empirical study shows that
MRNN with the duplex attention is significantly superior to existing models
used for ad-hoc retrieval on benchmark datasets including SQuAD, WikiQA,
QUASAR, and TrecQA.

negative:
Sentence position is a strong feature for news summarization, since the lead
often (but not always) summarizes the key points of the article. In this paper,
we show that recent neural systems excessively exploit this trend, which
although powerful for many inputs, is also detrimental when summarizing
documents where important content should be extracted from later parts of the
article. We propose two techniques to make systems sensitive to the importance
of content in different parts of the article. The first technique employs
'unbiased' data; i.e., randomly shuffled sentences of the source document, to
pretrain the model. The second technique uses an auxiliary ROUGE-based loss
that encourages the model to distribute importance scores throughout a document
by mimicking sentence-level ROUGE scores on the training data. We show that
these techniques significantly improve the performance of a competitive
reinforcement learning based extractive system, with the auxiliary loss being
more powerful than pretraining.

negative:
A character network is a graph extracted from a narrative, in which vertices
represent characters and edges correspond to interactions between them. A
number of narrative-related problems can be addressed automatically through the
analysis of character networks, such as summarization, classification, or role
detection. Character networks are particularly relevant when considering works
of fictions (e.g. novels, plays, movies, TV series), as their exploitation
allows developing information retrieval and recommendation systems. However,
works of fiction possess specific properties making these tasks harder. This
survey aims at presenting and organizing the scientific literature related to
the extraction of character networks from works of fiction, as well as their
analysis. We first describe the extraction process in a generic way, and
explain how its constituting steps are implemented in practice, depending on
the medium of the narrative, the goal of the network analysis, and other
factors. We then review the descriptive tools used to characterize character
networks, with a focus on the way they are interpreted in this context. We
illustrate the relevance of character networks by also providing a review of
applications derived from their analysis. Finally, we identify the limitations
of the existing approaches, and the most promising perspectives.

negative:
We present a novel model for the problem of ranking a collection of documents
according to their semantic similarity to a source (query) document. While the
problem of document-to-document similarity ranking has been studied, most
modern methods are limited to relatively short documents or rely on the
existence of "ground-truth" similarity labels. Yet, in most common real-world
cases, similarity ranking is an unsupervised problem as similarity labels are
unavailable. Moreover, an ideal model should not be restricted by documents'
length. Hence, we introduce SDR, a self-supervised method for document
similarity that can be applied to documents of arbitrary length. Importantly,
SDR can be effectively applied to extremely long documents, exceeding the 4,096
maximal token limits of Longformer. Extensive evaluations on large document
datasets show that SDR significantly outperforms its alternatives across all
metrics. To accelerate future research on unlabeled long document similarity
ranking, and as an additional contribution to the community, we herein publish
two human-annotated test sets of long documents similarity evaluation. The SDR
code and datasets are publicly available.

negative:
Additive Manufacturing (AM) is a crucial component of the smart industry. In
this paper, we propose an automated quality grading system for the AM process
using a deep convolutional neural network (CNN) model. The CNN model is trained
offline using the images of the internal and surface defects in the
layer-by-layer deposition of materials and tested online by studying the
performance of detecting and classifying the failure in AM process at different
extruder speeds and temperatures. The model demonstrates the accuracy of 94%
and specificity of 96%, as well as above 75% in three classifier measures of
the Fscore, the sensitivity, and precision for classifying the quality of the
printing process in five grades in real-time. The proposed online model adds an
automated, consistent, and non-contact quality control signal to the AM process
that eliminates the manual inspection of parts after they are entirely built.
The quality monitoring signal can also be used by the machine to suggest
remedial actions by adjusting the parameters in real-time. The proposed quality
predictive model serves as a proof-of-concept for any type of AM machines to
produce reliable parts with fewer quality hiccups while limiting the waste of
both time and materials.

negative:
Text Mining is a field that aims at extracting information from textual data.
One of the challenges of such field of study comes from the pre-processing
stage in which a vector (and structured) representation should be extracted
from unstructured data. The common extraction creates large and sparse vectors
representing the importance of each term to a document. As such, this usually
leads to the curse-of-dimensionality that plagues most machine learning
algorithms. To cope with this issue, in this paper we propose a new supervised
feature extraction and reduction algorithm, named DCDistance, that creates
features based on the distance between a document to a representative of each
class label. As such, the proposed technique can reduce the features set in
more than 99% of the original set. Additionally, this algorithm was also
capable of improving the classification accuracy over a set of benchmark
datasets when compared to traditional and state-of-the-art features selection
algorithms.

negative:
Document AI, or Document Intelligence, is a relatively new research topic
that refers to the techniques for automatically reading, understanding, and
analyzing business documents. It is an important research direction for natural
language processing and computer vision. In recent years, the popularity of
deep learning technology has greatly advanced the development of Document AI,
such as document layout analysis, visual information extraction, document
visual question answering, document image classification, etc. This paper
briefly reviews some of the representative models, tasks, and benchmark
datasets. Furthermore, we also introduce early-stage heuristic rule-based
document analysis, statistical machine learning algorithms, and deep learning
approaches especially pre-training methods. Finally, we look into future
directions for Document AI research.

negative:
This paper discusses learning a structured feedback control to obtain
sufficient robustness to exogenous inputs for linear dynamic systems with
unknown state matrix. The structural constraint on the controller is necessary
for many cyber-physical systems, and our approach presents a design for any
generic structure, paving the way for distributed learning control. The ideas
from reinforcement learning (RL) in conjunction with control-theoretic
sufficient stability and performance guarantees are used to develop the
methodology. First, a model-based framework is formulated using dynamic
programming to embed the structural constraint in the linear quadratic
regulator (LQR) setting along with sufficient robustness conditions.
Thereafter, we translate these conditions to a data-driven learning-based
framework - robust structured reinforcement learning (RSRL) that enjoys the
control-theoretic guarantees on stability and convergence. We validate our
theoretical results with a simulation on a multi-agent network with 6 agents.

negative:
As a new type of e-commerce platform developed in recent years, local
consumer service platform provides users with software to consume service to
the nearby store or to the home, such as Groupon and Koubei. Different from
other common e-commerce platforms, the behavior of users on the local consumer
service platform is closely related to their real-time local context
information. Therefore, building a context-aware user behavior prediction
system is able to provide both merchants and users better service in local
consumer service platforms. However, most of the previous work just treats the
contextual information as an ordinary feature into the prediction model to
obtain the prediction list under a specific context, which ignores the fact
that the interest of a user in different contexts is often significantly
different. Hence, in this paper, we propose a context-aware heterogeneous graph
attention network (CHGAT) to dynamically generate the representation of the
user and to estimate the probability for future behavior. Specifically, we
first construct the meta-path based heterogeneous graphs with the historical
behaviors from multiple sources and comprehend heterogeneous vertices in the
graph with a novel unified knowledge representing approach. Next, a multi-level
attention mechanism is introduced for context-aware aggregation with graph
vertices, which contains the vertex-level attention network and the path-level
attention network. Both of them aim to capture the semantic correlation between
information contained in the graph and the outside real-time contextual
information in the search system. Then the model proposed in this paper
aggregates specific graphs with their corresponding context features and
obtains the representation of user interest under a specific context and input
it into the prediction network to finally obtain the predicted probability of
user behavior.

negative:
Deciding which scripts to turn into movies is a costly and time-consuming
process for filmmakers. Thus, building a tool to aid script selection, an
initial phase in movie production, can be very beneficial. Toward that goal, in
this work, we present a method to evaluate the quality of a screenplay based on
linguistic cues. We address this in a two-fold approach: (1) we define the task
as predicting nominations of scripts at major film awards with the hypothesis
that the peer-recognized scripts should have a greater chance to succeed. (2)
based on industry opinions and narratology, we extract and integrate
domain-specific features into common classification techniques. We face two
challenges (1) scripts are much longer than other document datasets (2)
nominated scripts are limited and thus difficult to collect. However, with
narratology-inspired modeling and domain features, our approach offers clear
improvements over strong baselines. Our work provides a new approach for future
work in screenplay analysis.

negative:
AI systems often rely on two key components: a specified goal or reward
function and an optimization algorithm to compute the optimal behavior for that
goal. This approach is intended to provide value for a principal: the user on
whose behalf the agent acts. The objectives given to these agents often refer
to a partial specification of the principal's goals. We consider the cost of
this incompleteness by analyzing a model of a principal and an agent in a
resource constrained world where the $L$ attributes of the state correspond to
different sources of utility for the principal. We assume that the reward
function given to the agent only has support on $J < L$ attributes. The
contributions of our paper are as follows: 1) we propose a novel model of an
incomplete principal-agent problem from artificial intelligence; 2) we provide
necessary and sufficient conditions under which indefinitely optimizing for any
incomplete proxy objective leads to arbitrarily low overall utility; and 3) we
show how modifying the setup to allow reward functions that reference the full
state or allowing the principal to update the proxy objective over time can
lead to higher utility solutions. The results in this paper argue that we
should view the design of reward functions as an interactive and dynamic
process and identifies a theoretical scenario where some degree of
interactivity is desirable.

negative:
Reward functions are easy to misspecify; although designers can make
corrections after observing mistakes, an agent pursuing a misspecified reward
function can irreversibly change the state of its environment. If that change
precludes optimization of the correctly specified reward function, then
correction is futile. For example, a robotic factory assistant could break
expensive equipment due to a reward misspecification; even if the designers
immediately correct the reward function, the damage is done. To mitigate this
risk, we introduce an approach that balances optimization of the primary reward
function with preservation of the ability to optimize auxiliary reward
functions. Surprisingly, even when the auxiliary reward functions are randomly
generated and therefore uninformative about the correctly specified reward
function, this approach induces conservative, effective behavior.

negative:
Reward is the driving force for reinforcement-learning agents. This paper is
dedicated to understanding the expressivity of reward as a way to capture tasks
that we would want an agent to perform. We frame this study around three new
abstract notions of "task" that might be desirable: (1) a set of acceptable
behaviors, (2) a partial ordering over behaviors, or (3) a partial ordering
over trajectories. Our main results prove that while reward can express many of
these tasks, there exist instances of each task type that no Markov reward
function can capture. We then provide a set of polynomial-time algorithms that
construct a Markov reward function that allows an agent to optimize tasks of
each of these three types, and correctly determine when no such reward function
exists. We conclude with an empirical study that corroborates and illustrates
our theoretical findings.

negative:
We present an empirical investigation of pre-trained Transformer-based
auto-regressive language models for the task of open-domain dialogue
generation. Training paradigm of pre-training and fine-tuning is employed to
conduct the parameter learning. Corpora of News and Wikipedia in Chinese and
English are collected for the pre-training stage respectively. Dialogue context
and response are concatenated into a single sequence utilized as the input of
the models during the fine-tuning stage. A weighted joint prediction paradigm
for both context and response is designed to evaluate the performance of models
with or without the loss term for context prediction. Various of decoding
strategies such as greedy search, beam search, top-k sampling, etc. are
employed to conduct the response text generation. Extensive experiments are
conducted on the typical single-turn and multi-turn dialogue corpora such as
Weibo, Douban, Reddit, DailyDialog, and Persona-Chat. Detailed numbers of
automatic evaluation metrics on relevance and diversity of the generated
results for the languages models as well as the baseline approaches are
reported.

negative:
While most conversational AI systems focus on textual dialogue only,
conditioning utterances on visual context (when it's available) can lead to
more realistic conversations. Unfortunately, a major challenge for
incorporating visual context into conversational dialogue is the lack of
large-scale labeled datasets. We provide a solution in the form of a new
visually conditioned Future Utterance Prediction task. Our task involves
predicting the next utterance in a video, using both visual frames and
transcribed speech as context. By exploiting the large number of instructional
videos online, we train a model to solve this task at scale, without the need
for manual annotations. Leveraging recent advances in multimodal learning, our
model consists of a novel co-attentional multimodal video transformer, and when
trained on both textual and visual context, outperforms baselines that use
textual inputs alone. Further, we demonstrate that our model trained for this
task on unlabelled videos achieves state-of-the-art performance on a number of
downstream VideoQA benchmarks such as MSRVTT-QA, MSVD-QA, ActivityNet-QA and
How2QA.

negative:
Most popular goal-oriented dialogue agents are capable of understanding the
conversational context. However, with the surge of virtual assistants with
screen, the next generation of agents are required to also understand screen
context in order to provide a proper interactive experience, and better
understand users' goals. In this paper, we propose a novel multimodal
conversational framework, where the dialogue agent's next action and their
arguments are derived jointly conditioned both on the conversational and the
visual context. Specifically, we propose a new model, that can reason over the
visual context within a conversation and populate API arguments with visual
entities given the user query. Our model can recognize visual features such as
color and shape as well as the metadata based features such as price or star
rating associated with a visual entity. In order to train our model, due to a
lack of suitable multimodal conversational datasets, we also propose a novel
multimodal dialog simulator to generate synthetic data and also collect
realistic user data from MTurk to improve model robustness. The proposed model
achieves a reasonable 85% model accuracy, without high inference latency. We
also demonstrate the proposed approach in a prototypical furniture shopping
experience for a multimodal virtual assistant.

negative:
Based on recent advances in realistic language modeling (GPT-3) and
cross-modal representations (CLIP), Gaud\'i was developed to help designers
search for inspirational images using natural language. In the early stages of
the design process, with the goal of eliciting a client's preferred creative
direction, designers will typically create thematic collections of
inspirational images called "mood-boards". Creating a mood-board involves
sequential image searches which are currently performed using keywords or
images. Gaud\'i transforms this process into a conversation where the user is
gradually detailing the mood-board's theme. This representation allows our AI
to generate new search queries from scratch, straight from a project briefing,
following a theme hypothesized by GPT-3. Compared to previous computational
approaches to mood-board creation, to the best of our knowledge, ours is the
first attempt to represent mood-boards as the stories that designers tell when
presenting a creative direction to a client.

negative:
Recent studies emphasize the need of document context in human evaluation of
machine translations, but little research has been done on the impact of user
interfaces on annotator productivity and the reliability of assessments. In
this work, we compare human assessment data from the last two WMT evaluation
campaigns collected via two different methods for document-level evaluation.
Our analysis shows that a document-centric approach to evaluation where the
annotator is presented with the entire document context on a screen leads to
higher quality segment and document level assessments. It improves the
correlation between segment and document scores and increases inter-annotator
agreement for document scores but is considerably more time consuming for
annotators.

negative:
In the context of investigative journalism, we address the problem of
automatically identifying which claims in a given document are most worthy and
should be prioritized for fact-checking. Despite its importance, this is a
relatively understudied problem. Thus, we create a new dataset of political
debates, containing statements that have been fact-checked by nine reputable
sources, and we train machine learning models to predict which claims should be
prioritized for fact-checking, i.e., we model the problem as a ranking task.
Unlike previous work, which has looked primarily at sentences in isolation, in
this paper we focus on a rich input representation modeling the context:
relationship between the target statement and the larger context of the debate,
interaction between the opponents, and reaction by the moderator and by the
public. Our experiments show state-of-the-art results, outperforming a strong
rivaling system by a margin, while also confirming the importance of the
contextual information.

negative:
Automatic question generation is an important technique that can improve the
training of question answering, help chatbots to start or continue a
conversation with humans, and provide assessment materials for educational
purposes. Existing neural question generation models are not sufficient mainly
due to their inability to properly model the process of how each word in the
question is selected, i.e., whether repeating the given passage or being
generated from a vocabulary. In this paper, we propose our Clue Guided Copy
Network for Question Generation (CGC-QG), which is a sequence-to-sequence
generative model with copying mechanism, yet employing a variety of novel
components and techniques to boost the performance of question generation. In
CGC-QG, we design a multi-task labeling strategy to identify whether a question
word should be copied from the input passage or be generated instead, guiding
the model to learn the accurate boundaries between copying and generation.
Furthermore, our input passage encoder takes as input, among a diverse range of
other features, the prediction made by a clue word predictor, which helps
identify whether each word in the input passage is a potential clue to be
copied into the target question. The clue word predictor is designed based on a
novel application of Graph Convolutional Networks onto a syntactic dependency
tree representation of each passage, thus being able to predict clue words only
based on their context in the passage and their relative positions to the
answer in the tree. We jointly train the clue prediction as well as question
generation with multi-task learning and a number of practical strategies to
reduce the complexity. Extensive evaluations show that our model significantly
improves the performance of question generation and out-performs all previous
state-of-the-art neural question generation models by a substantial margin.

negative:
With the increase of information, document classification as one of the
methods of text mining, plays vital role in many management and organizing
information. Document classification is the process of assigning a document to
one or more predefined category labels. Document classification includes
different parts such as text processing, term selection, term weighting and
final classification. The accuracy of document classification is very
important. Thus improvement in each part of classification should lead to
better results and higher precision. Term weighting has a great impact on the
accuracy of the classification. Most of the existing weighting methods exploit
the statistical information of terms in documents and do not consider semantic
relations between words. In this paper, an automated document classification
system is presented that uses a novel term weighting method based on semantic
relations between terms. To evaluate the proposed method, three standard
Persian corpuses are used. Experiment results show 2 to 4 percent improvement
in classification accuracy compared with the best previous designed system for
Persian documents.

negative:
The creation of fictional stories is a very complex task that usually implies
a creative process where the author has to combine characters, conflicts and
plots to create an engaging narrative. This work presents a simulated
environment with hundreds of characters that allows the study of coherent and
interesting literary archetypes (or behaviours), plots and sub-plots. We will
use this environment to perform a study about the number of profiles
(parameters that define the personality of a character) needed to create two
emergent scenes of archetypes: "natality control" and "revenge". A Genetic
Algorithm (GA) will be used to find the fittest number of profiles and
parameter configuration that enables the existence of the desired archetypes
(played by the characters without their explicit knowledge). The results show
that parametrizing this complex system is possible and that these kind of
archetypes can emerge in the given environment.

negative:
Term frequency normalization is a serious issue since lengths of documents
are various. Generally, documents become long due to two different reasons -
verbosity and multi-topicality. First, verbosity means that the same topic is
repeatedly mentioned by terms related to the topic, so that term frequency is
more increased than the well-summarized one. Second, multi-topicality indicates
that a document has a broad discussion of multi-topics, rather than single
topic. Although these document characteristics should be differently handled,
all previous methods of term frequency normalization have ignored these
differences and have used a simplified length-driven approach which decreases
the term frequency by only the length of a document, causing an unreasonable
penalization. To attack this problem, we propose a novel TF normalization
method which is a type of partially-axiomatic approach. We first formulate two
formal constraints that the retrieval model should satisfy for documents having
verbose and multi-topicality characteristic, respectively. Then, we modify
language modeling approaches to better satisfy these two constraints, and
derive novel smoothing methods. Experimental results show that the proposed
method increases significantly the precision for keyword queries, and
substantially improves MAP (Mean Average Precision) for verbose queries.

negative:
We study a security threat to reinforcement learning where an attacker
poisons the learning environment to force the agent into executing a target
policy chosen by the attacker. As a victim, we consider RL agents whose
objective is to find a policy that maximizes reward in infinite-horizon problem
settings. The attacker can manipulate the rewards and the transition dynamics
in the learning environment at training-time, and is interested in doing so in
a stealthy manner. We propose an optimization framework for finding an optimal
stealthy attack for different measures of attack cost. We provide lower/upper
bounds on the attack cost, and instantiate our attacks in two settings: (i) an
offline setting where the agent is doing planning in the poisoned environment,
and (ii) an online setting where the agent is learning a policy with poisoned
feedback. Our results show that the attacker can easily succeed in teaching any
target policy to the victim under mild conditions and highlight a significant
security threat to reinforcement learning agents in practice.

negative:
Autonomous agents optimize the reward function we give them. What they don't
know is how hard it is for us to design a reward function that actually
captures what we want. When designing the reward, we might think of some
specific training scenarios, and make sure that the reward will lead to the
right behavior in those scenarios. Inevitably, agents encounter new scenarios
(e.g., new types of terrain) where optimizing that same reward may lead to
undesired behavior. Our insight is that reward functions are merely
observations about what the designer actually wants, and that they should be
interpreted in the context in which they were designed. We introduce inverse
reward design (IRD) as the problem of inferring the true objective based on the
designed reward and the training MDP. We introduce approximate methods for
solving IRD problems, and use their solution to plan risk-averse behavior in
test MDPs. Empirical results suggest that this approach can help alleviate
negative side effects of misspecified reward functions and mitigate reward
hacking.

negative:
The increasing amount of online content motivated the development of
multi-document summarization methods. In this work, we explore straightforward
approaches to extend single-document summarization methods to multi-document
summarization. The proposed methods are based on the hierarchical combination
of single-document summaries, and achieves state of the art results.

negative:
Most of the textual information available to us are temporally variable. In a
world where information is dynamic, time-stamping them is a very important
task. Documents are a good source of information and are used for many tasks
like, sentiment analysis, classification of reviews etc. The knowledge of
creation date of documents facilitates several tasks like summarization, event
extraction, temporally focused information extraction etc. Unfortunately, for
most of the documents on the web, the time-stamp meta-data is either erroneous
or missing. Thus document dating is a challenging problem which requires
inference over the temporal structure of the document alongside the contextual
information of the document. Prior document dating systems have largely relied
on handcrafted features while ignoring such document-internal structures. In
this paper we propose NeuralDater, a Graph Convolutional Network (GCN) based
document dating approach which jointly exploits syntactic and temporal graph
structures of document in a principled way. We also pointed out some
limitations of NeuralDater and tried to utilize both context and temporal
information in documents in a more flexible and intuitive manner proposing AD3:
Attentive Deep Document Dater, an attention-based document dating system. To
the best of our knowledge these are the first application of deep learning
methods for the task. Through extensive experiments on real-world datasets, we
find that our models significantly outperforms state-of-the-art baselines by a
significant margin.

negative:
It is essential for dialogue-based spatial reasoning systems to maintain
memory of historical states of the world. In addition to conveying that the
dialogue agent is mentally present and engaged with the task, referring to
historical states may be crucial for enabling collaborative planning (e.g., for
planning to return to a previous state, or diagnosing a past misstep). In this
paper, we approach the problem of spatial memory in a multi-modal spoken
dialogue system capable of answering questions about interaction history in a
physical blocks world setting. This work builds upon a full spatial
question-answering pipeline consisting of a vision system, speech input and
output mediated by an animated avatar, a dialogue system that robustly
interprets spatial queries, and a constraint solver that derives answers based
on 3-D spatial modelling. The contributions of this work include a symbolic
dialogue context registering knowledge about discourse history and changes in
the world, as well as a natural language understanding module capable of
interpreting free-form historical questions and querying the dialogue context
to form an answer.

negative:
Mood recognition is an important problem in music informatics and has key
applications in music discovery and recommendation. These applications have
become even more relevant with the rise of music streaming. Our work
investigates the research question of whether we can leverage audio metadata
such as artist and year, which is readily available, to improve the performance
of mood classification models. To this end, we propose a multi-task learning
approach in which a shared model is simultaneously trained for mood and
metadata prediction tasks with the goal to learn richer representations.
Experimentally, we demonstrate that applying our technique on the existing
state-of-the-art convolutional neural networks for mood classification improves
their performances consistently. We conduct experiments on multiple datasets
and report that our approach can lead to improvements in the average precision
metric by up to 8.7 points.

negative:
Probability theory, epistemically interpreted, provides an excellent, if not
the best available account of inductive reasoning. This is so because there are
general and definite rules for the change of subjective probabilities through
information or experience; induction and belief change are one and same topic,
after all. The most basic of these rules is simply to conditionalize with
respect to the information received; and there are similar and more general
rules. 1 Hence, a fundamental reason for the epistemological success of
probability theory is that there at all exists a well-behaved concept of
conditional probability. Still, people have, and have reasons for, various
concerns over probability theory. One of these is my starting point:
Intuitively, we have the notion of plain belief; we believe propositions2 to be
true (or to be false or neither). Probability theory, however, offers no formal
counterpart to this notion. Believing A is not the same as having probability 1
for A, because probability 1 is incorrigible3; but plain belief is clearly
corrigible. And believing A is not the same as giving A a probability larger
than some 1 - c, because believing A and believing B is usually taken to be
equivalent to believing A & B.4 Thus, it seems that the formal representation
of plain belief has to take a non-probabilistic route. Indeed, representing
plain belief seems easy enough: simply represent an epistemic state by the set
of all propositions believed true in it or, since I make the common assumption
that plain belief is deductively closed, by the conjunction of all propositions
believed true in it. But this does not yet provide a theory of induction, i.e.
an answer to the question how epistemic states so represented are changed
tbrough information or experience. There is a convincing partial answer: if the
new information is compatible with the old epistemic state, then the new
epistemic state is simply represented by the conjunction of the new information
and the old beliefs. This answer is partial because it does not cover the quite
common case where the new information is incompatible with the old beliefs. It
is, however, important to complete the answer and to cover this case, too;
otherwise, we would not represent plain belief as conigible. The crucial
problem is that there is no good completion. When epistemic states are
represented simply by the conjunction of all propositions believed true in it,
the answer cannot be completed; and though there is a lot of fruitful work, no
other representation of epistemic states has been proposed, as far as I know,
which provides a complete solution to this problem. In this paper, I want to
suggest such a solution. In [4], I have more fully argued that this is the only
solution, if certain plausible desiderata are to be satisfied. Here, in section
2, I will be content with formally defining and intuitively explaining my
proposal. I will compare my proposal with probability theory in section 3. It
will turn out that the theory I am proposing is structurally homomorphic to
probability theory in important respects and that it is thus equally easily
implementable, but moreover computationally simpler. Section 4 contains a very
brief comparison with various kinds of logics, in particular conditional logic,
with Shackle's functions of potential surprise and related theories, and with
the Dempster - Shafer theory of belief functions.

negative:
Text alignment finds application in tasks such as citation recommendation and
plagiarism detection. Existing alignment methods operate at a single,
predefined level and cannot learn to align texts at, for example, sentence and
document levels. We propose a new learning approach that equips previously
established hierarchical attention encoders for representing documents with a
cross-document attention component, enabling structural comparisons across
different levels (document-to-document and sentence-to-document). Our component
is weakly supervised from document pairs and can align at multiple levels. Our
evaluation on predicting document-to-document relationships and
sentence-to-document relationships on the tasks of citation recommendation and
plagiarism detection shows that our approach outperforms previously established
hierarchical, attention encoders based on recurrent and transformer
contextualization that are unaware of structural correspondence between
documents.

negative:
We investigate the pertinence of methods from algebraic topology for text
data analysis. These methods enable the development of
mathematically-principled isometric-invariant mappings from a set of vectors to
a document embedding, which is stable with respect to the geometry of the
document in the selected metric space. In this work, we evaluate the utility of
these topology-based document representations in traditional NLP tasks,
specifically document clustering and sentiment classification. We find that the
embeddings do not benefit text analysis. In fact, performance is worse than
simple techniques like $\textit{tf-idf}$, indicating that the geometry of the
document does not provide enough variability for classification on the basis of
topic or sentiment in the chosen datasets.

negative:
Conceptual entanglement is a crucial phenomenon in quantum cognition because
it implies that classical probabilities cannot model non--compositional
conceptual phenomena. While several psychological experiments have been
developed to test conceptual entanglement, this has not been explored in the
context of Natural Language Processing. In this paper, we apply the hypothesis
that words of a document are traces of the concepts that a person has in mind
when writing the document. Therefore, if these concepts are entangled, we
should be able to observe traces of their entanglement in the documents. In
particular, we test conceptual entanglement by contrasting language simulations
with results obtained from a text corpus. Our analysis indicates that
conceptual entanglement is strongly linked to the way in which language is
structured. We discuss the implications of this finding in the context of
conceptual modeling and of Natural Language Processing.

negative:
Grammatical tense and mood are important linguistic phenomena to consider in
natural language processing (NLP) research. We consider the correspondence
between English and German tense and mood in translation. Human translators do
not find this correspondence easy, and as we will show through careful
analysis, there are no simplistic ways to map tense and mood from one language
to another. Our observations about the challenges of human translation of tense
and mood have important implications for multilingual NLP. Of particular
importance is the challenge of modeling tense and mood in rule-based,
phrase-based statistical and neural machine translation.

negative:
Today all kind of information is getting digitized and along with all this
digitization, the huge archive of various kinds of documents is being digitized
too. We know that, Optical Character Recognition is the method through which,
newspapers and other paper documents convert into digital resources. But, it is
a fact that this method works on texts only. As a result, if we try to process
any document which contains non-textual zones, then we will get garbage texts
as output. That is why; in order to digitize documents properly they should be
prepossessed carefully. And while preprocessing, segmenting document in
different regions according to the category properly is most important. But,
the Optical Character Recognition processes available for Bangla language have
no such algorithm that can categorize a newspaper/book page fully. So we worked
to decompose a document into its several parts like headlines, sub headlines,
columns, images etc. And if the input is skewed and rotated, then the input was
also deskewed and de-rotated. To decompose any Bangla document we found out the
edges of the input image. Then we find out the horizontal and vertical area of
every pixel where it lies in. Later on the input image was cut according to
these areas. Then we pick each and every sub image and found out their
height-width ratio, line height. Then according to these values the sub images
were categorized. To deskew the image we found out the skew angle and de skewed
the image according to this angle. To de-rotate the image we used the line
height, matra line, pixel ratio of matra line.

negative:
In this paper, I will discuss the work I am currently doing as a Ph.D.
student at the University of Potsdam, under the tutoring of T. Schaub. I'm
currently looking into action description in ASP. More precisely, my goal is to
explore how to represent actions with durations in ASP, in different contexts.
Right now, I'm focused on Multi-Agent Path Finding (MAPF), looking at how to
represent speeds for different agents and contexts.
  Before tackling duration, I wanted to explore and compare different
representations of action taking in ASP. For this, I started comparing
different simple encodings tackling the MAPF problem. Even in simple code,
choices and assumptions have been made in their creations. The objective of my
work is to present the consequences of those design decisions in terms of
performance and knowledge representation. As far as I know, there is no current
research on this topic.
  Besides that, I'm also exploring different ways to represent duration and to
solve related problems. I planed to compare them the same way I described
before. I also want this to help me find innovative and effective ways to solve
problems with duration.

negative:
To train a statistical spoken dialogue system (SDS) it is essential that an
accurate method for measuring task success is available. To date training has
relied on presenting a task to either simulated or paid users and inferring the
dialogue's success by observing whether this presented task was achieved or
not. Our aim however is to be able to learn from real users acting under their
own volition, in which case it is non-trivial to rate the success as any prior
knowledge of the task is simply unavailable. User feedback may be utilised but
has been found to be inconsistent. Hence, here we present two neural network
models that evaluate a sequence of turn-level features to rate the success of a
dialogue. Importantly these models make no use of any prior knowledge of the
user's task. The models are trained on dialogues generated by a simulated user
and the best model is then used to train a policy on-line which is shown to
perform at least as well as a baseline system using prior knowledge of the
user's task. We note that the models should also be of interest for evaluating
SDS and for monitoring a dialogue in rule-based SDS.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
The coordination and cooperation of all the stakeholders involved is a
decisive point for the control and the resolution of problems. In the
insecurity events, the resolution should refer to a plan that defines a general
framework of the procedures to be undertaken and the instructions to be
complied with; also, a more precise process must be defined by the actors to
deal with the case represented by the particular problem of the current
situation. Indeed, this process has to cope with a dynamic, unstable and
unpredictable environment, due to the heterogeneity and multiplicity of
stakeholders, and finally due to their possible geographical distribution. In
this article, we will present the first steps of validation of a collaborative
decision-making approach in the context of crisis situations such as road
accidents. This approach is based on ontologies and multi-agent systems.

negative:
A mind-map is a diagram that represents the central concept and key ideas in
a hierarchical way. Converting plain text into a mind-map will reveal its key
semantic structure and be easier to understand. Given a document, the existing
automatic mind-map generation method extracts the relationships of every
sentence pair to generate the directed semantic graph for this document. The
computation complexity increases exponentially with the length of the document.
Moreover, it is difficult to capture the overall semantics. To deal with the
above challenges, we propose an efficient mind-map generation network that
converts a document into a graph via sequence-to-graph. To guarantee a
meaningful mind-map, we design a graph refinement module to adjust the relation
graph in a reinforcement learning manner. Extensive experimental results
demonstrate that the proposed approach is more effective and efficient than the
existing methods. The inference time is reduced by thousands of times compared
with the existing methods. The case studies verify that the generated mind-maps
better reveal the underlying semantic structures of the document.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.

negative:
We address the problem of solving mixed random linear equations. We have
unlabeled observations coming from multiple linear regressions, and each
observation corresponds to exactly one of the regression models. The goal is to
learn the linear regressors from the observations. Classically, Alternating
Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used
to solve this problem. AM iteratively alternates between the estimation of
labels and solving the regression problems with the estimated labels.
Empirically, it is observed that, for a large variety of non-convex problems
including mixed linear regression, AM converges at a much faster rate compared
to gradient based algorithms. However, the existing theory suggests similar
rate of convergence for AM and gradient based methods, failing to capture this
empirical behavior. In this paper, we close this gap between theory and
practice for the special case of a mixture of $2$ linear regressions. We show
that, provided initialized properly, AM enjoys a \emph{super-linear} rate of
convergence in certain parameter regimes. To the best of our knowledge, this is
the first work that theoretically establishes such rate for AM. Hence, if we
want to recover the unknown regressors upto an error (in $\ell_2$ norm) of
$\epsilon$, AM only takes $\mathcal{O}(\log \log (1/\epsilon))$ iterations.
Furthermore, we compare AM with a gradient based heuristic algorithm
empirically and show that AM dominates in iteration complexity as well as
wall-clock time.