query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I need a literature review of the clinician system for a project I am planning to do. I am specifically interested in the application of reinforcement learning (RL) in the clinician system and identify the benefits of RL for the healthcare research community. I hope the literature review can contain detailed exploration of the findings from the literature, including an explanation of the clinician system, how intensive care data can be utilized, and how reinforcement learning can help to make treatment recommendations. I hope the literature review also compares the differences of different RL algorithms used in the clinician system field, and finds out the one that is most efficient under large amounts of data.

positive:
The rapid increase in the percentage of chronic disease patients along with
the recent pandemic pose immediate threats on healthcare expenditure and
elevate causes of death. This calls for transforming healthcare systems away
from one-on-one patient treatment into intelligent health systems, to improve
services, access and scalability, while reducing costs. Reinforcement Learning
(RL) has witnessed an intrinsic breakthrough in solving a variety of complex
problems for diverse applications and services. Thus, we conduct in this paper
a comprehensive survey of the recent models and techniques of RL that have been
developed/used for supporting Intelligent-healthcare (I-health) systems. This
paper can guide the readers to deeply understand the state-of-the-art regarding
the use of RL in the context of I-health. Specifically, we first present an
overview for the I-health systems challenges, architecture, and how RL can
benefit these systems. We then review the background and mathematical modeling
of different RL, Deep RL (DRL), and multi-agent RL models. After that, we
provide a deep literature review for the applications of RL in I-health
systems. In particular, three main areas have been tackled, i.e., edge
intelligence, smart core network, and dynamic treatment regimes. Finally, we
highlight emerging challenges and outline future research directions in driving
the future success of RL in I-health systems, which opens the door for
exploring some interesting and unsolved problems.

positive:
Dynamic treatment recommendation systems based on large-scale electronic
health records (EHRs) become a key to successfully improve practical clinical
outcomes. Prior relevant studies recommend treatments either use supervised
learning (e.g. matching the indicator signal which denotes doctor
prescriptions), or reinforcement learning (e.g. maximizing evaluation signal
which indicates cumulative reward from survival rates). However, none of these
studies have considered to combine the benefits of supervised learning and
reinforcement learning. In this paper, we propose Supervised Reinforcement
Learning with Recurrent Neural Network (SRL-RNN), which fuses them into a
synergistic learning framework. Specifically, SRL-RNN applies an off-policy
actor-critic framework to handle complex relations among multiple medications,
diseases and individual characteristics. The "actor" in the framework is
adjusted by both the indicator signal and evaluation signal to ensure effective
prescription and low mortality. RNN is further utilized to solve the
Partially-Observed Markov Decision Process (POMDP) problem due to the lack of
fully observed states in real world applications. Experiments on the publicly
real-world dataset, i.e., MIMIC-3, illustrate that our model can reduce the
estimated mortality, while providing promising accuracy in matching doctors'
prescriptions.

positive:
Although reinforcement learning (RL) has tremendous success in many fields,
applying RL to real-world settings such as healthcare is challenging when the
reward is hard to specify and no exploration is allowed. In this work, we focus
on recovering clinicians' rewards in treating patients. We incorporate the
what-if reasoning to explain the clinician's treatments based on their
potential future outcomes. We use generalized additive models (GAMs) - a class
of accurate, interpretable models - to recover the reward. In both simulation
and a real-world hospital dataset, we show our model outperforms baselines.
Finally, our model's explanations match several clinical guidelines when
treating patients while we found the commonly-used linear model often
contradicts them.

positive:
Reinforcement Learning (RL) has recently been applied to sequential
estimation and prediction problems identifying and developing hypothetical
treatment strategies for septic patients, with a particular focus on offline
learning with observational data. In practice, successful RL relies on
informative latent states derived from sequential observations to develop
optimal treatment strategies. To date, how best to construct such states in a
healthcare setting is an open question. In this paper, we perform an empirical
study of several information encoding architectures using data from septic
patients in the MIMIC-III dataset to form representations of a patient state.
We evaluate the impact of representation dimension, correlations with
established acuity scores, and the treatment policies derived from them. We
find that sequentially formed state representations facilitate effective policy
learning in batch settings, validating a more thoughtful approach to
representation learning that remains faithful to the sequential and partial
nature of healthcare data.

positive:
Many healthcare decisions involve navigating through a multitude of treatment
options in a sequential and iterative manner to find an optimal treatment
pathway with the goal of an optimal patient outcome. Such optimization problems
may be amenable to reinforcement learning. A reinforcement learning agent could
be trained to provide treatment recommendations for physicians, acting as a
decision support tool. However, a number of difficulties arise when using RL
beyond benchmark environments, such as specifying the reward function, choosing
an appropriate state representation and evaluating the learned policy.

positive:
As a subfield of machine learning, reinforcement learning (RL) aims at
empowering one's capabilities in behavioural decision making by using
interaction experience with the world and an evaluative feedback. Unlike
traditional supervised learning methods that usually rely on one-shot,
exhaustive and supervised reward signals, RL tackles with sequential decision
making problems with sampled, evaluative and delayed feedback simultaneously.
Such distinctive features make RL technique a suitable candidate for developing
powerful solutions in a variety of healthcare domains, where diagnosing
decisions or treatment regimes are usually characterized by a prolonged and
sequential procedure. This survey discusses the broad applications of RL
techniques in healthcare domains, in order to provide the research community
with systematic understanding of theoretical foundations, enabling methods and
techniques, existing challenges, and new insights of this emerging paradigm. By
first briefly examining theoretical foundations and key techniques in RL
research from efficient and representational directions, we then provide an
overview of RL applications in healthcare domains ranging from dynamic
treatment regimes in chronic diseases and critical care, automated medical
diagnosis from both unstructured and structured clinical data, as well as many
other control or scheduling domains that have infiltrated many aspects of a
healthcare system. Finally, we summarize the challenges and open issues in
current research, and point out some potential solutions and directions for
future research.

negative:
Our aim is to establish a framework where reinforcement learning (RL) of
optimizing interventions retrospectively allows us a regulatory compliant
pathway to prospective clinical testing of the learned policies in a clinical
deployment. We focus on infections in intensive care units which are one of the
major causes of death and difficult to treat because of the complex and opaque
patient dynamics, and the clinically debated, highly-divergent set of
intervention policies required by each individual patient, yet intensive care
units are naturally data rich. In our work, we build on RL approaches in
healthcare ("AI Clinicians"), and learn off-policy continuous dosing policy of
pharmaceuticals for sepsis treatment using historical intensive care data under
partially observable MDPs (POMDPs). POMPDs capture uncertainty in patient state
better by taking in all historical information, yielding an efficient
representation, which we investigate through ablations. We compensate for the
lack of exploration in our retrospective data by evaluating each encountered
state with a best-first tree search. We mitigate state distributional shift by
optimizing our policy in the vicinity of the clinicians' compound policy.
Crucially, we evaluate our model recommendations using not only conventional
policy evaluations but a novel framework that incorporates human experts: a
model-agnostic pre-clinical evaluation method to estimate the accuracy and
uncertainty of clinician's decisions versus our system recommendations when
confronted with the same individual patient history ("shadow mode").

negative:
Off-policy reinforcement learning enables near-optimal policy from suboptimal
experience, thereby provisions opportunity for artificial intelligence
applications in healthcare. Previous works have mainly framed patient-clinician
interactions as Markov decision processes, while true physiological states are
not necessarily fully observable from clinical data. We capture this situation
with partially observable Markov decision process, in which an agent optimises
its actions in a belief represented as a distribution of patient states
inferred from individual history trajectories. A Gaussian mixture model is
fitted for the observed data. Moreover, we take into account the fact that
nuance in pharmaceutical dosage could presumably result in significantly
different effect by modelling a continuous policy through a Gaussian
approximator directly in the policy space, i.e. the actor. To address the
challenge of infinite number of possible belief states which renders exact
value iteration intractable, we evaluate and plan for only every encountered
belief, through heuristic search tree by tightly maintaining lower and upper
bounds of the true value of belief. We further resort to function
approximations to update value bounds estimation, i.e. the critic, so that the
tree search can be improved through more compact bounds at the fringe nodes
that will be back-propagated to the root. Both actor and critic parameters are
learned via gradient-based approaches. Our proposed policy trained from real
intensive care unit data is capable of dictating dosing on vasopressors and
intravenous fluids for sepsis patients that lead to the best patient outcomes.

negative:
This paper presents the first deep reinforcement learning (DRL) framework to
estimate the optimal Dynamic Treatment Regimes from observational medical data.
This framework is more flexible and adaptive for high dimensional action and
state spaces than existing reinforcement learning methods to model real-life
complexity in heterogeneous disease progression and treatment choices, with the
goal of providing doctor and patients the data-driven personalized decision
recommendations. The proposed DRL framework comprises (i) a supervised learning
step to predict the most possible expert actions, and (ii) a deep reinforcement
learning step to estimate the long-term value function of Dynamic Treatment
Regimes. Both steps depend on deep neural networks.
  As a key motivational example, we have implemented the proposed framework on
a data set from the Center for International Bone Marrow Transplant Research
(CIBMTR) registry database, focusing on the sequence of prevention and
treatments for acute and chronic graft versus host disease after
transplantation. In the experimental results, we have demonstrated promising
accuracy in predicting human experts' decisions, as well as the high expected
reward function in the DRL-based dynamic treatment regimes.

negative:
A large set of the explainable Artificial Intelligence (XAI) literature is
emerging on feature relevance techniques to explain a deep neural network (DNN)
output or explaining models that ingest image source data. However, assessing
how XAI techniques can help understand models beyond classification tasks, e.g.
for reinforcement learning (RL), has not been extensively studied. We review
recent works in the direction to attain Explainable Reinforcement Learning
(XRL), a relatively new subfield of Explainable Artificial Intelligence,
intended to be used in general public applications, with diverse audiences,
requiring ethical, responsible and trustable algorithms. In critical situations
where it is essential to justify and explain the agent's behaviour, better
explainability and interpretability of RL models could help gain scientific
insight on the inner workings of what is still considered a black box. We
evaluate mainly studies directly linking explainability to RL, and split these
into two categories according to the way the explanations are generated:
transparent algorithms and post-hoc explainaility. We also review the most
prominent XAI works from the lenses of how they could potentially enlighten the
further deployment of the latest advances in RL, in the demanding present and
future of everyday problems.

negative:
This article surveys reinforcement learning approaches in social robotics.
Reinforcement learning is a framework for decision-making problems in which an
agent interacts through trial-and-error with its environment to discover an
optimal behavior. Since interaction is a key component in both reinforcement
learning and social robotics, it can be a well-suited approach for real-world
interactions with physically embodied social robots. The scope of the paper is
focused particularly on studies that include social physical robots and
real-world human-robot interactions with users. We present a thorough analysis
of reinforcement learning approaches in social robotics. In addition to a
survey, we categorize existent reinforcement learning approaches based on the
used method and the design of the reward mechanisms. Moreover, since
communication capability is a prominent feature of social robots, we discuss
and group the papers based on the communication medium used for reward
formulation. Considering the importance of designing the reward function, we
also provide a categorization of the papers based on the nature of the reward.
This categorization includes three major themes: interactive reinforcement
learning, intrinsically motivated methods, and task performance-driven methods.
The benefits and challenges of reinforcement learning in social robotics,
evaluation methods of the papers regarding whether or not they use subjective
and algorithmic measures, a discussion in the view of real-world reinforcement
learning challenges and proposed solutions, the points that remain to be
explored, including the approaches that have thus far received less attention
is also given in the paper. Thus, this paper aims to become a starting point
for researchers interested in using and applying reinforcement learning methods
in this particular research field.

negative:
Healthcare data suffers from both noise and lack of ground truth. The cost of
data increases as it is cleaned and annotated in healthcare. Unlike other data
sets, medical data annotation, which is critical to accurate ground truth,
requires medical domain expertise for a better patient outcome. In this work,
we report on the use of reinforcement learning to mimic the decision making
process of annotators for medical events, to automate annotation and labelling.
The reinforcement agent learns to annotate alarm data based on annotations done
by an expert. Our method shows promising results on medical alarm data sets. We
trained DQN and A2C agents using the data from monitoring devices annotated by
an expert. Initial results from these RL agents learning the expert annotation
behavior are promising. The A2C agent performs better in terms of learning the
sparse events in a given state, thereby choosing more right actions compared to
DQN agent. To the best of our knowledge, this is the first reinforcement
learning application for the automation of medical events annotation, which has
far-reaching practical use.

negative:
Reinforcement learning (RL) has the potential to significantly improve
clinical decision making. However, treatment policies learned via RL from
observational data are sensitive to subtle choices in study design. We
highlight a simple approach, trajectory inspection, to bring clinicians into an
iterative design process for model-based RL studies. We identify where the
model recommends unexpectedly aggressive treatments or expects surprisingly
positive outcomes from its recommendations. Then, we examine clinical
trajectories simulated with the learned model and policy alongside the actual
hospital course. Applying this approach to recent work on RL for sepsis
management, we uncover a model bias towards discharge, a preference for high
vasopressor doses that may be linked to small sample sizes, and clinically
implausible expectations of discharge without weaning off vasopressors. We hope
that iterations of detecting and addressing the issues unearthed by our method
will result in RL policies that inspire more confidence in deployment.

negative:
Much attention has been devoted recently to the development of machine
learning algorithms with the goal of improving treatment policies in
healthcare. Reinforcement learning (RL) is a sub-field within machine learning
that is concerned with learning how to make sequences of decisions so as to
optimize long-term effects. Already, RL algorithms have been proposed to
identify decision-making strategies for mechanical ventilation, sepsis
management and treatment of schizophrenia. However, before implementing
treatment policies learned by black-box algorithms in high-stakes clinical
decision problems, special care must be taken in the evaluation of these
policies.
  In this document, our goal is to expose some of the subtleties associated
with evaluating RL algorithms in healthcare. We aim to provide a conceptual
starting point for clinical and computational researchers to ask the right
questions when designing and evaluating algorithms for new ways of treating
patients. In the following, we describe how choices about how to summarize a
history, variance of statistical estimators, and confounders in more ad-hoc
measures can result in unreliable, even misleading estimates of the quality of
a treatment policy. We also provide suggestions for mitigating these
effects---for while there is much promise for mining observational health data
to uncover better treatment policies, evaluation must be performed
thoughtfully.

negative:
From 2017 to 2018 the number of scientific publications found via PubMed
search using the keyword "Machine Learning" increased by 46% (4,317 to 6,307).
The results of studies involving machine learning, artificial intelligence
(AI), and big data have captured the attention of healthcare practitioners,
healthcare managers, and the public at a time when Western medicine grapples
with unmitigated cost increases and public demands for accountability. The
complexity involved in healthcare applications of machine learning and the size
of the associated data sets has afforded many researchers an uncontested
opportunity to satisfy these demands with relatively little oversight. In a
recent Nature Medicine article, "The Artificial Intelligence Clinician learns
optimal treatment strategies for sepsis in intensive care," Komorowski and his
coauthors propose methods to train an artificial intelligence clinician to
treat sepsis patients with vasopressors and IV fluids. In this post, we will
closely examine the claims laid out in this paper. In particular, we will study
the individual treatment profiles suggested by their AI Clinician to gain
insight into how their AI Clinician intends to treat patients on an individual
level.

negative:
Many real-world systems problems require reasoning about the long term
consequences of actions taken to configure and manage the system. These
problems with delayed and often sequentially aggregated reward, are often
inherently reinforcement learning problems and present the opportunity to
leverage the recent substantial advances in deep reinforcement learning.
However, in some cases, it is not clear why deep reinforcement learning is a
good fit for the problem. Sometimes, it does not perform better than the
state-of-the-art solutions. And in other cases, random search or greedy
algorithms could outperform deep reinforcement learning. In this paper, we
review, discuss, and evaluate the recent trends of using deep reinforcement
learning in system optimization. We propose a set of essential metrics to guide
future works in evaluating the efficacy of using deep reinforcement learning in
system optimization. Our evaluation includes challenges, the types of problems,
their formulation in the deep reinforcement learning setting, embedding, the
model used, efficiency, and robustness. We conclude with a discussion on open
challenges and potential directions for pushing further the integration of
reinforcement learning in system optimization.

negative:
A long-term goal of reinforcement learning agents is to be able to perform
tasks in complex real-world scenarios. The use of external information is one
way of scaling agents to more complex problems. However, there is a general
lack of collaboration or interoperability between different approaches using
external information. In this work, while reviewing externally-influenced
methods, we propose a conceptual framework and taxonomy for assisted
reinforcement learning, aimed at fostering collaboration by classifying and
comparing various methods that use external information in the learning
process. The proposed taxonomy details the relationship between the external
information source and the learner agent, highlighting the process of
information decomposition, structure, retention, and how it can be used to
influence agent learning. As well as reviewing state-of-the-art methods, we
identify current streams of reinforcement learning that use external
information in order to improve the agent's performance and its decision-making
process. These include heuristic reinforcement learning, interactive
reinforcement learning, learning from demonstration, transfer learning, and
learning from multiple sources, among others. These streams of reinforcement
learning operate with the shared objective of scaffolding the learner agent.
Lastly, we discuss further possibilities for future work in the field of
assisted reinforcement learning systems.

negative:
While reinforcement learning (RL) has proven to be the approach of choice for
tackling many complex problems, it remains challenging to develop and deploy RL
agents in real-life scenarios successfully. This paper presents pH-RL
(personalization in e-Health with RL) a general RL architecture for
personalization to bring RL to health practice. pH-RL allows for various levels
of personalization in health applications and allows for online and batch
learning. Furthermore, we provide a general-purpose implementation framework
that can be integrated with various healthcare applications. We describe a
step-by-step guideline for the successful deployment of RL policies in a mobile
application. We implemented our open-source RL architecture and integrated it
with the MoodBuster mobile application for mental health to provide messages to
increase daily adherence to the online therapeutic modules. We then performed a
comprehensive study with human participants over a sustained period. Our
experimental results show that the developed policies learn to select
appropriate actions consistently using only a few days' worth of data.
Furthermore, we empirically demonstrate the stability of the learned policies
during the study.

negative:
In mobile health (mHealth) smart devices deliver behavioral treatments
repeatedly over time to a user with the goal of helping the user adopt and
maintain healthy behaviors. Reinforcement learning appears ideal for learning
how to optimally make these sequential treatment decisions. However,
significant challenges must be overcome before reinforcement learning can be
effectively deployed in a mobile healthcare setting. In this work we are
concerned with the following challenges: 1) individuals who are in the same
context can exhibit differential response to treatments 2) only a limited
amount of data is available for learning on any one individual, and 3)
non-stationary responses to treatment. To address these challenges we
generalize Thompson-Sampling bandit algorithms to develop IntelligentPooling.
IntelligentPooling learns personalized treatment policies thus addressing
challenge one. To address the second challenge, IntelligentPooling updates each
user's degree of personalization while making use of available data on other
users to speed up learning. Lastly, IntelligentPooling allows responsivity to
vary as a function of a user's time since beginning treatment, thus addressing
challenge three. We show that IntelligentPooling achieves an average of 26%
lower regret than state-of-the-art. We demonstrate the promise of this approach
and its ability to learn from even a small group of users in a live clinical
trial.

negative:
In this article, we aim to provide a literature review of different
formulations and approaches to continual reinforcement learning (RL), also
known as lifelong or non-stationary RL. We begin by discussing our perspective
on why RL is a natural fit for studying continual learning. We then provide a
taxonomy of different continual RL formulations by mathematically
characterizing two key properties of non-stationarity, namely, the scope and
driver non-stationarity. This offers a unified view of various formulations.
Next, we review and present a taxonomy of continual RL approaches. We go on to
discuss evaluation of continual RL agents, providing an overview of benchmarks
used in the literature and important metrics for understanding agent
performance. Finally, we highlight open problems and challenges in bridging the
gap between the current state of continual RL and findings in neuroscience.
While still in its early days, the study of continual RL has the promise to
develop better incremental reinforcement learners that can function in
increasingly realistic applications where non-stationarity plays a vital role.
These include applications such as those in the fields of healthcare,
education, logistics, and robotics.

negative:
Deep reinforcement learning augments the reinforcement learning framework and
utilizes the powerful representation of deep neural networks. Recent works have
demonstrated the remarkable successes of deep reinforcement learning in various
domains including finance, medicine, healthcare, video games, robotics, and
computer vision. In this work, we provide a detailed review of recent and
state-of-the-art research advances of deep reinforcement learning in computer
vision. We start with comprehending the theories of deep learning,
reinforcement learning, and deep reinforcement learning. We then propose a
categorization of deep reinforcement learning methodologies and discuss their
advantages and limitations. In particular, we divide deep reinforcement
learning into seven main categories according to their applications in computer
vision, i.e. (i)landmark localization (ii) object detection; (iii) object
tracking; (iv) registration on both 2D image and 3D image volumetric data (v)
image segmentation; (vi) videos analysis; and (vii) other applications. Each of
these categories is further analyzed with reinforcement learning techniques,
network design, and performance. Moreover, we provide a comprehensive analysis
of the existing publicly available datasets and examine source code
availability. Finally, we present some open issues and discuss future research
directions on deep reinforcement learning in computer vision

negative:
The increasing availability of large collections of electronic health record
(EHR) data and unprecedented technical advances in deep learning (DL) have
sparked a surge of research interest in developing DL based clinical decision
support systems for diagnosis, prognosis, and treatment. Despite the
recognition of the value of deep learning in healthcare, impediments to further
adoption in real healthcare settings remain due to the black-box nature of DL.
Therefore, there is an emerging need for interpretable DL, which allows end
users to evaluate the model decision making to know whether to accept or reject
predictions and recommendations before an action is taken. In this review, we
focus on the interpretability of the DL models in healthcare. We start by
introducing the methods for interpretability in depth and comprehensively as a
methodological reference for future researchers or clinical practitioners in
this field. Besides the methods' details, we also include a discussion of
advantages and disadvantages of these methods and which scenarios each of them
is suitable for, so that interested readers can know how to compare and choose
among them for use. Moreover, we discuss how these methods, originally
developed for solving general-domain problems, have been adapted and applied to
healthcare problems and how they can help physicians better understand these
data-driven technologies. Overall, we hope this survey can help researchers and
practitioners in both artificial intelligence (AI) and clinical fields
understand what methods we have for enhancing the interpretability of their DL
models and choose the optimal one accordingly.

negative:
Reinforcement learning (RL) has been widely applied to game-playing and
surpassed the best human-level performance in many domains, yet there are few
use-cases in industrial or commercial settings. We introduce OR-Gym, an
open-source library for developing reinforcement learning algorithms to address
operations research problems. In this paper, we apply reinforcement learning to
the knapsack, multi-dimensional bin packing, multi-echelon supply chain, and
multi-period asset allocation model problems, as well as benchmark the RL
solutions against MILP and heuristic models. These problems are used in
logistics, finance, engineering, and are common in many business operation
settings. We develop environments based on prototypical models in the
literature and implement various optimization and heuristic models in order to
benchmark the RL results. By re-framing a series of classic optimization
problems as RL tasks, we seek to provide a new tool for the operations research
community, while also opening those in the RL community to many of the problems
and challenges in the OR field.

negative:
Bayesian methods for machine learning have been widely investigated, yielding
principled methods for incorporating prior information into inference
algorithms. In this survey, we provide an in-depth review of the role of
Bayesian methods for the reinforcement learning (RL) paradigm. The major
incentives for incorporating Bayesian reasoning in RL are: 1) it provides an
elegant approach to action-selection (exploration/exploitation) as a function
of the uncertainty in learning; and 2) it provides a machinery to incorporate
prior knowledge into the algorithms. We first discuss models and methods for
Bayesian inference in the simple single-step Bandit model. We then review the
extensive recent literature on Bayesian methods for model-based RL, where prior
information can be expressed on the parameters of the Markov model. We also
present Bayesian methods for model-free RL, where priors are expressed over the
value function or policy class. The objective of the paper is to provide a
comprehensive survey on Bayesian RL algorithms and their theoretical and
empirical properties.

negative:
Rising costs in recent years of developing new drugs and treatments have led
to extensive research in optimization techniques in biomolecular design.
Currently, the most widely used approach in biomolecular design is directed
evolution, which is a greedy hill-climbing algorithm that simulates biological
evolution. In this paper, we propose a new benchmark of applying reinforcement
learning to RNA sequence design, in which the objective function is defined to
be the free energy in the sequence's secondary structure. In addition to
experimenting with the vanilla implementations of each reinforcement learning
algorithm from standard libraries, we analyze variants of each algorithm in
which we modify the algorithm's reward function and tune the model's
hyperparameters. We show results of the ablation analysis that we do for these
algorithms, as well as graphs indicating the algorithm's performance across
batches and its ability to search the possible space of RNA sequences. We find
that our DQN algorithm performs by far the best in this setting, contrasting
with, in which PPO performs the best among all tested algorithms. Our results
should be of interest to those in the biomolecular design community and should
serve as a baseline for future experiments involving machine learning in
molecule design.

negative:
Recent efforts in bioinformatics have achieved tremendous progress in the
machine reading of biomedical literature, and the assembly of the extracted
biochemical interactions into large-scale models such as protein signaling
pathways. However, batch machine reading of literature at today's scale (PubMed
alone indexes over 1 million papers per year) is unfeasible due to both cost
and processing overhead. In this work, we introduce a focused reading approach
to guide the machine reading of biomedical literature towards what literature
should be read to answer a biomedical query as efficiently as possible. We
introduce a family of algorithms for focused reading, including an intuitive,
strong baseline, and a second approach which uses a reinforcement learning (RL)
framework that learns when to explore (widen the search) or exploit (narrow
it). We demonstrate that the RL approach is capable of answering more queries
than the baseline, while being more efficient, i.e., reading fewer documents.

negative:
We present Catalyst.RL, an open-source PyTorch framework for reproducible and
sample efficient reinforcement learning (RL) research. Main features of
Catalyst.RL include large-scale asynchronous distributed training, efficient
implementations of various RL algorithms and auxiliary tricks, such as n-step
returns, value distributions, hyperbolic reinforcement learning, etc. To
demonstrate the effectiveness of Catalyst.RL, we applied it to a physics-based
reinforcement learning challenge "NeurIPS 2019: Learn to Move -- Walk Around"
with the objective to build a locomotion controller for a human musculoskeletal
model. The environment is computationally expensive, has a high-dimensional
continuous action space and is stochastic. Our team took the 2nd place,
capitalizing on the ability of Catalyst.RL to train high-quality and
sample-efficient RL agents in only a few hours of training time. The
implementation along with experiments is open-sourced so results can be
reproduced and novel ideas tried out.

negative:
Designing optimal controllers continues to be challenging as systems are
becoming complex and are inherently nonlinear. The principal advantage of
reinforcement learning (RL) is its ability to learn from the interaction with
the environment and provide optimal control strategy. In this paper, RL is
explored in the context of control of the benchmark cartpole dynamical system
with no prior knowledge of the dynamics. RL algorithms such as
temporal-difference, policy gradient actor-critic, and value function
approximation are compared in this context with the standard LQR solution.
Further, we propose a novel approach to integrate RL and swing-up controllers.

negative:
Sequential decision making, commonly formalized as Markov Decision Process
(MDP) optimization, is a important challenge in artificial intelligence. Two
key approaches to this problem are reinforcement learning (RL) and planning.
This paper presents a survey of the integration of both fields, better known as
model-based reinforcement learning. Model-based RL has two main steps. First,
we systematically cover approaches to dynamics model learning, including
challenges like dealing with stochasticity, uncertainty, partial observability,
and temporal abstraction. Second, we present a systematic categorization of
planning-learning integration, including aspects like: where to start planning,
what budgets to allocate to planning and real data collection, how to plan, and
how to integrate planning in the learning and acting loop. After these two
sections, we also discuss implicit model-based RL as an end-to-end alternative
for model learning and planning, and we cover the potential benefits of
model-based RL. Along the way, the survey also draws connections to several
related RL fields, like hierarchical RL and transfer learning. Altogether, the
survey presents a broad conceptual overview of the combination of planning and
learning for MDP optimization.

negative:
Offline reinforcement learning (RL) defines the task of learning from a fixed
batch of data. Due to errors in value estimation from out-of-distribution
actions, most offline RL algorithms take the approach of constraining or
regularizing the policy with the actions contained in the dataset. Built on
pre-existing RL algorithms, modifications to make an RL algorithm work offline
comes at the cost of additional complexity. Offline RL algorithms introduce new
hyperparameters and often leverage secondary components such as generative
models, while adjusting the underlying RL algorithm. In this paper we aim to
make a deep RL algorithm work while making minimal changes. We find that we can
match the performance of state-of-the-art offline RL algorithms by simply
adding a behavior cloning term to the policy update of an online RL algorithm
and normalizing the data. The resulting algorithm is a simple to implement and
tune baseline, while more than halving the overall run time by removing the
additional computational overhead of previous methods.

negative:
With the development of deep representation learning, the domain of
reinforcement learning (RL) has become a powerful learning framework now
capable of learning complex policies in high dimensional environments. This
review summarises deep reinforcement learning (DRL) algorithms and provides a
taxonomy of automated driving tasks where (D)RL methods have been employed,
while addressing key computational challenges in real world deployment of
autonomous driving agents. It also delineates adjacent domains such as behavior
cloning, imitation learning, inverse reinforcement learning that are related
but are not classical RL algorithms. The role of simulators in training agents,
methods to validate, test and robustify existing solutions in RL are discussed.

negative:
We discuss deep reinforcement learning in an overview style. We draw a big
picture, filled with details. We discuss six core elements, six important
mechanisms, and twelve applications, focusing on contemporary work, and in
historical contexts. We start with background of artificial intelligence,
machine learning, deep learning, and reinforcement learning (RL), with
resources. Next we discuss RL core elements, including value function, policy,
reward, model, exploration vs. exploitation, and representation. Then we
discuss important mechanisms for RL, including attention and memory,
unsupervised learning, hierarchical RL, multi-agent RL, relational RL, and
learning to learn. After that, we discuss RL applications, including games,
robotics, natural language processing (NLP), computer vision, finance, business
management, healthcare, education, energy, transportation, computer systems,
and, science, engineering, and art. Finally we summarize briefly, discuss
challenges and opportunities, and close with an epilogue.

negative:
With AlphaGo defeats top human players, reinforcement learning(RL) algorithms
have gradually become the code-base of building stronger artificial
intelligence(AI). The RL algorithm design firstly needs to adapt to the
specific environment, so the designed environment guides the rapid and profound
development of RL algorithms. However, the existing environments, which can be
divided into real world games and customized toy environments, have obvious
shortcomings. For real world games, it is designed for human entertainment, and
too much difficult for most of RL researchers. For customized toy environments,
there is no widely accepted unified evaluation standard for all RL algorithms.
Therefore, we introduce the first virtual user-friendly environment framework
for RL. In this framework, the environment can be easily configured to realize
all kinds of RL tasks in the mainstream research. Then all the mainstream
state-of-the-art(SOTA) RL algorithms can be conveniently evaluated and
compared. Therefore, our contributions mainly includes the following aspects:
1.single configured environment for all classification of SOTA RL algorithms;
2.combined environment of more than one classification RL algorithms; 3.the
evaluation standard for all kinds of RL algorithms. With all these efforts, a
possibility for breeding an AI with capability of general competency in a
variety of tasks is provided, and maybe it will open up a new chapter for AI.

negative:
Reinforcement learning (RL) is a popular paradigm for addressing sequential
decision tasks in which the agent has only limited environmental feedback.
Despite many advances over the past three decades, learning in many domains
still requires a large amount of interaction with the environment, which can be
prohibitively expensive in realistic scenarios. To address this problem,
transfer learning has been applied to reinforcement learning such that
experience gained in one task can be leveraged when starting to learn the next,
harder task. More recently, several lines of research have explored how tasks,
or data samples themselves, can be sequenced into a curriculum for the purpose
of learning a problem that may otherwise be too difficult to learn from
scratch. In this article, we present a framework for curriculum learning (CL)
in reinforcement learning, and use it to survey and classify existing CL
methods in terms of their assumptions, capabilities, and goals. Finally, we use
our framework to find open problems and suggest directions for future RL
curriculum learning research.

negative:
Objectives-Geriatric clinical care is a multidisciplinary assessment designed
to evaluate older patients (age 65 years and above) functional ability,
physical health, and cognitive wellbeing. The majority of these patients suffer
from multiple chronic conditions and require special attention. Recently,
hospitals utilize various artificial intelligence (AI) systems to improve care
for elderly patients. The purpose of this systematic literature review is to
understand the current use of AI systems, particularly machine learning (ML),
in geriatric clinical care for chronic diseases. Materials and Methods-We
restricted our search to eight databases, namely PubMed, WorldCat, MEDLINE,
ProQuest, ScienceDirect, SpringerLink, Wiley, and ERIC, to analyze research
articles published in English between January 2010 and June 2019. We focused on
studies that used ML algorithms in the care of geriatrics patients with chronic
conditions. Results-We identified 35 eligible studies and classified in three
groups-psychological disorder (n=22), eye diseases (n=6), and others (n=7).
This review identified the lack of standardized ML evaluation metrics and the
need for data governance specific to health care applications. Conclusion- More
studies and ML standardization tailored to health care applications are
required to confirm whether ML could aid in improving geriatric clinical care.

negative:
Reinforcement Learning (RL) has achieved state-of-the-art results in domains
such as robotics and games. We build on this previous work by applying RL
algorithms to a selection of canonical online stochastic optimization problems
with a range of practical applications: Bin Packing, Newsvendor, and Vehicle
Routing. While there is a nascent literature that applies RL to these problems,
there are no commonly accepted benchmarks which can be used to compare proposed
approaches rigorously in terms of performance, scale, or generalizability. This
paper aims to fill that gap. For each problem we apply both standard approaches
as well as newer RL algorithms and analyze results. In each case, the
performance of the trained RL policy is competitive with or superior to the
corresponding baselines, while not requiring much in the way of domain
knowledge. This highlights the potential of RL in real-world dynamic resource
allocation problems.

negative:
In this chapter, we will give comprehensive examples of applying RL in
optimizing the physical layer of wireless communications by defining different
class of problems and the possible solutions to handle them. In Section 9.2, we
present all the basic theory needed to address a RL problem, i.e. Markov
decision process (MDP), Partially observable Markov decision process (POMDP),
but also two very important and widely used algorithms for RL, i.e. the
Q-learning and SARSA algorithms. We also introduce the deep reinforcement
learning (DRL) paradigm and the section ends with an introduction to the
multi-armed bandits (MAB) framework. Section 9.3 focuses on some toy examples
to illustrate how the basic concepts of RL are employed in communication
systems. We present applications extracted from literature with simplified
system models using similar notation as in Section 9.2 of this Chapter. In
Section 9.3, we also focus on modeling RL problems, i.e. how action and state
spaces and rewards are chosen. The Chapter is concluded in Section 9.4 with a
prospective thought on RL trends and it ends with a review of a broader state
of the art in Section 9.5.

negative:
The integration of deep learning to reinforcement learning (RL) has enabled
RL to perform efficiently in high-dimensional environments. Deep RL methods
have been applied to solve many complex real-world problems in recent years.
However, development of a deep RL-based system is challenging because of
various issues such as the selection of a suitable deep RL algorithm, its
network configuration, training time, training methods, and so on. This paper
proposes a comprehensive software framework that not only plays a vital role in
designing a connect-the-dots deep RL architecture but also provides a guideline
to develop a realistic RL application in a short time span. We have designed
and developed a deep RL-based software framework that strictly ensures
flexibility, robustness, and scalability. By inheriting the proposed
architecture, software managers can foresee any challenges when designing a
deep RL-based system. As a result, they can expedite the design process and
actively control every stage of software development, which is especially
critical in agile development environments. To enforce generalization, the
proposed architecture does not depend on a specific RL algorithm, a network
configuration, the number of agents, or the type of agents. Using our
framework, software developers can develop and integrate new RL algorithms or
new types of agents, and can flexibly change network configuration or the
number of agents.

negative:
The recent developments in the field of biomedicine have made large volumes
of biomedical literature available to the medical practitioners. Due to the
large size and lack of efficient searching strategies, medical practitioners
struggle to obtain necessary information available in the biomedical
literature. Moreover, the most sophisticated search engines of age are not
intelligent enough to interpret the clinicians' questions. These facts reflect
the urgent need of an information retrieval system that accepts the queries
from medical practitioners' in natural language and returns the answers quickly
and efficiently. In this paper, we present an implementation of a machine
intelligence based CLINIcal Question Answering system (CLINIQA) to answer
medical practitioner's questions. The system was rigorously evaluated on
different text mining algorithms and the best components for the system were
selected. The system makes use of Unified Medical Language System for semantic
analysis of both questions and medical documents. In addition, the system
employs supervised machine learning algorithms for classification of the
documents, identifying the focus of the question and answer selection.
Effective domain-specific heuristics are designed for answer ranking. The
performance evaluation on hundred clinical questions shows the effectiveness of
our approach.

negative:
Deep reinforcement learning is the combination of reinforcement learning (RL)
and deep learning. This field of research has been able to solve a wide range
of complex decision-making tasks that were previously out of reach for a
machine. Thus, deep RL opens up many new applications in domains such as
healthcare, robotics, smart grids, finance, and many more. This manuscript
provides an introduction to deep reinforcement learning models, algorithms and
techniques. Particular focus is on the aspects related to generalization and
how deep RL can be used for practical applications. We assume the reader is
familiar with basic machine learning concepts.

negative:
In recent years deep reinforcement learning (RL) systems have attained
superhuman performance in a number of challenging task domains. However, a
major limitation of such applications is their demand for massive amounts of
training data. A critical present objective is thus to develop deep RL methods
that can adapt rapidly to new tasks. In the present work we introduce a novel
approach to this challenge, which we refer to as deep meta-reinforcement
learning. Previous work has shown that recurrent networks can support
meta-learning in a fully supervised context. We extend this approach to the RL
setting. What emerges is a system that is trained using one RL algorithm, but
whose recurrent dynamics implement a second, quite separate RL procedure. This
second, learned RL algorithm can differ from the original one in arbitrary
ways. Importantly, because it is learned, it is configured to exploit structure
in the training domain. We unpack these points in a series of seven
proof-of-concept experiments, each of which examines a key aspect of deep
meta-RL. We consider prospects for extending and scaling up the approach, and
also point out some potentially important implications for neuroscience.

negative:
While current benchmark reinforcement learning (RL) tasks have been useful to
drive progress in the field, they are in many ways poor substitutes for
learning with real-world data. By testing increasingly complex RL algorithms on
low-complexity simulation environments, we often end up with brittle RL
policies that generalize poorly beyond the very specific domain. To combat
this, we propose three new families of benchmark RL domains that contain some
of the complexity of the natural world, while still supporting fast and
extensive data acquisition. The proposed domains also permit a characterization
of generalization through fair train/test separation, and easy comparison and
replication of results. Through this work, we challenge the RL research
community to develop more robust algorithms that meet high standards of
evaluation.

negative:
Sepsis is the leading cause of mortality in the ICU. It is challenging to
manage because individual patients respond differently to treatment. Thus,
tailoring treatment to the individual patient is essential for the best
outcomes. In this paper, we take steps toward this goal by applying a
mixture-of-experts framework to personalize sepsis treatment. The mixture model
selectively alternates between neighbor-based (kernel) and deep reinforcement
learning (DRL) experts depending on patient's current history. On a large
retrospective cohort, this mixture-based approach outperforms physician, kernel
only, and DRL-only experts.

negative:
Building upon prior research that highlighted the need for standardizing
environments for building control research, and inspired by recently introduced
challenges for real life reinforcement learning control, here we propose a
non-exhaustive set of nine real world challenges for reinforcement learning
control in grid-interactive buildings. We argue that research in this area
should be expressed in this framework in addition to providing a standardized
environment for repeatability. Advanced controllers such as model predictive
control and reinforcement learning (RL) control have both advantages and
disadvantages that prevent them from being implemented in real world problems.
Comparisons between the two are rare, and often biased. By focusing on the
challenges, we can investigate the performance of the controllers under a
variety of situations and generate a fair comparison. As a demonstration, we
implement the offline learning challenge in CityLearn and study the impact of
different levels of domain knowledge and complexity of RL algorithms. We show
that the sequence of operations utilized in a rule based controller (RBC) used
for offline training affects the performance of the RL agents when evaluated on
a set of four energy flexibility metrics. Longer offline learning from an
optimized RBC leads to improved performance in the long run. RL agents that
learn from a simplified RBC risk poorer performance as the offline learning
period increases. We also observe no impact on performance from information
sharing amongst agents. We call for a more interdisciplinary effort of the
research community to address the real world challenges, and unlock the
potential of grid-interactive building

negative:
Clinicians face several significant barriers to search and synthesize
accurate, succinct, updated, and trustworthy medical information from several
literature sources during the practice of medicine and patient care. In this
talk, we will be presenting our research behind the development of a Focused
Clinical Search Service, powered by a Healthcare Knowledge Graph, to interpret
the query intent behind clinical search queries and retrieve relevant medical
snippets from a diverse corpus of medical literature.

negative:
As a promising tool to navigate in the vast chemical space, artificial
intelligence (AI) is leveraged for drug design. From the year 2017 to 2021, the
number of applications of several recent AI models (i.e. graph neural network
(GNN), recurrent neural network (RNN), variation autoencoder (VAE), generative
adversarial network (GAN), flow and reinforcement learning (RL)) in drug design
increases significantly. Many relevant literature reviews exist. However, none
of them provides an in-depth summary of many applications of the recent AI
models in drug design. To complement the existing literature, this survey
includes the theoretical development of the previously mentioned AI models and
detailed summaries of 42 recent applications of AI in drug design. Concretely,
13 of them leverage GNN for molecular property prediction and 29 of them use RL
and/or deep generative models for molecule generation and optimization. In most
cases, the focus of the summary is the models, their variants, and
modifications for specific tasks in drug design. Moreover, 60 additional
applications of AI in molecule generation and optimization are briefly
summarized in a table. Finally, this survey provides a holistic discussion of
the abundant applications so that the tasks, potential solutions, and
challenges in AI-based drug design become evident.

negative:
Reinforcement learning is a learning paradigm for solving sequential
decision-making problems. Recent years have witnessed remarkable progress in
reinforcement learning upon the fast development of deep neural networks. Along
with the promising prospects of reinforcement learning in numerous domains such
as robotics and game-playing, transfer learning has arisen to tackle various
challenges faced by reinforcement learning, by transferring knowledge from
external expertise to facilitate the efficiency and effectiveness of the
learning process. In this survey, we systematically investigate the recent
progress of transfer learning approaches in the context of deep reinforcement
learning. Specifically, we provide a framework for categorizing the
state-of-the-art transfer learning approaches, under which we analyze their
goals, methodologies, compatible reinforcement learning backbones, and
practical applications. We also draw connections between transfer learning and
other relevant topics from the reinforcement learning perspective and explore
their potential challenges that await future research progress.

negative:
Much work aims to explain a model's prediction on a static input. We consider
explanations in a temporal setting where a stateful dynamical model produces a
sequence of risk estimates given an input at each time step. When the estimated
risk increases, the goal of the explanation is to attribute the increase to a
few relevant inputs from the past. While our formal setup and techniques are
general, we carry out an in-depth case study in a clinical setting. The goal
here is to alert a clinician when a patient's risk of deterioration rises. The
clinician then has to decide whether to intervene and adjust the treatment.
Given a potentially long sequence of new events since she last saw the patient,
a concise explanation helps her to quickly triage the alert. We develop methods
to lift static attribution techniques to the dynamical setting, where we
identify and address challenges specific to dynamics. We then experimentally
assess the utility of different explanations of clinical alerts through expert
evaluation.

negative:
A consequence of the fragmented and siloed healthcare landscape is that
patient care (and data) is split along multitude of different facilities and
computer systems and enabling interoperability between these systems is hard.
The lack interoperability not only hinders continuity of care and burdens
providers, but also hinders effective application of Machine Learning (ML)
algorithms. Thus, most current ML algorithms, designed to understand patient
care and facilitate clinical decision-support, are trained on limited datasets.
This approach is analogous to the Newtonian paradigm of Reductionism in which a
system is broken down into elementary components and a description of the whole
is formed by understanding those components individually. A key limitation of
the reductionist approach is that it ignores the component-component
interactions and dynamics within the system which are often of prime
significance in understanding the overall behaviour of complex adaptive systems
(CAS). Healthcare is a CAS.
  Though the application of ML on health data have shown incremental
improvements for clinical decision support, ML has a much a broader potential
to restructure care delivery as a whole and maximize care value. However, this
ML potential remains largely untapped: primarily due to functional limitations
of Electronic Health Records (EHR) and the inability to see the healthcare
system as a whole. This viewpoint (i) articulates the healthcare as a complex
system which has a biological and an organizational perspective, (ii) motivates
with examples, the need of a system's approach when addressing healthcare
challenges via ML and, (iii) emphasizes to unleash EHR functionality - while
duly respecting all ethical and legal concerns - to reap full benefits of ML.

negative:
We give an overview of recent exciting achievements of deep reinforcement
learning (RL). We discuss six core elements, six important mechanisms, and
twelve applications. We start with background of machine learning, deep
learning and reinforcement learning. Next we discuss core RL elements,
including value function, in particular, Deep Q-Network (DQN), policy, reward,
model, planning, and exploration. After that, we discuss important mechanisms
for RL, including attention and memory, unsupervised learning, transfer
learning, multi-agent RL, hierarchical RL, and learning to learn. Then we
discuss various applications of RL, including games, in particular, AlphaGo,
robotics, natural language processing, including dialogue systems, machine
translation, and text generation, computer vision, neural architecture design,
business management, finance, healthcare, Industry 4.0, smart grid, intelligent
transportation systems, and computer systems. We mention topics not reviewed
yet, and list a collection of RL resources. After presenting a brief summary,
we close with discussions.
  Please see Deep Reinforcement Learning, arXiv:1810.06339, for a significant
update.

negative:
Catastrophic forgetting remains a severe hindrance to the broad application
of artificial neural networks (ANNs), however, it continues to be a poorly
understood phenomenon. Despite the extensive amount of work on catastrophic
forgetting, we argue that it is still unclear how exactly the phenomenon should
be quantified, and, moreover, to what degree all of the choices we make when
designing learning systems affect the amount of catastrophic forgetting. We use
various testbeds from the reinforcement learning and supervised learning
literature to (1) provide evidence that the choice of which modern
gradient-based optimization algorithm is used to train an ANN has a significant
impact on the amount of catastrophic forgetting and show that-surprisingly-in
many instances classical algorithms such as vanilla SGD experience less
catastrophic forgetting than the more modern algorithms such as Adam. We
empirically compare four different existing metrics for quantifying
catastrophic forgetting and (2) show that the degree to which the learning
systems experience catastrophic forgetting is sufficiently sensitive to the
metric used that a change from one principled metric to another is enough to
change the conclusions of a study dramatically. Our results suggest that a much
more rigorous experimental methodology is required when looking at catastrophic
forgetting. Based on our results, we recommend inter-task forgetting in
supervised learning must be measured with both retention and relearning metrics
concurrently, and intra-task forgetting in reinforcement learning must-at the
very least-be measured with pairwise interference.

negative:
Deep reinforcement learning is poised to revolutionise the field of AI and
represents a step towards building autonomous systems with a higher level
understanding of the visual world. Currently, deep learning is enabling
reinforcement learning to scale to problems that were previously intractable,
such as learning to play video games directly from pixels. Deep reinforcement
learning algorithms are also applied to robotics, allowing control policies for
robots to be learned directly from camera inputs in the real world. In this
survey, we begin with an introduction to the general field of reinforcement
learning, then progress to the main streams of value-based and policy-based
methods. Our survey will cover central algorithms in deep reinforcement
learning, including the deep $Q$-network, trust region policy optimisation, and
asynchronous advantage actor-critic. In parallel, we highlight the unique
advantages of deep neural networks, focusing on visual understanding via
reinforcement learning. To conclude, we describe several current areas of
research within the field.

negative:
Motivated by recent advancements in Deep Reinforcement Learning (RL), we have
developed an RL agent to manage the operation of storage devices in a household
and is designed to maximize demand-side cost savings. The proposed technique is
data-driven, and the RL agent learns from scratch how to efficiently use the
energy storage device given variable tariff structures. In most of the studies,
the RL agent is considered as a black box, and how the agent has learned is
often ignored. We explain the learning progression of the RL agent, and the
strategies it follows based on the capacity of the storage device.

negative:
Reinforcement learning (RL) combines a control problem with statistical
estimation: The system dynamics are not known to the agent, but can be learned
through experience. A recent line of research casts `RL as inference' and
suggests a particular framework to generalize the RL problem as probabilistic
inference. Our paper surfaces a key shortcoming in that approach, and clarifies
the sense in which RL can be coherently cast as an inference problem. In
particular, an RL agent must consider the effects of its actions upon future
rewards and observations: The exploration-exploitation tradeoff. In all but the
most simple settings, the resulting inference is computationally intractable so
that practical RL algorithms must resort to approximation. We demonstrate that
the popular `RL as inference' approximation can perform poorly in even very
basic problems. However, we show that with a small modification the framework
does yield algorithms that can provably perform well, and we show that the
resulting algorithm is equivalent to the recently proposed K-learning, which we
further connect with Thompson sampling.

negative:
We start with a brief introduction to reinforcement learning (RL), about its
successful stories, basics, an example, issues, the ICML 2019 Workshop on RL
for Real Life, how to use it, study material and an outlook. Then we discuss a
selection of RL applications, including recommender systems, computer systems,
energy, finance, healthcare, robotics, and transportation.

negative:
Rapid online adaptation to changing tasks is an important problem in machine
learning and, recently, a focus of meta-reinforcement learning. However,
reinforcement learning (RL) algorithms struggle in POMDP environments because
the state of the system, essential in a RL framework, is not always visible.
Additionally, hand-designed meta-RL architectures may not include suitable
computational structures for specific learning problems. The evolution of
online learning mechanisms, on the contrary, has the ability to incorporate
learning strategies into an agent that can (i) evolve memory when required and
(ii) optimize adaptation speed to specific online learning problems. In this
paper, we exploit the highly adaptive nature of neuromodulated neural networks
to evolve a controller that uses the latent space of an autoencoder in a POMDP.
The analysis of the evolved networks reveals the ability of the proposed
algorithm to acquire inborn knowledge in a variety of aspects such as the
detection of cues that reveal implicit rewards, and the ability to evolve
location neurons that help with navigation. The integration of inborn knowledge
and online plasticity enabled fast adaptation and better performance in
comparison to some non-evolutionary meta-reinforcement learning algorithms. The
algorithm proved also to succeed in the 3D gaming environment Malmo Minecraft.

negative:
With the continuous development of machine learning technology, major
e-commerce platforms have launched recommendation systems based on it to serve
a large number of customers with different needs more efficiently. Compared
with traditional supervised learning, reinforcement learning can better capture
the user's state transition in the decision-making process, and consider a
series of user actions, not just the static characteristics of the user at a
certain moment. In theory, it will have a long-term perspective, producing a
more effective recommendation. The special requirements of reinforcement
learning for data make it need to rely on an offline virtual system for
training. Our project mainly establishes a virtual user environment for offline
training. At the same time, we tried to improve a reinforcement learning
algorithm based on bi-clustering to expand the action space and recommended
path space of the recommendation agent.

negative:
This paper surveys the field of deep multiagent reinforcement learning. The
combination of deep neural networks with reinforcement learning has gained
increased traction in recent years and is slowly shifting the focus from
single-agent to multiagent environments. Dealing with multiple agents is
inherently more complex as (a) the future rewards depend on multiple players'
joint actions and (b) the computational complexity increases. We present the
most common multiagent problem representations and their main challenges, and
identify five research areas that address one or more of these challenges:
centralised training and decentralised execution, opponent modelling,
communication, efficient coordination, and reward shaping. We find that many
computational studies rely on unrealistic assumptions or are not generalisable
to other settings; they struggle to overcome the curse of dimensionality or
nonstationarity. Approaches from psychology and sociology capture promising
relevant behaviours, such as communication and coordination, to help agents
achieve better performance in multiagent settings. We suggest that, for
multiagent reinforcement learning to be successful, future research should
address these challenges with an interdisciplinary approach to open up new
possibilities in multiagent reinforcement learning.

negative:
In constrained reinforcement learning (RL), a learning agent seeks to not
only optimize the overall reward but also satisfy the additional safety,
diversity, or budget constraints. Consequently, existing constrained RL
solutions require several new algorithmic ingredients that are notably
different from standard RL. On the other hand, reward-free RL is independently
developed in the unconstrained literature, which learns the transition dynamics
without using the reward information, and thus naturally capable of addressing
RL with multiple objectives under the common dynamics. This paper bridges
reward-free RL and constrained RL. Particularly, we propose a simple
meta-algorithm such that given any reward-free RL oracle, the approachability
and constrained RL problems can be directly solved with negligible overheads in
sample complexity. Utilizing the existing reward-free RL solvers, our framework
provides sharp sample complexity results for constrained RL in the tabular MDP
setting, matching the best existing results up to a factor of horizon
dependence; our framework directly extends to a setting of tabular two-player
Markov games, and gives a new result for constrained RL with linear function
approximation.

negative:
In recent years some researchers have explored the use of reinforcement
learning (RL) algorithms as key components in the solution of various natural
language processing tasks. For instance, some of these algorithms leveraging
deep neural learning have found their way into conversational systems. This
paper reviews the state of the art of RL methods for their possible use for
different problems of natural language processing, focusing primarily on
conversational systems, mainly due to their growing relevance. We provide
detailed descriptions of the problems as well as discussions of why RL is
well-suited to solve them. Also, we analyze the advantages and limitations of
these methods. Finally, we elaborate on promising research directions in
natural language processing that might benefit from reinforcement learning.

negative:
With increasing and extensive use of electronic health records, clinicians
are often under time pressure when they need to retrieve important information
efficiently among large amounts of patients' health records in clinics. While a
search function can be a useful alternative to browsing through a patient's
record, it is cumbersome for clinicians to search repeatedly for the same or
similar information on similar patients. Under such circumstances, there is a
critical need to build effective recommender systems that can generate accurate
search term recommendations for clinicians. In this manuscript, we developed a
hybrid collaborative filtering model using patients' encounter and search term
information to recommend the next search terms for clinicians to retrieve
important information fast in clinics. For each patient, the model will
recommend terms that either have high co-occurrence frequencies with his/her
most recent ICD codes or are highly relevant to the most recent search terms on
this patient. We have conducted comprehensive experiments to evaluate the
proposed model, and the experimental results demonstrate that our model can
outperform all the state-of-the-art baseline methods for top-N search term
recommendation on different datasets.

negative:
Deep reinforcement learning is revolutionizing the artificial intelligence
field. Currently, it serves as a good starting point for constructing
intelligent autonomous systems which offer a better knowledge of the visual
world. It is possible to scale deep reinforcement learning with the use of deep
learning and do amazing tasks such as use of pixels in playing video games. In
this paper, key concepts of deep reinforcement learning including reward
function, differences between reinforcement learning and supervised learning
and models for implementation of reinforcement are discussed. Key challenges
related to the implementation of reinforcement learning in conversational AI
domain are identified as well as discussed in detail. Various conversational
models which are based on deep reinforcement learning (as well as deep
learning) are also discussed. In summary, this paper discusses key aspects of
deep reinforcement learning which are crucial for designing an efficient
conversational AI.

negative:
This article is an introduction to machine learning for financial
forecasting, planning and analysis (FP\&A). Machine learning appears well
suited to support FP\&A with the highly automated extraction of information
from large amounts of data. However, because most traditional machine learning
techniques focus on forecasting (prediction), we discuss the particular care
that must be taken to avoid the pitfalls of using them for planning and
resource allocation (causal inference). While the naive application of machine
learning usually fails in this context, the recently developed double machine
learning framework can address causal questions of interest. We review the
current literature on machine learning in FP\&A and illustrate in a simulation
study how machine learning can be used for both forecasting and planning. We
also investigate how forecasting and planning improve as the number of data
points increases.

negative:
Automatic machine learning (AutoML) is an area of research aimed at
automating machine learning (ML) activities that currently require human
experts. One of the most challenging tasks in this field is the automatic
generation of end-to-end ML pipelines: combining multiple types of ML
algorithms into a single architecture used for end-to-end analysis of
previously-unseen data. This task has two challenging aspects: the first is the
need to explore a large search space of algorithms and pipeline architectures.
The second challenge is the computational cost of training and evaluating
multiple pipelines. In this study we present DeepLine, a reinforcement learning
based approach for automatic pipeline generation. Our proposed approach
utilizes an efficient representation of the search space and leverages past
knowledge gained from previously-analyzed datasets to make the problem more
tractable. Additionally, we propose a novel hierarchical-actions algorithm that
serves as a plugin, mediating the environment-agent interaction in deep
reinforcement learning problems. The plugin significantly speeds up the
training process of our model. Evaluation on 56 datasets shows that DeepLine
outperforms state-of-the-art approaches both in accuracy and in computational
cost.

negative:
To achieve general artificial intelligence, reinforcement learning (RL)
agents should learn not only to optimize returns for one specific task but also
to constantly build more complex skills and scaffold their knowledge about the
world, without forgetting what has already been learned. In this paper, we
discuss the desired characteristics of environments that can support the
training and evaluation of lifelong reinforcement learning agents, review
existing environments from this perspective, and propose recommendations for
devising suitable environments in the future.

negative:
Reinforcement learning (RL) algorithms find applications in inventory
control, recommender systems, vehicular traffic management, cloud computing and
robotics. The real-world complications of many tasks arising in these domains
makes them difficult to solve with the basic assumptions underlying classical
RL algorithms. RL agents in these applications often need to react and adapt to
changing operating conditions. A significant part of research on single-agent
RL techniques focuses on developing algorithms when the underlying assumption
of stationary environment model is relaxed. This paper provides a survey of RL
methods developed for handling dynamically varying environment models. The goal
of methods not limited by the stationarity assumption is to help autonomous
agents adapt to varying operating conditions. This is possible either by
minimizing the rewards lost during learning by RL agent or by finding a
suitable policy for the RL agent which leads to efficient operation of the
underlying system. A representative collection of these algorithms is discussed
in detail in this work along with their categorization and their relative
merits and demerits. Additionally we also review works which are tailored to
application domains. Finally, we discuss future enhancements for this field.

negative:
An expert classification system having statistical information about the
prior probabilities of the different classes should be able to use this
knowledge to reduce the amount of additional information that it must collect,
e.g., through questions, in order to make a correct classification. This paper
examines how best to use such prior information and additional
information-collection opportunities to reduce uncertainty about the class to
which a case belongs, thus minimizing the average cost or effort required to
correctly classify new cases.

negative:
Recent years have witnessed significant advances in reinforcement learning
(RL), which has registered great success in solving various sequential
decision-making problems in machine learning. Most of the successful RL
applications, e.g., the games of Go and Poker, robotics, and autonomous
driving, involve the participation of more than one single agent, which
naturally fall into the realm of multi-agent RL (MARL), a domain with a
relatively long history, and has recently re-emerged due to advances in
single-agent RL techniques. Though empirically successful, theoretical
foundations for MARL are relatively lacking in the literature. In this chapter,
we provide a selective overview of MARL, with focus on algorithms backed by
theoretical analysis. More specifically, we review the theoretical results of
MARL algorithms mainly within two representative frameworks, Markov/stochastic
games and extensive-form games, in accordance with the types of tasks they
address, i.e., fully cooperative, fully competitive, and a mix of the two. We
also introduce several significant but challenging applications of these
algorithms. Orthogonal to the existing reviews on MARL, we highlight several
new angles and taxonomies of MARL theory, including learning in extensive-form
games, decentralized MARL with networked agents, MARL in the mean-field regime,
(non-)convergence of policy-based methods for learning in games, etc. Some of
the new angles extrapolate from our own research endeavors and interests. Our
overall goal with this chapter is, beyond providing an assessment of the
current state of the field on the mark, to identify fruitful future research
directions on theoretical studies of MARL. We expect this chapter to serve as
continuing stimulus for researchers interested in working on this exciting
while challenging topic.

negative:
Causal inference methods are widely applied in the fields of medicine,
policy, and economics. Central to these applications is the estimation of
treatment effects to make decisions. Current methods make binary yes-or-no
decisions based on the treatment effect of a single outcome dimension. These
methods are unable to capture continuous space treatment policies with a
measure of intensity. They also lack the capacity to consider the complexity of
treatment such as matching candidate treatments with the subject. We propose to
formulate the effectiveness of treatment as a parametrizable model, expanding
to a multitude of treatment intensities and complexities through the continuous
policy treatment function, and the likelihood of matching. Our proposal to
decompose treatment effect functions into effectiveness factors presents a
framework to model a rich space of actions using causal inference. We utilize
deep learning to optimize the desired holistic metric space instead of
predicting single-dimensional treatment counterfactual. This approach employs a
population-wide effectiveness measure and significantly improves the overall
effectiveness of the model. The performance of our algorithms is. demonstrated
with experiments. When using generic continuous space treatments and matching
architecture, we observe a 41% improvement upon prior art with
cost-effectiveness and 68% improvement upon a similar method in the average
treatment effect. The algorithms capture subtle variations in treatment space,
structures the efficient optimizations techniques, and opens up the arena for
many applications.

negative:
In the artificial intelligence field, learning often corresponds to changing
the parameters of a parameterized function. A learning rule is an algorithm or
mathematical expression that specifies precisely how the parameters should be
changed. When creating an artificial intelligence system, we must make two
decisions: what representation should be used (i.e., what parameterized
function should be used) and what learning rule should be used to search
through the resulting set of representable functions. Using most learning
rules, these two decisions are coupled in a subtle (and often unintentional)
way. That is, using the same learning rule with two different representations
that can represent the same sets of functions can result in two different
outcomes. After arguing that this coupling is undesirable, particularly when
using artificial neural networks, we present a method for partially decoupling
these two decisions for a broad class of learning rules that span unsupervised
learning, reinforcement learning, and supervised learning.

negative:
Hierarchical reinforcement learning has demonstrated significant success at
solving difficult reinforcement learning (RL) tasks. Previous works have
motivated the use of hierarchy by appealing to a number of intuitive benefits,
including learning over temporally extended transitions, exploring over
temporally extended periods, and training and exploring in a more semantically
meaningful action space, among others. However, in fully observed, Markovian
settings, it is not immediately clear why hierarchical RL should provide
benefits over standard "shallow" RL architectures. In this work, we isolate and
evaluate the claimed benefits of hierarchical RL on a suite of tasks
encompassing locomotion, navigation, and manipulation. Surprisingly, we find
that most of the observed benefits of hierarchy can be attributed to improved
exploration, as opposed to easier policy learning or imposed hierarchical
structures. Given this insight, we present exploration techniques inspired by
hierarchy that achieve performance competitive with hierarchical RL while at
the same time being much simpler to use and implement.

negative:
Prior work on training generative Visual Dialog models with reinforcement
learning(Das et al.) has explored a Qbot-Abot image-guessing game and shown
that this 'self-talk' approach can lead to improved performance at the
downstream dialog-conditioned image-guessing task. However, this improvement
saturates and starts degrading after a few rounds of interaction, and does not
lead to a better Visual Dialog model. We find that this is due in part to
repeated interactions between Qbot and Abot during self-talk, which are not
informative with respect to the image. To improve this, we devise a simple
auxiliary objective that incentivizes Qbot to ask diverse questions, thus
reducing repetitions and in turn enabling Abot to explore a larger state space
during RL ie. be exposed to more visual concepts to talk about, and varied
questions to answer. We evaluate our approach via a host of automatic metrics
and human studies, and demonstrate that it leads to better dialog, ie. dialog
that is more diverse (ie. less repetitive), consistent (ie. has fewer
conflicting exchanges), fluent (ie. more human-like),and detailed, while still
being comparably image-relevant as prior work and ablations.

negative:
Reinforcement learning is one of the core components in designing an
artificial intelligent system emphasizing real-time response. Reinforcement
learning influences the system to take actions within an arbitrary environment
either having previous knowledge about the environment model or not. In this
paper, we present a comprehensive study on Reinforcement Learning focusing on
various dimensions including challenges, the recent development of different
state-of-the-art techniques, and future directions. The fundamental objective
of this paper is to provide a framework for the presentation of available
methods of reinforcement learning that is informative enough and simple to
follow for the new researchers and academics in this domain considering the
latest concerns. First, we illustrated the core techniques of reinforcement
learning in an easily understandable and comparable way. Finally, we analyzed
and depicted the recent developments in reinforcement learning approaches. My
analysis pointed out that most of the models focused on tuning policy values
rather than tuning other things in a particular state of reasoning.

negative:
The increasing incorporation of Artificial Intelligence in the form of
automated systems into decision-making procedures highlights not only the
importance of decision theory for automated systems but also the need for these
decision procedures to be explainable to the people involved in them.
Traditional realist accounts of explanation, wherein explanation is a relation
that holds (or does not hold) eternally between an explanans and an
explanandum, are not adequate to account for the notion of explanation required
for artificial decision procedures. We offer an alternative account of
explanation as used in the context of automated decision-making that makes
explanation an epistemic phenomenon, and one that is dependent on context. This
account of explanation better accounts for the way that we talk about, and use,
explanations and derived concepts, such as `explanatory power', and also allows
us to differentiate between reasons or causes on the one hand, which do not
need to have an epistemic aspect, and explanations on the other, which do have
such an aspect. Against this theoretical backdrop we then review existing
approaches to explanation in Artificial Intelligence and Machine Learning, and
suggest desiderata which truly explainable decision systems should fulfill.

negative:
Our world is open-ended, non-stationary, and constantly evolving; thus what
we talk about and how we talk about it change over time. This inherent dynamic
nature of language contrasts with the current static language modelling
paradigm, which trains and evaluates models on utterances from overlapping time
periods. Despite impressive recent progress, we demonstrate that Transformer-XL
language models perform worse in the realistic setup of predicting future
utterances from beyond their training period, and that model performance
becomes increasingly worse with time. We find that, while increasing model size
alone -- a key driver behind recent progress -- does not solve this problem,
having models that continually update their knowledge with new information can
indeed mitigate this performance degradation over time. Hence, given the
compilation of ever-larger language modelling datasets, combined with the
growing list of language-model-based NLP applications that require up-to-date
factual knowledge about the world, we argue that now is the right time to
rethink the static way in which we currently train and evaluate our language
models, and develop adaptive language models that can remain up-to-date with
respect to our ever-changing and non-stationary world. We publicly release our
dynamic, streaming language modelling benchmarks for WMT and arXiv to
facilitate language model evaluation that takes temporal dynamics into account.

negative:
Brain science and artificial intelligence have made great progress toward the
understanding and engineering of the human mind. The progress has accelerated
significantly since the turn of the century thanks to new methods for probing
the brain (both structure and function), and rapid development in deep learning
research. However, despite these new developments, there are still many open
questions, such as how to understand the brain at the system level, and various
robustness issues and limitations of deep learning. In this informal essay, I
will talk about some of the concepts that are central to brain science and
artificial intelligence, such as information and memory, and discuss how a
different view on these concepts can help us move forward, beyond current
limits of our understanding in these fields.

negative:
This paper provides novel formal methods and empirical demonstrations of a
particular safety concern in reinforcement learning (RL)-based recommendation
algorithms. We call this safety concern `user tampering' -- a phenomenon
whereby an RL-based recommender system might manipulate a media user's opinions
via its recommendations as part of a policy to increase long-term user
engagement. We then apply techniques from causal modelling to analyse the
leading approaches in the literature for implementing scalable RL-based
recommenders, and we observe that the current approaches permit user tampering.
Additionally, we review the existing mitigation strategies for reward tampering
problems and show that they do not transfer well to the user tampering
phenomenon found in the recommendation context. Furthermore, we provide a
simulation study of a media RL-based recommendation problem constrained to the
recommendation of political content. We show that a Q-learning algorithm
consistently learns to exploit its opportunities to polarise simulated users
with its early recommendations in order to have more consistent success with
later recommendations catering to that polarisation. This latter contribution
calls for urgency in designing safer RL-based recommenders; the former suggests
that creating such safe recommenders will require a fundamental shift in design
away from the approaches we have seen in the recent literature.

negative:
The potential presented by Artificial Intelligence (AI) for healthcare has
long been recognised by the technical community. More recently, this potential
has been recognised by policymakers, resulting in considerable public and
private investment in the development of AI for healthcare across the globe.
Despite this, excepting limited success stories, real-world implementation of
AI systems into front-line healthcare has been limited. There are numerous
reasons for this, but a main contributory factor is the lack of internationally
accepted, or formalised, regulatory standards to assess AI safety and impact
and effectiveness. This is a well-recognised problem with numerous ongoing
research and policy projects to overcome it. Our intention here is to
contribute to this problem-solving effort by seeking to set out a minimally
viable framework for evaluating the safety, acceptability and efficacy of AI
systems for healthcare. We do this by conducting a systematic search across
Scopus, PubMed and Google Scholar to identify all the relevant literature
published between January 1970 and November 2020 related to the evaluation of:
output performance; efficacy; and real-world use of AI systems, and
synthesising the key themes according to the stages of evaluation: pre-clinical
(theoretical phase); exploratory phase; definitive phase; and post-market
surveillance phase (monitoring). The result is a framework to guide AI system
developers, policymakers, and regulators through a sufficient evaluation of an
AI system designed for use in healthcare.

negative:
No comprehensive review of Bayesian networks (BNs) in healthcare has been
published in the past, making it difficult to organize the research
contributions in the present and identify challenges and neglected areas that
need to be addressed in the future. This unique and novel scoping review of BNs
in healthcare provides an analytical framework for comprehensively
characterizing the domain and its current state. The review shows that: (1) BNs
in healthcare are not used to their full potential; (2) a generic BN
development process is lacking; (3) limitations exists in the way BNs in
healthcare are presented in the literature, which impacts understanding,
consensus towards systematic methodologies, practice and adoption of BNs; and
(4) a gap exists between having an accurate BN and a useful BN that impacts
clinical practice. This review empowers researchers and clinicians with an
analytical framework and findings that will enable understanding of the need to
address the problems of restricted aims of BNs, ad hoc BN development methods,
and the lack of BN adoption in practice. To map the way forward, the paper
proposes future research directions and makes recommendations regarding BN
development methods and adoption in practice.

negative:
A neuroscience method to understanding the brain is to find and study the
preferred stimuli that highly activate an individual cell or groups of cells.
Recent advances in machine learning enable a family of methods to synthesize
preferred stimuli that cause a neuron in an artificial or biological brain to
fire strongly. Those methods are known as Activation Maximization (AM) or
Feature Visualization via Optimization. In this chapter, we (1) review existing
AM techniques in the literature; (2) discuss a probabilistic interpretation for
AM; and (3) review the applications of AM in debugging and explaining networks.

negative:
Question answering (QA) systems provide a way of querying the information
available in various formats including, but not limited to, unstructured and
structured data in natural languages. It constitutes a considerable part of
conversational artificial intelligence (AI) which has led to the introduction
of a special research topic on Conversational Question Answering (CQA), wherein
a system is required to understand the given context and then engages in
multi-turn QA to satisfy the user's information needs. Whilst the focus of most
of the existing research work is subjected to single-turn QA, the field of
multi-turn QA has recently grasped attention and prominence owing to the
availability of large-scale, multi-turn QA datasets and the development of
pre-trained language models. With a good amount of models and research papers
adding to the literature every year recently, there is a dire need of arranging
and presenting the related work in a unified manner to streamline future
research. This survey, therefore, is an effort to present a comprehensive
review of the state-of-the-art research trends of CQA primarily based on
reviewed papers from 2016-2021. Our findings show that there has been a trend
shift from single-turn to multi-turn QA which empowers the field of
Conversational AI from different perspectives. This survey is intended to
provide an epitome for the research community with the hope of laying a strong
foundation for the field of CQA.

negative:
We consider the problem of providing users of deep Reinforcement Learning
(RL) based systems with a better understanding of when their output can be
trusted. We offer an explainable artificial intelligence (XAI) framework that
provides a three-fold explanation: a graphical depiction of the systems
generalization and performance in the current game state, how well the agent
would play in semantically similar environments, and a narrative explanation of
what the graphical information implies. We created a user-interface for our XAI
framework and evaluated its efficacy via a human-user experiment. The results
demonstrate a statistically significant increase in user trust and acceptance
of the AI system with explanation, versus the AI system without explanation.

negative:
P-values are widely used in both the social and natural sciences to quantify
the statistical significance of observed results. The recent surge of big data
research has made the p-value an even more popular tool to test the
significance of a study. However, substantial literature has been produced
critiquing how p-values are used and understood. In this paper we review this
recent critical literature, much of which is routed in the life sciences, and
consider its implications for social scientific research. We provide a coherent
picture of what the main criticisms are, and draw together and disambiguate
common themes. In particular, we explain how the False Discovery Rate is
calculated, and how this differs from a p-value. We also make explicit the
Bayesian nature of many recent criticisms, a dimension that is often
underplayed or ignored. We conclude by identifying practical steps to help
remediate some of the concerns identified. We recommend that (i) far lower
significance levels are used, such as $0.01$ or $0.001$, and (ii) p-values are
interpreted contextually, and situated within both the findings of the
individual study and the broader field of inquiry (through, for example,
meta-analyses).

negative:
Campbell-Goodhart's law relates to the causal inference error whereby
decision-making agents aim to influence variables which are correlated to their
goal objective but do not reliably cause it. This is a well known error in
Economics and Political Science but not widely labelled in Artificial
Intelligence research. Through a simple example, we show how off-the-shelf deep
Reinforcement Learning (RL) algorithms are not necessarily immune to this
cognitive error. The off-policy learning method is tricked, whilst the
on-policy method is not. The practical implication is that naive application of
RL to complex real life problems can result in the same types of policy errors
that humans make. Great care should be taken around understanding the causal
model that underpins a solution derived from Reinforcement Learning.

negative:
In this paper we introduce DeepCrawl, a fully-playable Roguelike prototype
for iOS and Android in which all agents are controlled by policy networks
trained using Deep Reinforcement Learning (DRL). Our aim is to understand
whether recent advances in DRL can be used to develop convincing behavioral
models for non-player characters in videogames. We begin with an analysis of
requirements that such an AI system should satisfy in order to be practically
applicable in video game development, and identify the elements of the DRL
model used in the DeepCrawl prototype. The successes and limitations of
DeepCrawl are documented through a series of playability tests performed on the
final game. We believe that the techniques we propose offer insight into
innovative new avenues for the development of behaviors for non-player
characters in video games, as they offer the potential to overcome critical
issues with

negative:
In this paper, we study how to learn an appropriate lane changing strategy
for autonomous vehicles by using deep reinforcement learning. We show that the
reward of the system should consider the overall traffic efficiency instead of
the travel efficiency of an individual vehicle. In summary, cooperation leads
to a more harmonic and efficient traffic system rather than competition

negative:
A new design of an AI assistant that has become increasingly popular is that
of an "aggregated assistant" -- realized as an orchestrated composition of
several individual skills or agents that can each perform atomic tasks. In this
paper, we will talk about the role of planning in the automated composition of
such assistants and explore how concepts in automated planning can help to
establish transparency of the inner workings of the assistant to the end-user.

negative:
Moores Paradox is a test case for any formal theory of belief. In Knowledge
and Belief, Hintikka developed a multimodal logic for statements that express
sentences containing the epistemic notions of knowledge and belief. His account
purports to offer an explanation of the paradox. In this paper I argue that
Hintikkas interpretation of one of the doxastic operators is philosophically
problematic and leads to an unnecessarily strong logical system. I offer a
weaker alternative that captures in a more accurate way our logical intuitions
about the notion of belief without sacrificing the possibility of providing an
explanation for problematic cases such as Moores Paradox.

negative:
This paper presents a plausible reasoning system to illustrate some broad
issues in knowledge representation: dualities between different reasoning
forms, the difficulty of unifying complementary reasoning styles, and the
approximate nature of plausible reasoning. These issues have a common
underlying theme: there should be an underlying belief calculus of which the
many different reasoning forms are special cases, sometimes approximate. The
system presented allows reasoning about defaults, likelihood, necessity and
possibility in a manner similar to the earlier work of Adams. The system is
based on the belief calculus of subjective Bayesian probability which itself is
based on a few simple assumptions about how belief should be manipulated.
Approximations, semantics, consistency and consequence results are presented
for the system. While this puts these often discussed plausible reasoning forms
on a probabilistic footing, useful application to practical problems remains an
issue.

negative:
This paper makes a first step towards a logic of learning from experiments.
For this, we investigate formal frameworks for modeling the interaction of
causal and (qualitative) epistemic reasoning. Crucial for our approach is the
idea that the notion of an intervention can be used as a formal expression of a
(real or hypothetical) experiment. In a first step we extend the well-known
causal models with a simple Hintikka-style representation of the epistemic
state of an agent. In the resulting setting, one can talk not only about the
knowledge of an agent about the values of variables and how interventions
affect them, but also about knowledge update. The resulting logic can model
reasoning about thought experiments. However, it is unable to account for
learning from experiments, which is clearly brought out by the fact that it
validates the no learning principle for interventions. Therefore, in a second
step, we implement a more complex notion of knowledge that allows an agent to
observe (measure) certain variables when an experiment is carried out. This
extended system does allow for learning from experiments. For all the proposed
logical systems, we provide a sound and complete axiomatization.

negative:
Visualization tools for supervised learning have allowed users to interpret,
introspect, and gain intuition for the successes and failures of their models.
While reinforcement learning practitioners ask many of the same questions,
existing tools are not applicable to the RL setting. In this work, we describe
our initial attempt at constructing a prototype of these ideas, through
identifying possible features that such a system should encapsulate. Our design
is motivated by envisioning the system to be a platform on which to experiment
with interpretable reinforcement learning.

negative:
Knowledge graphs in manufacturing and production aim to make production lines
more efficient and flexible with higher quality output. This makes knowledge
graphs attractive for companies to reach Industry 4.0 goals. However, existing
research in the field is quite preliminary, and more research effort on
analyzing how knowledge graphs can be applied in the field of manufacturing and
production is needed. Therefore, we have conducted a systematic literature
review as an attempt to characterize the state-of-the-art in this field, i.e.,
by identifying exiting research and by identifying gaps and opportunities for
further research. To do that, we have focused on finding the primary studies in
the existing literature, which were classified and analyzed according to four
criteria: bibliometric key facts, research type facets, knowledge graph
characteristics, and application scenarios. Besides, an evaluation of the
primary studies has also been carried out to gain deeper insights in terms of
methodology, empirical evidence, and relevance. As a result, we can offer a
complete picture of the domain, which includes such interesting aspects as the
fact that knowledge fusion is currently the main use case for knowledge graphs,
that empirical research and industrial application are still missing to a large
extent, that graph embeddings are not fully exploited, and that technical
literature is fast-growing but seems to be still far from its peak.

negative:
Systematic literature reviews play a vital role in identifying the best
available evidence for health and social care policy. The resources required to
produce systematic reviews can be significant, and a key to the success of any
review is the search strategy used to identify relevant literature. However,
the methods used to construct search strategies can be complex, time consuming,
resource intensive and error prone. In this review, we examine the state of the
art in resolving complex structured information needs, focusing primarily on
the healthcare context. We analyse the literature to identify key challenges
and issues and explore appropriate solutions and workarounds. From this
analysis we propose a way forward to facilitate trust and transparency and to
aid explainability, reproducibility and replicability through a set of key
design principles for tools to support the development of search strategies in
systematic literature reviews.

negative:
This paper describes an application of reinforcement learning to the mention
detection task. We define a novel action-based formulation for the mention
detection task, in which a model can flexibly revise past labeling decisions by
grouping together tokens and assigning partial mention labels. We devise a
method to create mention-level episodes and we train a model by rewarding
correctly labeled complete mentions, irrespective of the inner structure
created. The model yields results which are on par with a competitive
supervised counterpart while being more flexible in terms of achieving targeted
behavior through reward modeling and generating internal mention structure,
especially on longer mentions.

negative:
Two interpretations about syllogistic statements are described in this paper.
One is the so-called set-based interpretation, which assumes that quantified
statements and syllogisms talk about quantity-relationships between sets. The
other one, the so-called conditional interpretation, assumes that quantified
propositions talk about conditional propositions and how strong are the links
between the antecedent and the consequent. Both interpretations are compared
attending to three different questions (existential import, singular statements
and non-proportional quantifiers) from the point of view of their impact on the
further development of this type of reasoning.

negative:
Currently, knowledge discovery in databases is an essential step to identify
valid, novel and useful patterns for decision making. There are many real-world
scenarios, such as bankruptcy prediction, option pricing or medical diagnosis,
where the classification models to be learned need to fulfil restrictions of
monotonicity (i.e. the target class label should not decrease when input
attributes values increase). For instance, it is rational to assume that a
higher debt ratio of a company should never result in a lower level of
bankruptcy risk. Consequently, there is a growing interest from the data mining
research community concerning monotonic predictive models. This paper aims to
present an overview about the literature in the field, analyzing existing
techniques and proposing a taxonomy of the algorithms based on the type of
model generated. For each method, we review the quality metrics considered in
the evaluation and the different data sets and monotonic problems used in the
analysis. In this way, this paper serves as an overview of the research about
monotonic classification in specialized literature and can be used as a
functional guide of the field.

negative:
We evaluate four computational models of explanation in Bayesian networks by
comparing model predictions to human judgments. In two experiments, we present
human participants with causal structures for which the models make divergent
predictions and either solicit the best explanation for an observed event
(Experiment 1) or have participants rate provided explanations for an observed
event (Experiment 2). Across two versions of two causal structures and across
both experiments, we find that the Causal Explanation Tree and Most Relevant
Explanation models provide better fits to human data than either Most Probable
Explanation or Explanation Tree models. We identify strengths and shortcomings
of these models and what they can reveal about human explanation. We conclude
by suggesting the value of pursuing computational and psychological
investigations of explanation in parallel.

negative:
Human input has enabled autonomous systems to improve their capabilities and
achieve complex behaviors that are otherwise challenging to generate
automatically. Recent work focuses on how robots can use such input - like
demonstrations or corrections - to learn intended objectives. These techniques
assume that the human's desired objective already exists within the robot's
hypothesis space. In reality, this assumption is often inaccurate: there will
always be situations where the person might care about aspects of the task that
the robot does not know about. Without this knowledge, the robot cannot infer
the correct objective. Hence, when the robot's hypothesis space is
misspecified, even methods that keep track of uncertainty over the objective
fail because they reason about which hypothesis might be correct, and not
whether any of the hypotheses are correct. In this paper, we posit that the
robot should reason explicitly about how well it can explain human inputs given
its hypothesis space and use that situational confidence to inform how it
should incorporate human input. We demonstrate our method on a 7
degree-of-freedom robot manipulator in learning from two important types of
human input: demonstrations of manipulation tasks, and physical corrections
during the robot's task execution.

negative:
There is knowledge. There is belief. And there is tacit agreement.' 'We may
talk about objects. We may talk about attributes of the objects. Or we may talk
both about objects and their attributes.' This work inspects tacit agreements
on assumptions about the relation between objects and their attributes, and
studies a way of expressing them, presenting as the result what we term gradual
logic in which the sense of truth gradually shifts. It extends classical logic
instances with a new logical connective capturing the object-attribute
relation. A formal semantics is presented. Decidability is proved. Para-
consistent/epistemic/conditional/intensional/description/combined logics are
compared.

negative:
Multi-agent reinforcement learning offers a way to study how communication
could emerge in communities of agents needing to solve specific problems. In
this paper, we study the emergence of communication in the negotiation
environment, a semi-cooperative model of agent interaction. We introduce two
communication protocols -- one grounded in the semantics of the game, and one
which is \textit{a priori} ungrounded and is a form of cheap talk. We show that
self-interested agents can use the pre-grounded communication channel to
negotiate fairly, but are unable to effectively use the ungrounded channel.
However, prosocial agents do learn to use cheap talk to find an optimal
negotiating strategy, suggesting that cooperation is necessary for language to
emerge. We also study communication behaviour in a setting where one agent
interacts with agents in a community with different levels of prosociality and
show how agent identifiability can aid negotiation.

negative:
In the era of intelligent biohybrid neurotechnologies for brain repair, new
fanciful terms are appearing in the scientific dictionary to define what has so
far been unimaginable. As the emerging neurotechnologies are becoming
increasingly polyhedral and sophisticated, should we talk about evolution and
rank the intelligence of these devices?

negative:
We argue that logical semantics might have faltered due to its failure in
distinguishing between two fundamentally very different types of concepts:
ontological concepts, that should be types in a strongly-typed ontology, and
logical concepts, that are predicates corresponding to properties of and
relations between objects of various ontological types. We will then show that
accounting for these differences amounts to the integration of lexical and
compositional semantics in one coherent framework, and to an embedding in our
logical semantics of a strongly-typed ontology that reflects our commonsense
view of the world and the way we talk about it in ordinary language. We will
show that in such a framework a number of challenges in natural language
semantics can be adequately and systematically treated.