query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I aim to develop a reinforcement learning framework that utilizes human feedback as sparse reward signals. This framework will have the capacity to incorporate any model for a variety of potential downstream tasks, such as a generative model for art creation or a chatbot for customer service. The unique aspect of my framework is its ability to apply reinforcement learning to adjust and optimize the behavior of any downstream model based on various forms of human feedback, such as user satisfaction or the model's adherence to ethical standards. I understand that obtaining a large amount of human feedback data may be challenging, so my framework needs to be capable of learning from sparse and limited human feedback data. I am interested in finding out if there has been any previous work in this area. Furthermore, I would like to know if a reinforcement learning algorithm would still function effectively when the supervised human feedback data is scarce.

positive:
Recent successes combine reinforcement learning algorithms and deep neural
networks, despite reinforcement learning not being widely applied to robotics
and real world scenarios. This can be attributed to the fact that current
state-of-the-art, end-to-end reinforcement learning approaches still require
thousands or millions of data samples to converge to a satisfactory policy and
are subject to catastrophic failures during training. Conversely, in real world
scenarios and after just a few data samples, humans are able to either provide
demonstrations of the task, intervene to prevent catastrophic actions, or
simply evaluate if the policy is performing correctly. This research
investigates how to integrate these human interaction modalities to the
reinforcement learning loop, increasing sample efficiency and enabling
real-time reinforcement learning in robotics and real world scenarios. This
novel theoretical foundation is called Cycle-of-Learning, a reference to how
different human interaction modalities, namely, task demonstration,
intervention, and evaluation, are cycled and combined to reinforcement learning
algorithms. Results presented in this work show that the reward signal that is
learned based upon human interaction accelerates the rate of learning of
reinforcement learning algorithms and that learning from a combination of human
demonstrations and interventions is faster and more sample efficient when
compared to traditional supervised learning algorithms. Finally,
Cycle-of-Learning develops an effective transition between policies learned
using human demonstrations and interventions to reinforcement learning. The
theoretical foundation developed by this research opens new research paths to
human-agent teaming scenarios where autonomous agents are able to learn from
human teammates and adapt to mission performance metrics in real-time and in
real world scenarios.

positive:
To widen their accessibility and increase their utility, intelligent agents
must be able to learn complex behaviors as specified by (non-expert) human
users. Moreover, they will need to learn these behaviors within a reasonable
amount of time while efficiently leveraging the sparse feedback a human trainer
is capable of providing. Recent work has shown that human feedback can be
characterized as a critique of an agent's current behavior rather than as an
alternative reward signal to be maximized, culminating in the COnvergent
Actor-Critic by Humans (COACH) algorithm for making direct policy updates based
on human feedback. Our work builds on COACH, moving to a setting where the
agent's policy is represented by a deep neural network. We employ a series of
modifications on top of the original COACH algorithm that are critical for
successfully learning behaviors from high-dimensional observations, while also
satisfying the constraint of obtaining reduced sample complexity. We
demonstrate the effectiveness of our Deep COACH algorithm in the rich 3D world
of Minecraft with an agent that learns to complete tasks by mapping from raw
pixels to actions using only real-time human feedback in 10-15 minutes of
interaction.

positive:
Reinforcement learning has been successful in training autonomous agents to
accomplish goals in complex environments. Although this has been adapted to
multiple settings, including robotics and computer games, human players often
find it easier to obtain higher rewards in some environments than reinforcement
learning algorithms. This is especially true of high-dimensional state spaces
where the reward obtained by the agent is sparse or extremely delayed. In this
paper, we seek to effectively integrate feedback signals supplied by a human
operator with deep reinforcement learning algorithms in high-dimensional state
spaces. We call this FRESH (Feedback-based REward SHaping). During training, a
human operator is presented with trajectories from a replay buffer and then
provides feedback on states and actions in the trajectory. In order to
generalize feedback signals provided by the human operator to previously unseen
states and actions at test-time, we use a feedback neural network. We use an
ensemble of neural networks with a shared network architecture to represent
model uncertainty and the confidence of the neural network in its output. The
output of the feedback neural network is converted to a shaping reward that is
augmented to the reward provided by the environment. We evaluate our approach
on the Bowling and Skiing Atari games in the arcade learning environment.
Although human experts have been able to achieve high scores in these
environments, state-of-the-art deep learning algorithms perform poorly. We
observe that FRESH is able to achieve much higher scores than state-of-the-art
deep learning algorithms in both environments. FRESH also achieves a 21.4%
higher score than a human expert in Bowling and does as well as a human expert
in Skiing.

positive:
Human explanation (e.g., in terms of feature importance) has been recently
used to extend the communication channel between human and agent in interactive
machine learning. Under this setting, human trainers provide not only the
ground truth but also some form of explanation. However, this kind of human
guidance was only investigated in supervised learning tasks, and it remains
unclear how to best incorporate this type of human knowledge into deep
reinforcement learning. In this paper, we present the first study of using
human visual explanations in human-in-the-loop reinforcement learning (HRL). We
focus on the task of learning from feedback, in which the human trainer not
only gives binary evaluative "good" or "bad" feedback for queried state-action
pairs, but also provides a visual explanation by annotating relevant features
in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the
model to encode task-relevant features through a context-aware data
augmentation that only perturbs irrelevant features in human salient
information. We choose five tasks, namely Pixel-Taxi and four Atari games, to
evaluate the performance and sample efficiency of this approach. We show that
our method significantly outperforms methods leveraging human explanation that
are adapted from supervised learning, and Human-in-the-loop RL baselines that
only utilize evaluative feedback.

positive:
Deep Reinforcement Learning (DRL) has become a powerful methodology to solve
complex decision-making problems. However, DRL has several limitations when
used in real-world problems (e.g., robotics applications). For instance, long
training times are required and cannot be accelerated in contrast to simulated
environments, and reward functions may be hard to specify/model and/or to
compute. Moreover, the transfer of policies learned in a simulator to the
real-world has limitations (reality gap). On the other hand, machine learning
methods that rely on the transfer of human knowledge to an agent have shown to
be time efficient for obtaining well performing policies and do not require a
reward function. In this context, we analyze the use of human corrective
feedback during task execution to learn policies with high-dimensional state
spaces, by using the D-COACH framework, and we propose new variants of this
framework. D-COACH is a Deep Learning based extension of COACH (COrrective
Advice Communicated by Humans), where humans are able to shape policies through
corrective advice. The enhanced version of D-COACH, which is proposed in this
paper, largely reduces the time and effort of a human for training a policy.
Experimental results validate the efficiency of the D-COACH framework in three
different problems (simulated and with real robots), and show that its enhanced
version reduces the human training effort considerably, and makes it feasible
to learn policies within periods of time in which a DRL agent do not reach any
improvement.

positive:
Conveying complex objectives to reinforcement learning (RL) agents can often
be difficult, involving meticulous design of reward functions that are
sufficiently informative yet easy enough to provide. Human-in-the-loop RL
methods allow practitioners to instead interactively teach agents through
tailored feedback; however, such approaches have been challenging to scale
since human feedback is very expensive. In this work, we aim to make this
process more sample- and feedback-efficient. We present an off-policy,
interactive RL algorithm that capitalizes on the strengths of both feedback and
off-policy learning. Specifically, we learn a reward model by actively querying
a teacher's preferences between two clips of behavior and use it to train an
agent. To enable off-policy learning, we relabel all the agent's past
experience when its reward model changes. We additionally show that
pre-training our agents with unsupervised exploration substantially increases
the mileage of its queries. We demonstrate that our approach is capable of
learning tasks of higher complexity than previously considered by
human-in-the-loop methods, including a variety of locomotion and robotic
manipulation skills. We also show that our method is able to utilize real-time
human feedback to effectively prevent reward exploitation and learn new
behaviors that are difficult to specify with standard reward functions.

positive:
While reinforcement learning (RL) has been successful in natural language
processing (NLP) domains such as dialogue generation and text-based games, it
typically faces the problem of sparse rewards that leads to slow or no
convergence. Traditional methods that use text descriptions to extract only a
state representation ignore the feedback inherently present in them. In
text-based games, for example, descriptions like "Good Job! You ate the food}"
indicate progress, and descriptions like "You entered a new room" indicate
exploration. Positive and negative cues like these can be converted to rewards
through sentiment analysis. This technique converts the sparse reward problem
into a dense one, which is easier to solve. Furthermore, this can enable
reinforcement learning without rewards, in which the agent learns entirely from
these intrinsic sentiment rewards. This framework is similar to intrinsic
motivation, where the environment does not necessarily provide the rewards, but
the agent analyzes and realizes them by itself. We find that providing dense
rewards in text-based games using sentiment analysis improves performance under
some conditions.

positive:
Reinforcement learning (RL) requires skillful definition and remarkable
computational efforts to solve optimization and control problems, which could
impair its prospect. Introducing human guidance into reinforcement learning is
a promising way to improve learning performance. In this paper, a comprehensive
human guidance-based reinforcement learning framework is established. A novel
prioritized experience replay mechanism that adapts to human guidance in the
reinforcement learning process is proposed to boost the efficiency and
performance of the reinforcement learning algorithm. To relieve the heavy
workload on human participants, a behavior model is established based on an
incremental online learning method to mimic human actions. We design two
challenging autonomous driving tasks for evaluating the proposed algorithm.
Experiments are conducted to access the training and testing performance and
learning mechanism of the proposed algorithm. Comparative results against the
state-of-the-art methods suggest the advantages of our algorithm in terms of
learning efficiency, performance, and robustness.

positive:
This paper contributes a preliminary report on the advantages and
disadvantages of incorporating simultaneous human control and feedback signals
in the training of a reinforcement learning robotic agent. While robotic
human-machine interfaces have become increasingly complex in both form and
function, control remains challenging for users. This has resulted in an
increasing gap between user control approaches and the number of robotic motors
which can be controlled. One way to address this gap is to shift some autonomy
to the robot. Semi-autonomous actions of the robotic agent can then be shaped
by human feedback, simplifying user control. Most prior work on agent shaping
by humans has incorporated training with feedback, or has included indirect
control signals. By contrast, in this paper we explore how a human can provide
concurrent feedback signals and real-time myoelectric control signals to train
a robot's actor-critic reinforcement learning control system. Using both a
physical and a simulated robotic system, we compare training performance on a
simple movement task when reward is derived from the environment, when reward
is provided by the human, and combinations of these two approaches. Our results
indicate that some benefit can be gained with the inclusion of human generated
feedback.

positive:
Interactive reinforcement learning agents use human feedback or instruction
to help them learn in complex environments. Often, this feedback comes in the
form of a discrete signal that is either positive or negative. While
informative, this information can be difficult to generalize on its own. In
this work, we explore how natural language advice can be used to provide a
richer feedback signal to a reinforcement learning agent by extending policy
shaping, a well-known Interactive reinforcement learning technique. Usually
policy shaping employs a human feedback policy to help an agent to learn more
about how to achieve its goal. In our case, we replace this human feedback
policy with policy generated based on natural language advice. We aim to
inspect if the generated natural language reasoning provides support to a deep
reinforcement learning agent to decide its actions successfully in any given
environment. So, we design our model with three networks: first one is the
experience driven, next is the advice generator and third one is the advice
driven. While the experience driven reinforcement learning agent chooses its
actions being influenced by the environmental reward, the advice driven neural
network with generated feedback by the advice generator for any new state
selects its actions to assist the reinforcement learning agent to better policy
shaping.

positive:
While recent advances in deep reinforcement learning have allowed autonomous
learning agents to succeed at a variety of complex tasks, existing algorithms
generally require a lot of training data. One way to increase the speed at
which agents are able to learn to perform tasks is by leveraging the input of
human trainers. Although such input can take many forms, real-time,
scalar-valued feedback is especially useful in situations where it proves
difficult or impossible for humans to provide expert demonstrations. Previous
approaches have shown the usefulness of human input provided in this fashion
(e.g., the TAMER framework), but they have thus far not considered
high-dimensional state spaces or employed the use of deep learning. In this
paper, we do both: we propose Deep TAMER, an extension of the TAMER framework
that leverages the representational power of deep neural networks in order to
learn complex tasks in just a short amount of time with a human trainer. We
demonstrate Deep TAMER's success by using it and just 15 minutes of
human-provided feedback to train an agent that performs better than humans on
the Atari game of Bowling - a task that has proven difficult for even
state-of-the-art reinforcement learning methods.

positive:
We describe a method to use discrete human feedback to enhance the
performance of deep learning agents in virtual three-dimensional environments
by extending deep-reinforcement learning to model the confidence and
consistency of human feedback. This enables deep reinforcement learning
algorithms to determine the most appropriate time to listen to the human
feedback, exploit the current policy model, or explore the agent's environment.
Managing the trade-off between these three strategies allows DRL agents to be
robust to inconsistent or intermittent human feedback. Through experimentation
using a synthetic oracle, we show that our technique improves the training
speed and overall performance of deep reinforcement learning in navigating
three-dimensional environments using Minecraft. We further show that our
technique is robust to highly innacurate human feedback and can also operate
when no human feedback is given.

positive:
In reinforcement learning (RL), sparse rewards are a natural way to specify
the task to be learned. However, most RL algorithms struggle to learn in this
setting since the learning signal is mostly zeros. In contrast, humans are good
at assessing and predicting the future consequences of actions and can serve as
good reward/policy shapers to accelerate the robot learning process. Previous
works have shown that the human brain generates an error-related signal,
measurable using electroencephelography (EEG), when the human perceives the
task being done erroneously. In this work, we propose a method that uses
evaluative feedback obtained from human brain signals measured via scalp EEG to
accelerate RL for robotic agents in sparse reward settings. As the robot learns
the task, the EEG of a human observer watching the robot attempts is recorded
and decoded into noisy error feedback signal. From this feedback, we use
supervised learning to obtain a policy that subsequently augments the behavior
policy and guides exploration in the early stages of RL. This bootstraps the RL
learning process to enable learning from sparse reward. Using a robotic
navigation task as a test bed, we show that our method achieves a stable
obstacle-avoidance policy with high success rate, outperforming learning from
sparse rewards only that struggles to achieve obstacle avoidance behavior or
fails to advance to the goal.

positive:
How can we train a dialog model to produce better conversations by learning
from human feedback, without the risk of humans teaching it harmful chat
behaviors? We start by hosting models online, and gather human feedback from
real-time, open-ended conversations, which we then use to train and improve the
models using offline reinforcement learning (RL). We identify implicit
conversational cues including language similarity, elicitation of laughter,
sentiment, and more, which indicate positive human feedback, and embed these in
multiple reward functions. A well-known challenge is that learning an RL policy
in an offline setting usually fails due to the lack of ability to explore and
the tendency to make over-optimistic estimates of future reward. These problems
become even harder when using RL for language models, which can easily have a
20,000 action vocabulary and many possible reward functions. We solve the
challenge by developing a novel class of offline RL algorithms. These
algorithms use KL-control to penalize divergence from a pre-trained prior
language model, and use a new strategy to make the algorithm pessimistic,
instead of optimistic, in the face of uncertainty. We test the resulting dialog
model with ratings from 80 users in an open-domain setting and find it achieves
significant improvements over existing deep offline RL approaches. The novel
offline RL method is viable for improving any existing generative dialog model
using a static dataset of human feedback.

positive:
Reinforcement Learning AI commonly uses reward/penalty signals that are
objective and explicit in an environment -- e.g. game score, completion time,
etc. -- in order to learn the optimal strategy for task performance. However,
Human-AI interaction for such AI agents should include additional reinforcement
that is implicit and subjective -- e.g. human preferences for certain AI
behavior -- in order to adapt the AI behavior to idiosyncratic human
preferences. Such adaptations would mirror naturally occurring processes that
increase trust and comfort during social interactions. Here, we show how a
hybrid brain-computer-interface (hBCI), which detects an individual's level of
interest in objects/events in a virtual environment, can be used to adapt the
behavior of a Deep Reinforcement Learning AI agent that is controlling a
virtual autonomous vehicle. Specifically, we show that the AI learns a driving
strategy that maintains a safe distance from a lead vehicle, and most novelly,
preferentially slows the vehicle when the human passengers of the vehicle
encounter objects of interest. This adaptation affords an additional 20\%
viewing time for subjectively interesting objects. This is the first
demonstration of how an hBCI can be used to provide implicit reinforcement to
an AI agent in a way that incorporates user preferences into the control
system.

positive:
In autonomous vehicle (AV) control, allowing mistakes can be quite dangerous
and costly in the real world. For this reason we investigate methods of
training an AV without allowing the agent to explore and instead having a human
explorer collect the data. Supervised learning has been explored for AV
control, but it encounters the issue of the covariate shift. That is, training
data collected from an optimal demonstration consists only of the states
induced by the optimal control policy, but at runtime, the trained agent may
encounter a vastly different state distribution with little relevant training
data. To mitigate this issue, we have our human explorer make sub-optimal
decisions. In order to have our agent not replicate these sub-optimal
decisions, supervised learning requires that we either erase these actions, or
replace these action with the correct action. Erasing is wasteful and replacing
is difficult, since it is not easy to know the correct action without driving.
We propose an alternate framework that includes continuous scalar feedback for
each action, marking which actions we should replicate, which we should avoid,
and how sure we are. Our framework learns continuous control from sub-optimal
demonstration and evaluative feedback collected before training. We find that a
human demonstrator can explore sub-optimal states in a safe manner, while still
getting enough gradation to benefit learning. The collection method for data
and feedback we call "Backseat Driver." We call the more general learning
framework ReNeg, since it learns a regression from states to actions given
negative as well as positive examples. We empirically validate several models
in the ReNeg framework, testing on lane-following with limited data. We find
that the best solution is a generalization of mean-squared error and
outperforms supervised learning on the positive examples alone.

positive:
The current reward learning from human preferences could be used to resolve
complex reinforcement learning (RL) tasks without access to a reward function
by defining a single fixed preference between pairs of trajectory segments.
However, the judgement of preferences between trajectories is not dynamic and
still requires human input over thousands of iterations. In this study, we
proposed a weak human preference supervision framework, for which we developed
a human preference scaling model that naturally reflects the human perception
of the degree of weak choices between trajectories and established a
human-demonstration estimator via supervised learning to generate the predicted
preferences for reducing the number of human inputs. The proposed weak human
preference supervision framework can effectively solve complex RL tasks and
achieve higher cumulative rewards in simulated robot locomotion -- MuJoCo games
-- relative to the single fixed human preferences. Furthermore, our established
human-demonstration estimator requires human feedback only for less than 0.01\%
of the agent's interactions with the environment and significantly reduces the
cost of human inputs by up to 30\% compared with the existing approaches. To
present the flexibility of our approach, we released a video
(https://youtu.be/jQPe1OILT0M) showing comparisons of the behaviours of agents
trained on different types of human input. We believe that our naturally
inspired human preferences with weakly supervised learning are beneficial for
precise reward learning and can be applied to state-of-the-art RL systems, such
as human-autonomy teaming systems.

positive:
Fluid human-agent communication is essential for the future of
human-in-the-loop reinforcement learning. An agent must respond appropriately
to feedback from its human trainer even before they have significant experience
working together. Therefore, it is important that learning agents respond well
to various feedback schemes human trainers are likely to provide. This work
analyzes the COnvergent Actor-Critic by Humans (COACH) algorithm under three
different types of feedback-policy feedback, reward feedback, and advantage
feedback. For these three feedback types, we find that COACH can behave
sub-optimally. We propose a variant of COACH, episodic COACH (E-COACH), which
we prove converges for all three types. We compare our COACH variant with two
other reinforcement-learning algorithms: Q-learning and TAMER.

positive:
To solve complex real-world problems with reinforcement learning, we cannot
rely on manually specified reward functions. Instead, we can have humans
communicate an objective to the agent directly. In this work, we combine two
approaches to learning from human feedback: expert demonstrations and
trajectory preferences. We train a deep neural network to model the reward
function and use its predicted reward to train an DQN-based deep reinforcement
learning agent on 9 Atari games. Our approach beats the imitation learning
baseline in 7 games and achieves strictly superhuman performance on 2 games
without using game rewards. Additionally, we investigate the goodness of fit of
the reward model, present some reward hacking problems, and study the effects
of noise in the human labels.

positive:
This paper extends recent work in interactive machine learning (IML) focused
on effectively incorporating human feedback. We show how control and feedback
signals complement each other in systems which model human reward. We
demonstrate that simultaneously incorporating human control and feedback
signals can improve interactive robotic systems' performance on a self-mirrored
movement control task where an RL-agent controlled right arm attempts to match
the preprogrammed movement pattern of the left arm. We illustrate the impact of
varying human feedback parameters on task performance by investigating the
probability of giving feedback on each time step and the likelihood of given
feedback being correct. We further illustrate that varying the temporal decay
with which the agent incorporates human feedback has a significant impact on
task performance. We found that smearing human feedback over time steps
improves performance and we show varying the probability of feedback at each
time step, and an increased likelihood of those feedbacks being 'correct' can
impact agent performance. We conclude that understanding latent variables in
human feedback is crucial for learning algorithms acting in human-machine
interaction domains.

positive:
The standard feedback model of reinforcement learning requires revealing the
reward of every visited state-action pair. However, in practice, it is often
the case that such frequent feedback is not available. In this work, we take a
first step towards relaxing this assumption and require a weaker form of
feedback, which we refer to as \emph{trajectory feedback}. Instead of observing
the reward obtained after every action, we assume we only receive a score that
represents the quality of the whole trajectory observed by the agent, namely,
the sum of all rewards obtained over this trajectory. We extend reinforcement
learning algorithms to this setting, based on least-squares estimation of the
unknown reward, for both the known and unknown transition model cases, and
study the performance of these algorithms by analyzing their regret. For cases
where the transition model is unknown, we offer a hybrid optimistic-Thompson
Sampling approach that results in a tractable algorithm.

positive:
Counterfactual learning from human bandit feedback describes a scenario where
user feedback on the quality of outputs of a historic system is logged and used
to improve a target system. We show how to apply this learning framework to
neural semantic parsing. From a machine learning perspective, the key challenge
lies in a proper reweighting of the estimator so as to avoid known degeneracies
in counterfactual learning, while still being applicable to stochastic gradient
optimization. To conduct experiments with human users, we devise an easy-to-use
interface to collect human feedback on semantic parses. Our work is the first
to show that semantic parsers can be improved significantly by counterfactual
learning from logged human feedback data.

positive:
A promising approach to improve the robustness and exploration in
Reinforcement Learning is collecting human feedback and that way incorporating
prior knowledge of the target environment. It is, however, often too expensive
to obtain enough feedback of good quality. To mitigate the issue, we aim to
rely on a group of multiple experts (and non-experts) with different skill
levels to generate enough feedback. Such feedback can therefore be inconsistent
and infrequent. In this paper, we build upon prior work -- Advise, a Bayesian
approach attempting to maximise the information gained from human feedback --
extending the algorithm to accept feedback from this larger group of humans,
the trainers, while also estimating each trainer's reliability. We show how
aggregating feedback from multiple trainers improves the total feedback's
accuracy and make the collection process easier in two ways. Firstly, this
approach addresses the case of some of the trainers being adversarial.
Secondly, having access to the information about each trainer reliability
provides a second layer of robustness and offers valuable information for
people managing the whole system to improve the overall trust in the system. It
offers an actionable tool for improving the feedback collection process or
modifying the reward function design if needed. We empirically show that our
approach can accurately learn the reliability of each trainer correctly and use
it to maximise the information gained from the multiple trainers' feedback,
even if some of the sources are adversarial.

positive:
We present a study on reinforcement learning (RL) from human bandit feedback
for sequence-to-sequence learning, exemplified by the task of bandit neural
machine translation (NMT). We investigate the reliability of human bandit
feedback, and analyze the influence of reliability on the learnability of a
reward estimator, and the effect of the quality of reward estimates on the
overall RL task. Our analysis of cardinal (5-point ratings) and ordinal
(pairwise preferences) feedback shows that their intra- and inter-annotator
$\alpha$-agreement is comparable. Best reliability is obtained for standardized
cardinal feedback, and cardinal feedback is also easiest to learn and
generalize from. Finally, improvements of over 1 BLEU can be obtained by
integrating a regression-based reward estimator trained on cardinal feedback
for 800 translations into RL for NMT. This shows that RL is possible even from
small amounts of fairly reliable human feedback, pointing to a great potential
for applications at larger scale.

positive:
Successfully navigating a complex environment to obtain a desired outcome is
a difficult task, that up to recently was believed to be capable only by
humans. This perception has been broken down over time, especially with the
introduction of deep reinforcement learning, which has greatly increased the
difficulty of tasks that can be automated. However, for traditional
reinforcement learning agents this requires an environment to be able to
provide frequent extrinsic rewards, which are not known or accessible for many
real-world environments. This project aims to explore and contrast existing
reinforcement learning solutions that circumnavigate the difficulties of an
environment that provide sparse rewards. Different reinforcement solutions will
be implemented over a several video game environments with varying difficulty
and varying frequency of rewards, as to properly investigate the applicability
of these solutions. This project introduces a novel reinforcement learning
solution by combining aspects of two existing state of the art sparse reward
solutions, curiosity driven exploration and unsupervised auxiliary tasks.

positive:
Recent advancements in \textit{Learning from Human Feedback} present an
effective way to train robot agents via inputs from non-expert humans, without
a need for a specially designed reward function. However, this approach needs a
human to be present and attentive during robot learning to provide evaluative
feedback. In addition, the amount of feedback needed grows with the level of
task difficulty and the quality of human feedback might decrease over time
because of fatigue. To overcome these limitations and enable learning more
robot tasks with higher complexities, there is a need to maximize the quality
of expensive feedback received and reduce the amount of human cognitive
involvement required. In this work, we present an approach that uses active
learning to smartly choose queries for the human supervisor based on the
uncertainty of the robot and effectively reduces the amount of feedback needed
to learn a given task. We also use a novel multiple buffer system to improve
robustness to feedback noise and guard against catastrophic forgetting as the
robot learning evolves. This makes it possible to learn tasks with more
complexity using lesser amounts of human feedback compared to previous methods.
We demonstrate the utility of our proposed method on a robot arm reaching task
where the robot learns to reach a location in 3D without colliding with
obstacles. Our approach is able to learn this task faster, with less human
feedback and cognitive involvement, compared to previous methods that do not
use active learning.

positive:
Reward function, as an incentive representation that recognizes humans'
agency and rationalizes humans' actions, is particularly appealing for modeling
human behavior in human-robot interaction. Inverse Reinforcement Learning is an
effective way to retrieve reward functions from demonstrations. However, it has
always been challenging when applying it to multi-agent settings since the
mutual influence between agents has to be appropriately modeled. To tackle this
challenge, previous work either exploits equilibrium solution concepts by
assuming humans as perfectly rational optimizers with unbounded intelligence or
pre-assigns humans' interaction strategies a priori. In this work, we advocate
that humans are bounded rational and have different intelligence levels when
reasoning about others' decision-making process, and such an inherent and
latent characteristic should be accounted for in reward learning algorithms.
Hence, we exploit such insights from Theory-of-Mind and propose a new
multi-agent Inverse Reinforcement Learning framework that reasons about humans'
latent intelligence levels during learning. We validate our approach in both
zero-sum and general-sum games with synthetic agents and illustrate a practical
application to learning human drivers' reward functions from real driving data.
We compare our approach with two baseline algorithms. The results show that by
reasoning about humans' latent intelligence levels, the proposed approach has
more flexibility and capability to retrieve reward functions that explain
humans' driving behaviors better.

positive:
It is often difficult to hand-specify what the correct reward function is for
a task, so researchers have instead aimed to learn reward functions from human
behavior or feedback. The types of behavior interpreted as evidence of the
reward function have expanded greatly in recent years. We've gone from
demonstrations, to comparisons, to reading into the information leaked when the
human is pushing the robot away or turning it off. And surely, there is more to
come. How will a robot make sense of all these diverse types of behavior? Our
key insight is that different types of behavior can be interpreted in a single
unifying formalism - as a reward-rational choice that the human is making,
often implicitly. The formalism offers both a unifying lens with which to view
past work, as well as a recipe for interpreting new sources of information that
are yet to be uncovered. We provide two examples to showcase this: interpreting
a new feedback type, and reading into how the choice of feedback itself leaks
information about the reward.

positive:
Traditional collaborative filtering (CF) based recommender systems tend to
perform poorly when the user-item interactions/ratings are highly scarce. To
address this, we propose a learning framework that improves collaborative
filtering with a synthetic feedback loop (CF-SFL) to simulate the user
feedback. The proposed framework consists of a "recommender" and a "virtual
user". The "recommender" is formulated as a CF model, recommending items
according to observed user preference. The "virtual user" estimates rewards
from the recommended items and generates a \emph{feedback} in addition to the
observed user preference. The "recommender" connected with the "virtual user"
constructs a closed loop, that recommends users with items and imitates the
\emph{unobserved} feedback of the users to the recommended items. The synthetic
feedback is used to augment the observed user preference and improve
recommendation results. Theoretically, such model design can be interpreted as
inverse reinforcement learning, which can be learned effectively via rollout
(simulation). Experimental results show that the proposed framework is able to
enrich the learning of user preference and boost the performance of existing
collaborative filtering methods on multiple datasets.

positive:
The capability to interactively learn from human feedback would enable agents
in new settings. For example, even novice users could train service robots in
new tasks naturally and interactively. Human-in-the-loop Reinforcement Learning
(HRL) combines human feedback and Reinforcement Learning (RL) techniques.
State-of-the-art interactive learning techniques suffer from slow learning
speed, thus leading to a frustrating experience for the human. We approach this
problem by extending the HRL framework TAMER for evaluative feedback with the
possibility to enhance human feedback with two different types of
counterfactual explanations (action and state based). We experimentally show
that our extensions improve the speed of learning.

positive:
This paper contributes a first study into how different human users deliver
simultaneous control and feedback signals during human-robot interaction. As
part of this work, we formalize and present a general interactive learning
framework for online cooperation between humans and reinforcement learning
agents. In many human-machine interaction settings, there is a growing gap
between the degrees-of-freedom of complex semi-autonomous systems and the
number of human control channels. Simple human control and feedback mechanisms
are required to close this gap and allow for better collaboration between
humans and machines on complex tasks. To better inform the design of concurrent
control and feedback interfaces, we present experimental results from a
human-robot collaborative domain wherein the human must simultaneously deliver
both control and feedback signals to interactively train an actor-critic
reinforcement learning robot. We compare three experimental conditions: 1)
human delivered control signals, 2) reward-shaping feedback signals, and 3)
simultaneous control and feedback. Our results suggest that subjects provide
less feedback when simultaneously delivering feedback and control signals and
that control signal quality is not significantly diminished. Our data suggest
that subjects may also modify when and how they provide feedback. Through
algorithmic development and tuning informed by this study, we expect
semi-autonomous actions of robotic agents can be better shaped by human
feedback, allowing for seamless collaboration and improved performance in
difficult interactive domains.

positive:
A long-term goal of reinforcement learning agents is to be able to perform
tasks in complex real-world scenarios. The use of external information is one
way of scaling agents to more complex problems. However, there is a general
lack of collaboration or interoperability between different approaches using
external information. In this work, while reviewing externally-influenced
methods, we propose a conceptual framework and taxonomy for assisted
reinforcement learning, aimed at fostering collaboration by classifying and
comparing various methods that use external information in the learning
process. The proposed taxonomy details the relationship between the external
information source and the learner agent, highlighting the process of
information decomposition, structure, retention, and how it can be used to
influence agent learning. As well as reviewing state-of-the-art methods, we
identify current streams of reinforcement learning that use external
information in order to improve the agent's performance and its decision-making
process. These include heuristic reinforcement learning, interactive
reinforcement learning, learning from demonstration, transfer learning, and
learning from multiple sources, among others. These streams of reinforcement
learning operate with the shared objective of scaffolding the learner agent.
Lastly, we discuss further possibilities for future work in the field of
assisted reinforcement learning systems.

positive:
In this paper, we propose a framework that enables a human teacher to shape a
robot behaviour by interactively providing it with unlabeled instructions. We
ground the meaning of instruction signals in the task-learning process, and use
them simultaneously for guiding the latter. We implement our framework as a
modular architecture, named TICS (Task-Instruction-Contingency-Shaping) that
combines different information sources: a predefined reward function, human
evaluative feedback and unlabeled instructions. This approach provides a novel
perspective for robotic task learning that lies between Reinforcement Learning
and Supervised Learning paradigms. We evaluate our framework both in simulation
and with a real robot. The experimental results demonstrate the effectiveness
of our framework in accelerating the task-learning process and in reducing the
number of required teaching signals.

positive:
The combination of deep neural network models and reinforcement learning
algorithms can make it possible to learn policies for robotic behaviors that
directly read in raw sensory inputs, such as camera images, effectively
subsuming both estimation and control into one model. However, real-world
applications of reinforcement learning must specify the goal of the task by
means of a manually programmed reward function, which in practice requires
either designing the very same perception pipeline that end-to-end
reinforcement learning promises to avoid, or else instrumenting the environment
with additional sensors to determine if the task has been performed
successfully. In this paper, we propose an approach for removing the need for
manual engineering of reward specifications by enabling a robot to learn from a
modest number of examples of successful outcomes, followed by actively
solicited queries, where the robot shows the user a state and asks for a label
to determine whether that state represents successful completion of the task.
While requesting labels for every single state would amount to asking the user
to manually provide the reward signal, our method requires labels for only a
tiny fraction of the states seen during training, making it an efficient and
practical approach for learning skills without manually engineered rewards. We
evaluate our method on real-world robotic manipulation tasks where the
observations consist of images viewed by the robot's camera. In our
experiments, our method effectively learns to arrange objects, place books, and
drape cloth, directly from images and without any manually specified reward
functions, and with only 1-4 hours of interaction with the real world.

positive:
Reinforcement learning usually uses the feedback rewards of environmental to
train agents. But the rewards in the actual environment are sparse, and even
some environments will not rewards. Most of the current methods are difficult
to get good performance in sparse reward or non-reward environments. Although
using shaped rewards is effective when solving sparse reward tasks, it is
limited to specific problems and learning is also susceptible to local optima.
We propose a model-free method that does not rely on environmental rewards to
solve the problem of sparse rewards in the general environment. Our method use
the minimum number of transitions between states as the distance to replace the
rewards of environmental, and proposes a goal-distance gradient to achieve
policy improvement. We also introduce a bridge point planning method based on
the characteristics of our method to improve exploration efficiency, thereby
solving more complex tasks. Experiments show that our method performs better on
sparse reward and local optimal problems in complex environments than previous
work.

positive:
Deep reinforcement learning has obtained significant breakthroughs in recent
years. Most methods in deep-RL achieve good results via the maximization of the
reward signal provided by the environment, typically in the form of discounted
cumulative returns. Such reward signals represent the immediate feedback of a
particular action performed by an agent. However, tasks with sparse reward
signals are still challenging to on-policy methods. In this paper, we introduce
an effective characterization of past reward statistics (which can be seen as
long-term feedback signals) to supplement this immediate reward feedback. In
particular, value functions are learned with multi-critics supervision,
enabling complex value functions to be more easily approximated in on-policy
learning, even when the reward signals are sparse. We also introduce a novel
exploration mechanism called "hot-wiring" that can give a boost to seemingly
trapped agents. We demonstrate the effectiveness of our advantage actor
multi-critic (A2MC) method across the discrete domains in Atari games as well
as continuous domains in the MuJoCo environments. A video demo is provided at
https://youtu.be/zBmpf3Yz8tc.

positive:
As language models become more powerful, training and evaluation are
increasingly bottlenecked by the data and metrics used for a particular task.
For example, summarization models are often trained to predict human reference
summaries and evaluated using ROUGE, but both of these metrics are rough
proxies for what we really care about -- summary quality. In this work, we show
that it is possible to significantly improve summary quality by training a
model to optimize for human preferences. We collect a large, high-quality
dataset of human comparisons between summaries, train a model to predict the
human-preferred summary, and use that model as a reward function to fine-tune a
summarization policy using reinforcement learning. We apply our method to a
version of the TL;DR dataset of Reddit posts and find that our models
significantly outperform both human reference summaries and much larger models
fine-tuned with supervised learning alone. Our models also transfer to CNN/DM
news articles, producing summaries nearly as good as the human reference
without any news-specific fine-tuning. We conduct extensive analyses to
understand our human feedback dataset and fine-tuned models We establish that
our reward model generalizes to new datasets, and that optimizing our reward
model results in better summaries than optimizing ROUGE according to humans. We
hope the evidence from our paper motivates machine learning researchers to pay
closer attention to how their training loss affects the model behavior they
actually want.

positive:
Interactive Machine Learning is concerned with creating systems that operate
in environments alongside humans to achieve a task. A typical use is to extend
or amplify the capabilities of a human in cognitive or physical ways, requiring
the machine to adapt to the users' intentions and preferences. Often, this
takes the form of a human operator providing some type of feedback to the user,
which can be explicit feedback, implicit feedback, or a combination of both.
Explicit feedback, such as through a mouse click, carries a high cognitive
load. The focus of this study is to extend the current state of the art in
interactive machine learning by demonstrating that agents can learn a human
user's behavior and adapt to preferences with a reduced amount of explicit
human feedback in a mixed feedback setting. The learning agent perceives a
value of its own behavior from hand gestures given via a spatial interface.
This feedback mechanism is termed Spatial Interface Valuing. This method is
evaluated experimentally in a simulated environment for a grasping task using a
robotic arm with variable grip settings. Preliminary results indicate that
learning agents using spatial interface valuing can learn a value function
mapping spatial gestures to expected future rewards much more quickly as
compared to those same agents just receiving explicit feedback, demonstrating
that an agent perceiving feedback from a human user via a spatial interface can
serve as an effective complement to existing approaches.

positive:
Most research in Bayesian optimization (BO) has focused on \emph{direct
feedback} scenarios, where one has access to exact values of some
expensive-to-evaluate objective. This direction has been mainly driven by the
use of BO in machine learning hyper-parameter configuration problems. However,
in domains such as modelling human preferences, A/B tests, or recommender
systems, there is a need for methods that can replace direct feedback with
\emph{preferential feedback}, obtained via rankings or pairwise comparisons. In
this work, we present preferential batch Bayesian optimization (PBBO), a new
framework that allows finding the optimum of a latent function of interest,
given any type of parallel preferential feedback for a group of two or more
points. We do so by using a Gaussian process model with a likelihood specially
designed to enable parallel and efficient data collection mechanisms, which are
key in modern machine learning. We show how the acquisitions developed under
this framework generalize and augment previous approaches in Bayesian
optimization, expanding the use of these techniques to a wider range of
domains. An extensive simulation study shows the benefits of this approach,
both with simulated functions and four real data sets.

positive:
The history of learning for control has been an exciting back and forth
between two broad classes of algorithms: planning and reinforcement learning.
Planning algorithms effectively reason over long horizons, but assume access to
a local policy and distance metric over collision-free paths. Reinforcement
learning excels at learning policies and the relative values of states, but
fails to plan over long horizons. Despite the successes of each method in
various domains, tasks that require reasoning over long horizons with limited
feedback and high-dimensional observations remain exceedingly challenging for
both planning and reinforcement learning algorithms. Frustratingly, these sorts
of tasks are potentially the most useful, as they are simple to design (a human
only need to provide an example goal state) and avoid reward shaping, which can
bias the agent towards finding a sub-optimal solution. We introduce a general
control algorithm that combines the strengths of planning and reinforcement
learning to effectively solve these tasks. Our aim is to decompose the task of
reaching a distant goal state into a sequence of easier tasks, each of which
corresponds to reaching a subgoal. Planning algorithms can automatically find
these waypoints, but only if provided with suitable abstractions of the
environment -- namely, a graph consisting of nodes and edges. Our main insight
is that this graph can be constructed via reinforcement learning, where a
goal-conditioned value function provides edge weights, and nodes are taken to
be previously seen observations in a replay buffer. Using graph search over our
replay buffer, we can automatically generate this sequence of subgoals, even in
image-based environments. Our algorithm, search on the replay buffer (SoRB),
enables agents to solve sparse reward tasks over one hundred steps, and
generalizes substantially better than standard RL algorithms.

positive:
Interactive reinforcement learning provides a way for agents to learn to
solve tasks from evaluative feedback provided by a human user. Previous
research showed that humans give copious feedback early in training but very
sparsely thereafter. In this article, we investigate the potential of agent
learning from trainers' facial expressions via interpreting them as evaluative
feedback. To do so, we implemented TAMER which is a popular interactive
reinforcement learning method in a reinforcement-learning benchmark problem ---
Infinite Mario, and conducted the first large-scale study of TAMER involving
561 participants. With designed CNN-RNN model, our analysis shows that telling
trainers to use facial expressions and competition can improve the accuracies
for estimating positive and negative feedback using facial expressions. In
addition, our results with a simulation experiment show that learning solely
from predicted feedback based on facial expressions is possible and using
strong/effective prediction models or a regression method, facial responses
would significantly improve the performance of agents. Furthermore, our
experiment supports previous studies demonstrating the importance of
bi-directional feedback and competitive elements in the training interface.

positive:
In offline reinforcement learning (RL) agents are trained using a logged
dataset. It appears to be the most natural route to attack real-life
applications because in domains such as healthcare and robotics interactions
with the environment are either expensive or unethical. Training agents usually
requires reward functions, but unfortunately, rewards are seldom available in
practice and their engineering is challenging and laborious. To overcome this,
we investigate reward learning under the constraint of minimizing human reward
annotations. We consider two types of supervision: timestep annotations and
demonstrations. We propose semi-supervised learning algorithms that learn from
limited annotations and incorporate unlabelled data. In our experiments with a
simulated robotic arm, we greatly improve upon behavioural cloning and closely
approach the performance achieved with ground truth rewards. We further
investigate the relationship between the quality of the reward model and the
final policies. We notice, for example, that the reward models do not need to
be perfect to result in useful policies.

positive:
Mixed-initiative systems allow users to interactively provide feedback to
potentially improve system performance. Human feedback can correct model errors
and update model parameters to dynamically adapt to changing data.
Additionally, many users desire the ability to have a greater level of control
and fix perceived flaws in systems they rely on. However, how the ability to
provide feedback to autonomous systems influences user trust is a largely
unexplored area of research. Our research investigates how the act of providing
feedback can affect user understanding of an intelligent system and its
accuracy. We present a controlled experiment using a simulated object detection
system with image data to study the effects of interactive feedback collection
on user impressions. The results show that providing human-in-the-loop feedback
lowered both participants' trust in the system and their perception of system
accuracy, regardless of whether the system accuracy improved in response to
their feedback. These results highlight the importance of considering the
effects of allowing end-user feedback on user trust when designing intelligent
systems.

positive:
Robots will eventually be part of every household. It is thus critical to
enable algorithms to learn from and be guided by non-expert users. In this
paper, we bring a human in the loop, and enable a human teacher to give
feedback to a learning agent in the form of natural language. We argue that a
descriptive sentence can provide a much stronger learning signal than a numeric
reward in that it can easily point to where the mistakes are and how to correct
them. We focus on the problem of image captioning in which the quality of the
output can easily be judged by non-experts. We propose a hierarchical
phrase-based captioning model trained with policy gradients, and design a
feedback network that provides reward to the learner by conditioning on the
human-provided feedback. We show that by exploiting descriptive feedback our
model learns to perform better than when given independently written human
captions.

positive:
Human feedback is widely used to train agents in many domains. However,
previous works rarely consider the uncertainty when humans provide feedback,
especially in cases that the optimal actions are not obvious to the trainers.
For example, the reward of a sub-optimal action can be stochastic and sometimes
exceeds that of the optimal action, which is common in games or real-world.
Trainers are likely to provide positive feedback to sub-optimal actions,
negative feedback to the optimal actions and even do not provide feedback in
some confusing situations. Existing works, which utilize the Expectation
Maximization (EM) algorithm and treat the feedback model as hidden parameters,
do not consider uncertainties in the learning environment and human feedback.
To address this challenge, we introduce a novel feedback model that considers
the uncertainty of human feedback. However, this incurs intractable calculus in
the EM algorithm. To this end, we propose a novel approximate EM algorithm, in
which we approximate the expectation step with the Gradient Descent method.
Experimental results in both synthetic scenarios and two real-world scenarios
with human participants demonstrate the superior performance of our proposed
approach.

positive:
This article surveys reinforcement learning approaches in social robotics.
Reinforcement learning is a framework for decision-making problems in which an
agent interacts through trial-and-error with its environment to discover an
optimal behavior. Since interaction is a key component in both reinforcement
learning and social robotics, it can be a well-suited approach for real-world
interactions with physically embodied social robots. The scope of the paper is
focused particularly on studies that include social physical robots and
real-world human-robot interactions with users. We present a thorough analysis
of reinforcement learning approaches in social robotics. In addition to a
survey, we categorize existent reinforcement learning approaches based on the
used method and the design of the reward mechanisms. Moreover, since
communication capability is a prominent feature of social robots, we discuss
and group the papers based on the communication medium used for reward
formulation. Considering the importance of designing the reward function, we
also provide a categorization of the papers based on the nature of the reward.
This categorization includes three major themes: interactive reinforcement
learning, intrinsically motivated methods, and task performance-driven methods.
The benefits and challenges of reinforcement learning in social robotics,
evaluation methods of the papers regarding whether or not they use subjective
and algorithmic measures, a discussion in the view of real-world reinforcement
learning challenges and proposed solutions, the points that remain to be
explored, including the approaches that have thus far received less attention
is also given in the paper. Thus, this paper aims to become a starting point
for researchers interested in using and applying reinforcement learning methods
in this particular research field.

positive:
Existing methods for interactive image retrieval have demonstrated the merit
of integrating user feedback, improving retrieval results. However, most
current systems rely on restricted forms of user feedback, such as binary
relevance responses, or feedback based on a fixed set of relative attributes,
which limits their impact. In this paper, we introduce a new approach to
interactive image search that enables users to provide feedback via natural
language, allowing for more natural and effective interaction. We formulate the
task of dialog-based interactive image retrieval as a reinforcement learning
problem, and reward the dialog system for improving the rank of the target
image during each dialog turn. To mitigate the cumbersome and costly process of
collecting human-machine conversations as the dialog system learns, we train
our system with a user simulator, which is itself trained to describe the
differences between target and candidate images. The efficacy of our approach
is demonstrated in a footwear retrieval application. Experiments on both
simulated and real-world data show that 1) our proposed learning framework
achieves better accuracy than other supervised and reinforcement learning
baselines and 2) user feedback based on natural language rather than
pre-specified attributes leads to more effective retrieval results, and a more
natural and expressive communication interface.

positive:
We explore unconstrained natural language feedback as a learning signal for
artificial agents. Humans use rich and varied language to teach, yet most prior
work on interactive learning from language assumes a particular form of input
(e.g., commands). We propose a general framework which does not make this
assumption, using aspect-based sentiment analysis to decompose feedback into
sentiment about the features of a Markov decision process. We then perform an
analogue of inverse reinforcement learning, regressing the sentiment on the
features to infer the teacher's latent reward function. To evaluate our
approach, we first collect a corpus of teaching behavior in a cooperative task
where both teacher and learner are human. We implement three artificial
learners: sentiment-based "literal" and "pragmatic" models, and an inference
network trained end-to-end to predict latent rewards. We then repeat our
initial experiment and pair them with human teachers. All three successfully
learn from interactive human feedback. The sentiment models outperform the
inference network, with the "pragmatic" model approaching human performance.
Our work thus provides insight into the information structure of naturalistic
linguistic feedback as well as methods to leverage it for reinforcement
learning.

positive:
A key challenge in Imitation Learning (IL) is that optimal state actions
demonstrations are difficult for the teacher to provide. For example in
robotics, providing kinesthetic demonstrations on a robotic manipulator
requires the teacher to control multiple degrees of freedom at once. The
difficulty of requiring optimal state action demonstrations limits the space of
problems where the teacher can provide quality feedback. As an alternative to
state action demonstrations, the teacher can provide corrective feedback such
as their preferences or rewards. Prior work has created algorithms designed to
learn from specific types of noisy feedback, but across teachers and tasks
different forms of feedback may be required. Instead we propose that in order
to learn from a diversity of scenarios we need to learn from a variety of
feedback. To learn from a variety of feedback we make the following insight:
the teacher's cost function is latent and we can model a stream of feedback as
a stream of loss functions. We then use any online learning algorithm to
minimize the sum of these losses. With this insight we can learn from a
diversity of feedback that is weakly correlated with the teacher's true cost
function. We unify prior work into a general corrective feedback meta-algorithm
and show that regardless of feedback we can obtain the same regret bounds. We
demonstrate our approach by learning to perform a household navigation task on
a robotic racecar platform. Our results show that our approach can learn
quickly from a variety of noisy feedback.

positive:
Deep reinforcement learning has achieved many impressive results in recent
years. However, tasks with sparse rewards or long horizons continue to pose
significant challenges. To tackle these important problems, we propose a
general framework that first learns useful skills in a pre-training
environment, and then leverages the acquired skills for learning faster in
downstream tasks. Our approach brings together some of the strengths of
intrinsic motivation and hierarchical methods: the learning of useful skill is
guided by a single proxy reward, the design of which requires very minimal
domain knowledge about the downstream tasks. Then a high-level policy is
trained on top of these skills, providing a significant improvement of the
exploration and allowing to tackle sparse rewards in the downstream tasks. To
efficiently pre-train a large span of skills, we use Stochastic Neural Networks
combined with an information-theoretic regularizer. Our experiments show that
this combination is effective in learning a wide span of interpretable skills
in a sample-efficient way, and can significantly boost the learning performance
uniformly across a wide range of downstream tasks.

positive:
The last decade has seen a significant increase of interest in deep learning
research, with many public successes that have demonstrated its potential. As
such, these systems are now being incorporated into commercial products. With
this comes an additional challenge: how can we build AI systems that solve
tasks where there is not a crisp, well-defined specification? While multiple
solutions have been proposed, in this competition we focus on one in
particular: learning from human feedback. Rather than training AI systems using
a predefined reward function or using a labeled dataset with a predefined set
of categories, we instead train the AI system using a learning signal derived
from some form of human feedback, which can evolve over time as the
understanding of the task changes, or as the capabilities of the AI system
improve.
  The MineRL BASALT competition aims to spur forward research on this important
class of techniques. We design a suite of four tasks in Minecraft for which we
expect it will be hard to write down hardcoded reward functions. These tasks
are defined by a paragraph of natural language: for example, "create a
waterfall and take a scenic picture of it", with additional clarifying details.
Participants must train a separate agent for each task, using any method they
want. Agents are then evaluated by humans who have read the task description.
To help participants get started, we provide a dataset of human demonstrations
on each of the four tasks, as well as an imitation learning baseline that
leverages these demonstrations.
  Our hope is that this competition will improve our ability to build AI
systems that do what their designers intend them to do, even when the intent
cannot be easily formalized. Besides allowing AI to solve more tasks, this can
also enable more effective regulation of AI systems, as well as making progress
on the value alignment problem.

positive:
Human-computer interactive systems that rely on machine learning are becoming
paramount to the lives of millions of people who use digital assistants on a
daily basis. Yet, further advances are limited by the availability of data and
the cost of acquiring new samples. One way to address this problem is by
improving the sample efficiency of current approaches. As a solution path, we
present a model-based reinforcement learning algorithm for an interactive
dialogue task. We build on commonly used actor-critic methods, adding an
environment model and planner that augments a learning agent to learn the model
of the environment dynamics. Our results show that, on a simulation that mimics
the interactive task, our algorithm requires 70 times fewer samples, compared
to the baseline of commonly used model-free algorithm, and demonstrates 2~times
better performance asymptotically. Moreover, we introduce a novel contribution
of computing a soft planner policy and further updating a model-free policy
yielding a less computationally expensive model-free agent as good as the
model-based one. This model-based architecture serves as a foundation that can
be extended to other human-computer interactive tasks allowing further advances
in this direction.

positive:
Recent advances in reinforcement learning have inspired increasing interest
in learning user modeling adaptively through dynamic interactions, e.g., in
reinforcement learning based recommender systems. Reward function is crucial
for most of reinforcement learning applications as it can provide the guideline
about the optimization. However, current reinforcement-learning-based methods
rely on manually-defined reward functions, which cannot adapt to dynamic and
noisy environments. Besides, they generally use task-specific reward functions
that sacrifice generalization ability. We propose a generative inverse
reinforcement learning for user behavioral preference modelling, to address the
above issues. Instead of using predefined reward functions, our model can
automatically learn the rewards from user's actions based on discriminative
actor-critic network and Wasserstein GAN. Our model provides a general way of
characterizing and explaining underlying behavioral tendencies, and our
experiments show our method outperforms state-of-the-art methods in a variety
of scenarios, namely traffic signal control, online recommender systems, and
scanpath prediction.

positive:
Reinforcement learning is a powerful learning paradigm in which agents can
learn to maximize sparse and delayed reward signals. Although RL has had many
impressive successes in complex domains, learning can take hours, days, or even
years of training data. A major challenge of contemporary RL research is to
discover how to learn with less data. Previous work has shown that domain
information can be successfully used to shape the reward; by adding additional
reward information, the agent can learn with much less data. Furthermore, if
the reward is constructed from a potential function, the optimal policy is
guaranteed to be unaltered. While such potential-based reward shaping (PBRS)
holds promise, it is limited by the need for a well-defined potential function.
Ideally, we would like to be able to take arbitrary advice from a human or
other agent and improve performance without affecting the optimal policy. The
recently introduced dynamic potential based advice (DPBA) method tackles this
challenge by admitting arbitrary advice from a human or other agent and
improves performance without affecting the optimal policy. The main
contribution of this paper is to expose, theoretically and empirically, a flaw
in DPBA. Alternatively, to achieve the ideal goals, we present a simple method
called policy invariant explicit shaping (PIES) and show theoretically and
empirically that PIES succeeds where DPBA fails.

positive:
While recent progress in deep reinforcement learning has enabled robots to
learn complex behaviors, tasks with long horizons and sparse rewards remain an
ongoing challenge. In this work, we propose an effective reward shaping method
through predictive coding to tackle sparse reward problems. By learning
predictive representations offline and using these representations for reward
shaping, we gain access to reward signals that understand the structure and
dynamics of the environment. In particular, our method achieves better learning
by providing reward signals that 1) understand environment dynamics 2)
emphasize on features most useful for learning 3) resist noise in learned
representations through reward accumulation. We demonstrate the usefulness of
this approach in different domains ranging from robotic manipulation to
navigation, and we show that reward signals produced through predictive coding
are as effective for learning as hand-crafted rewards.

positive:
The ubiquitous nature of chatbots and their interaction with users generate
an enormous amount of data. Can we improve chatbots using this data? A
self-feeding chatbot improves itself by asking natural language feedback when a
user is dissatisfied with its response and uses this feedback as an additional
training sample. However, user feedback in most cases contains extraneous
sequences hindering their usefulness as a training sample. In this work, we
propose a generative adversarial model that converts noisy feedback into a
plausible natural response in a conversation. The generator's goal is to
convert the feedback into a response that answers the user's previous utterance
and to fool the discriminator which distinguishes feedback from natural
responses. We show that augmenting original training data with these modified
feedback responses improves the original chatbot performance from 69.94% to
75.96% in ranking correct responses on the Personachat dataset, a large
improvement given that the original model is already trained on 131k samples.

positive:
This paper addresses the problem of learning a task from demonstration. We
adopt the framework of inverse reinforcement learning, where tasks are
represented in the form of a reward function. Our contribution is a novel
active learning algorithm that enables the learning agent to query the expert
for more informative demonstrations, thus leading to more sample-efficient
learning. For this novel algorithm (Generalized Binary Search for Inverse
Reinforcement Learning, or GBS-IRL), we provide a theoretical bound on sample
complexity and illustrate its applicability on several different tasks. To our
knowledge, GBS-IRL is the first active IRL algorithm with provable sample
complexity bounds. We also discuss our method in light of other existing
methods in the literature and its general applicability in multi-class
classification problems. Finally, motivated by recent work on learning from
demonstration in robots, we also discuss how different forms of human feedback
can be integrated in a transparent manner in our learning framework.

positive:
We propose a general framework for interactively learning models, such as
(binary or non-binary) classifiers, orderings/rankings of items, or clusterings
of data points. Our framework is based on a generalization of Angluin's
equivalence query model and Littlestone's online learning model: in each
iteration, the algorithm proposes a model, and the user either accepts it or
reveals a specific mistake in the proposal. The feedback is correct only with
probability $p > 1/2$ (and adversarially incorrect with probability $1 - p$),
i.e., the algorithm must be able to learn in the presence of arbitrary noise.
The algorithm's goal is to learn the ground truth model using few iterations.
  Our general framework is based on a graph representation of the models and
user feedback. To be able to learn efficiently, it is sufficient that there be
a graph $G$ whose nodes are the models and (weighted) edges capture the user
feedback, with the property that if $s, s^*$ are the proposed and target
models, respectively, then any (correct) user feedback $s'$ must lie on a
shortest $s$-$s^*$ path in $G$. Under this one assumption, there is a natural
algorithm reminiscent of the Multiplicative Weights Update algorithm, which
will efficiently learn $s^*$ even in the presence of noise in the user's
feedback.
  From this general result, we rederive with barely any extra effort classic
results on learning of classifiers and a recent result on interactive
clustering; in addition, we easily obtain new interactive learning algorithms
for ordering/ranking.

positive:
The sense of touch, being the earliest sensory system to develop in a human
body [1], plays a critical part of our daily interaction with the environment.
In order to successfully complete a task, many manipulation interactions
require incorporating haptic feedback. However, manually designing a feedback
mechanism can be extremely challenging. In this work, we consider manipulation
tasks that need to incorporate tactile sensor feedback in order to modify a
provided nominal plan. To incorporate partial observation, we present a new
framework that models the task as a partially observable Markov decision
process (POMDP) and learns an appropriate representation of haptic feedback
which can serve as the state for a POMDP model. The model, that is parametrized
by deep recurrent neural networks, utilizes variational Bayes methods to
optimize the approximate posterior. Finally, we build on deep Q-learning to be
able to select the optimal action in each state without access to a simulator.
We test our model on a PR2 robot for multiple tasks of turning a knob until it
clicks.

positive:
An important application of interactive machine learning is extending or
amplifying the cognitive and physical capabilities of a human. To accomplish
this, machines need to learn about their human users' intentions and adapt to
their preferences. In most current research, a user has conveyed preferences to
a machine using explicit corrective or instructive feedback; explicit feedback
imposes a cognitive load on the user and is expensive in terms of human effort.
The primary objective of the current work is to demonstrate that a learning
agent can reduce the amount of explicit feedback required for adapting to the
user's preferences pertaining to a task by learning to perceive a value of its
behavior from the human user, particularly from the user's facial
expressions---we call this face valuing. We empirically evaluate face valuing
on a grip selection task. Our preliminary results suggest that an agent can
quickly adapt to a user's changing preferences with minimal explicit feedback
by learning a value function that maps facial features extracted from a camera
image to expected future reward. We believe that an agent learning to perceive
a value from the body language of its human user is complementary to existing
interactive machine learning approaches and will help in creating successful
human-machine interactive applications.

positive:
Recent progress in Reinforcement Learning (RL), fueled by its combination,
with Deep Learning has enabled impressive results in learning to interact with
complex virtual environments, yet real-world applications of RL are still
scarce. A key limitation is data efficiency, with current state-of-the-art
approaches requiring millions of training samples. A promising way to tackle
this problem is to augment RL with learning from human demonstrations. However,
human demonstration data is not yet readily available. This hinders progress in
this direction. The present work addresses this problem as follows. We (i)
collect and describe a large dataset of human Atari 2600 replays -- the largest
and most diverse such data set publicly released to date, (ii) illustrate an
example use of this dataset by analyzing the relation between demonstration
quality and imitation learning performance, and (iii) outline possible research
directions that are opened up by our work.

negative:
As robots enter human environments, they will be expected to accomplish a
tremendous range of tasks. It is not feasible for robot designers to
pre-program these behaviors or know them in advance, so one way to address this
is through end-user programming, such as via learning from demonstration (LfD).
While significant work has been done on the mechanics of enabling robot
learning from human teachers, one unexplored aspect is enabling mutual feedback
between both the human teacher and robot during the learning process, i.e.,
implicit learning. In this paper, we explore one aspect of this mutual
understanding, grounding sequences, where both a human and robot provide
non-verbal feedback to signify their mutual understanding during interaction.
We conducted a study where people taught an autonomous humanoid robot a dance,
and performed gesture analysis to measure people's responses to the robot
during correct and incorrect demonstrations.

negative:
Many real-world systems problems require reasoning about the long term
consequences of actions taken to configure and manage the system. These
problems with delayed and often sequentially aggregated reward, are often
inherently reinforcement learning problems and present the opportunity to
leverage the recent substantial advances in deep reinforcement learning.
However, in some cases, it is not clear why deep reinforcement learning is a
good fit for the problem. Sometimes, it does not perform better than the
state-of-the-art solutions. And in other cases, random search or greedy
algorithms could outperform deep reinforcement learning. In this paper, we
review, discuss, and evaluate the recent trends of using deep reinforcement
learning in system optimization. We propose a set of essential metrics to guide
future works in evaluating the efficacy of using deep reinforcement learning in
system optimization. Our evaluation includes challenges, the types of problems,
their formulation in the deep reinforcement learning setting, embedding, the
model used, efficiency, and robustness. We conclude with a discussion on open
challenges and potential directions for pushing further the integration of
reinforcement learning in system optimization.

negative:
Text-based games are a natural challenge domain for deep reinforcement
learning algorithms. Their state and action spaces are combinatorially large,
their reward function is sparse, and they are partially observable: the agent
is informed of the consequences of its actions through textual feedback. In
this paper we emphasize this latter point and consider the design of a deep
reinforcement learning agent that can play from feedback alone. Our design
recognizes and takes advantage of the structural characteristics of text-based
games. We first propose a contextualisation mechanism, based on accumulated
reward, which simplifies the learning problem and mitigates partial
observability. We then study different methods that rely on the notion that
most actions are ineffectual in any given situation, following Zahavy et al.'s
idea of an admissible action. We evaluate these techniques in a series of
text-based games of increasing difficulty based on the TextWorld framework, as
well as the iconic game Zork. Empirically, we find that these techniques
improve the performance of a baseline deep reinforcement learning agent applied
to text-based games.

negative:
We propose a novel reinforcement learning based framework PoBRL for solving
multi-document summarization. PoBRL jointly optimizes over the following three
objectives necessary for a high-quality summary: importance, relevance, and
length. Our strategy decouples this multi-objective optimization into different
subproblems that can be solved individually by reinforcement learning.
Utilizing PoBRL, we then blend each learned policies together to produce a
summary that is a concise and complete representation of the original input.
Our empirical analysis shows state-of-the-art performance on several
multi-document datasets. Human evaluation also shows that our method produces
high-quality output.

negative:
Imitation learning is an effective approach for autonomous systems to acquire
control policies when an explicit reward function is unavailable, using
supervision provided as demonstrations from an expert, typically a human
operator. However, standard imitation learning methods assume that the agent
receives examples of observation-action tuples that could be provided, for
instance, to a supervised learning algorithm. This stands in contrast to how
humans and animals imitate: we observe another person performing some behavior
and then figure out which actions will realize that behavior, compensating for
changes in viewpoint, surroundings, object positions and types, and other
factors. We term this kind of imitation learning "imitation-from-observation,"
and propose an imitation learning method based on video prediction with context
translation and deep reinforcement learning. This lifts the assumption in
imitation learning that the demonstration should consist of observations in the
same environment configuration, and enables a variety of interesting
applications, including learning robotic skills that involve tool use simply by
observing videos of human tool use. Our experimental results show the
effectiveness of our approach in learning a wide range of real-world robotic
tasks modeled after common household chores from videos of a human
demonstrator, including sweeping, ladling almonds, pushing objects as well as a
number of tasks in simulation.

negative:
In the artificial intelligence field, learning often corresponds to changing
the parameters of a parameterized function. A learning rule is an algorithm or
mathematical expression that specifies precisely how the parameters should be
changed. When creating an artificial intelligence system, we must make two
decisions: what representation should be used (i.e., what parameterized
function should be used) and what learning rule should be used to search
through the resulting set of representable functions. Using most learning
rules, these two decisions are coupled in a subtle (and often unintentional)
way. That is, using the same learning rule with two different representations
that can represent the same sets of functions can result in two different
outcomes. After arguing that this coupling is undesirable, particularly when
using artificial neural networks, we present a method for partially decoupling
these two decisions for a broad class of learning rules that span unsupervised
learning, reinforcement learning, and supervised learning.

negative:
Despite widespread interests in reinforcement-learning for task-oriented
dialogue systems, several obstacles can frustrate research and development
progress. First, reinforcement learners typically require interaction with the
environment, so conventional dialogue corpora cannot be used directly. Second,
each task presents specific challenges, requiring separate corpus of
task-specific annotated data. Third, collecting and annotating human-machine or
human-human conversations for task-oriented dialogues requires extensive domain
knowledge. Because building an appropriate dataset can be both financially
costly and time-consuming, one popular approach is to build a user simulator
based upon a corpus of example dialogues. Then, one can train reinforcement
learning agents in an online fashion as they interact with the simulator.
Dialogue agents trained on these simulators can serve as an effective starting
point. Once agents master the simulator, they may be deployed in a real
environment to interact with humans, and continue to be trained online. To ease
empirical algorithmic comparisons in dialogues, this paper introduces a new,
publicly available simulation framework, where our simulator, designed for the
movie-booking domain, leverages both rules and collected data. The simulator
supports two tasks: movie ticket booking and movie seeking. Finally, we
demonstrate several agents and detail the procedure to add and test your own
agent in the proposed framework.

negative:
Learning in sparse reward settings remains a challenge in Reinforcement
Learning, which is often addressed by using intrinsic rewards. One promising
strategy is inspired by human curiosity, requiring the agent to learn to
predict the future. In this paper a curiosity-driven agent is extended to use
these predictions directly for training. To achieve this, the agent predicts
the value function of the next state at any point in time. Subsequently, the
consistency of this prediction with the current value function is measured,
which is then used as a regularization term in the loss function of the
algorithm. Experiments were made on grid-world environments as well as on a 3D
navigation task, both with sparse rewards. In the first case the extended agent
is able to learn significantly faster than the baselines.

negative:
While deep reinforcement learning techniques have led to agents that are
successfully able to learn to perform a number of tasks that had been
previously unlearnable, these techniques are still susceptible to the
longstanding problem of reward sparsity. This is especially true for tasks such
as training an agent to play StarCraft II, a real-time strategy game where
reward is only given at the end of a game which is usually very long. While
this problem can be addressed through reward shaping, such approaches typically
require a human expert with specialized knowledge. Inspired by the vision of
enabling reward shaping through the more-accessible paradigm of
natural-language narration, we develop a technique that can provide the
benefits of reward shaping using natural language commands. Our
narration-guided RL agent projects sequences of natural-language commands into
the same high-dimensional representation space as corresponding goal states. We
show that we can get improved performance with our method compared to
traditional reward-shaping approaches. Additionally, we demonstrate the ability
of our method to generalize to unseen natural-language commands.

negative:
What is the role of real-time control and learning in the formation of social
conventions? To answer this question, we propose a computational model that
matches human behavioral data in a social decision-making game that was
analyzed both in discrete-time and continuous-time setups. Furthermore, unlike
previous approaches, our model takes into account the role of sensorimotor
control loops in embodied decision-making scenarios. For this purpose, we
introduce the Control-based Reinforcement Learning (CRL) model. CRL is grounded
in the Distributed Adaptive Control (DAC) theory of mind and brain, where
low-level sensorimotor control is modulated through perceptual and behavioral
learning in a layered structure. CRL follows these principles by implementing a
feedback control loop handling the agent's reactive behaviors (pre-wired
reflexes), along with an adaptive layer that uses reinforcement learning to
maximize long-term reward. We test our model in a multi-agent game-theoretic
task in which coordination must be achieved to find an optimal solution. We
show that CRL is able to reach human-level performance on standard
game-theoretic metrics such as efficiency in acquiring rewards and fairness in
reward distribution.

negative:
The recent history of machine learning research has taught us that machine
learning methods can be most effective when they are provided with very large,
high-capacity models, and trained on very large and diverse datasets. This has
spurred the community to search for ways to remove any bottlenecks to scale.
Often the foremost among such bottlenecks is the need for human effort,
including the effort of curating and labeling datasets. As a result,
considerable attention in recent years has been devoted to utilizing unlabeled
data, which can be collected in vast quantities. However, some of the most
widely used methods for training on such unlabeled data themselves require
human-designed objective functions that must correlate in some meaningful way
to downstream tasks. I will argue that a general, principled, and powerful
framework for utilizing unlabeled data can be derived from reinforcement
learning, using general purpose unsupervised or self-supervised reinforcement
learning objectives in concert with offline reinforcement learning methods that
can leverage large datasets. I will discuss how such a procedure is more
closely aligned with potential downstream tasks, and how it could build on
existing techniques that have been developed in recent years.

negative:
This paper presents a comprehensive literature review on applications of deep
reinforcement learning in communications and networking. Modern networks, e.g.,
Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become
more decentralized and autonomous. In such networks, network entities need to
make decisions locally to maximize the network performance under uncertainty of
network environment. Reinforcement learning has been efficiently used to enable
the network entities to obtain the optimal policy including, e.g., decisions or
actions, given their states when the state and action spaces are small.
However, in complex and large-scale networks, the state and action spaces are
usually large, and the reinforcement learning may not be able to find the
optimal policy in reasonable time. Therefore, deep reinforcement learning, a
combination of reinforcement learning with deep learning, has been developed to
overcome the shortcomings. In this survey, we first give a tutorial of deep
reinforcement learning from fundamental concepts to advanced models. Then, we
review deep reinforcement learning approaches proposed to address emerging
issues in communications and networking. The issues include dynamic network
access, data rate control, wireless caching, data offloading, network security,
and connectivity preservation which are all important to next generation
networks such as 5G and beyond. Furthermore, we present applications of deep
reinforcement learning for traffic routing, resource sharing, and data
collection. Finally, we highlight important challenges, open issues, and future
research directions of applying deep reinforcement learning.

negative:
The availability of large-scale datasets has driven the development of neural
models that create summaries from single documents, for generic purposes. When
using a summarization system, users often have specific intents with various
language realizations, which, depending on the information need, can range from
a single keyword to a long narrative composed of multiple questions. Existing
summarization systems, however, often either fail to support or act robustly on
this query focused summarization task. We introduce LaQSum, the first unified
text summarization system that learns Latent Queries from documents for
abstractive summarization with any existing query forms. Under a deep
generative framework, our system jointly optimizes a latent query model and a
conditional language model, allowing users to plug-and-play queries of any type
at test time. Despite learning from only generic summarization data and
requiring no further optimization for downstream summarization tasks, our
system robustly outperforms strong comparison systems across summarization
benchmarks with different query types, document settings, and target domains.

negative:
Reinforcement learning is a learning paradigm for solving sequential
decision-making problems. Recent years have witnessed remarkable progress in
reinforcement learning upon the fast development of deep neural networks. Along
with the promising prospects of reinforcement learning in numerous domains such
as robotics and game-playing, transfer learning has arisen to tackle various
challenges faced by reinforcement learning, by transferring knowledge from
external expertise to facilitate the efficiency and effectiveness of the
learning process. In this survey, we systematically investigate the recent
progress of transfer learning approaches in the context of deep reinforcement
learning. Specifically, we provide a framework for categorizing the
state-of-the-art transfer learning approaches, under which we analyze their
goals, methodologies, compatible reinforcement learning backbones, and
practical applications. We also draw connections between transfer learning and
other relevant topics from the reinforcement learning perspective and explore
their potential challenges that await future research progress.

negative:
In reinforcement learning, pre-trained low-level skills have the potential to
greatly facilitate exploration. However, prior knowledge of the downstream task
is required to strike the right balance between generality (fine-grained
control) and specificity (faster learning) in skill design. In previous work on
continuous control, the sensitivity of methods to this trade-off has not been
addressed explicitly, as locomotion provides a suitable prior for navigation
tasks, which have been of foremost interest. In this work, we analyze this
trade-off for low-level policy pre-training with a new benchmark suite of
diverse, sparse-reward tasks for bipedal robots. We alleviate the need for
prior knowledge by proposing a hierarchical skill learning framework that
acquires skills of varying complexity in an unsupervised manner. For
utilization on downstream tasks, we present a three-layered hierarchical
learning algorithm to automatically trade off between general and specific
skills as required by the respective task. In our experiments, we show that our
approach performs this trade-off effectively and achieves better results than
current state-of-the-art methods for end- to-end hierarchical reinforcement
learning and unsupervised skill discovery. Code and videos are available at
https://facebookresearch.github.io/hsd3 .

negative:
Reinforcement learning has seen great advancements in the past five years.
The successful introduction of deep learning in place of more traditional
methods allowed reinforcement learning to scale to very complex domains
achieving super-human performance in environments like the game of Go or
numerous video games. Despite great successes in multiple domains, these new
methods suffer from their own issues that make them often inapplicable to the
real world problems. Extreme lack of data efficiency, together with huge
variance and difficulty in enforcing safety constraints, is one of the three
most prominent issues in the field. Usually, millions of data points sampled
from the environment are necessary for these algorithms to converge to
acceptable policies.
  This thesis proposes novel Generative Adversarial Imaginative Reinforcement
Learning algorithm. It takes advantage of the recent introduction of highly
effective generative adversarial models, and Markov property that underpins
reinforcement learning setting, to model dynamics of the real environment
within the internal imagination module. Rollouts from the imagination are then
used to artificially simulate the real environment in a standard reinforcement
learning process to avoid, often expensive and dangerous, trial and error in
the real environment. Experimental results show that the proposed algorithm
more economically utilises experience from the real environment than the
current state-of-the-art Rainbow DQN algorithm, and thus makes an important
step towards sample efficient deep reinforcement learning.

negative:
We propose Object-oriented Neural Programming (OONP), a framework for
semantically parsing documents in specific domains. Basically, OONP reads a
document and parses it into a predesigned object-oriented data structure
(referred to as ontology in this paper) that reflects the domain-specific
semantics of the document. An OONP parser models semantic parsing as a decision
process: a neural net-based Reader sequentially goes through the document, and
during the process it builds and updates an intermediate ontology to summarize
its partial understanding of the text it covers. OONP supports a rich family of
operations (both symbolic and differentiable) for composing the ontology, and a
big variety of forms (both symbolic and differentiable) for representing the
state and the document. An OONP parser can be trained with supervision of
different forms and strength, including supervised learning (SL) ,
reinforcement learning (RL) and hybrid of the two. Our experiments on both
synthetic and real-world document parsing tasks have shown that OONP can learn
to handle fairly complicated ontology with training data of modest sizes.

negative:
Deep reinforcement learning is revolutionizing the artificial intelligence
field. Currently, it serves as a good starting point for constructing
intelligent autonomous systems which offer a better knowledge of the visual
world. It is possible to scale deep reinforcement learning with the use of deep
learning and do amazing tasks such as use of pixels in playing video games. In
this paper, key concepts of deep reinforcement learning including reward
function, differences between reinforcement learning and supervised learning
and models for implementation of reinforcement are discussed. Key challenges
related to the implementation of reinforcement learning in conversational AI
domain are identified as well as discussed in detail. Various conversational
models which are based on deep reinforcement learning (as well as deep
learning) are also discussed. In summary, this paper discusses key aspects of
deep reinforcement learning which are crucial for designing an efficient
conversational AI.

negative:
Humans have mental models that allow them to plan, experiment, and reason in
the physical world. How should an intelligent agent go about learning such
models? In this paper, we will study if models of the world learned in an
open-ended physics environment, without any specific tasks, can be reused for
downstream physics reasoning tasks. To this end, we build a benchmark
Open-ended Physics ENvironment (OPEn) and also design several tasks to test
learning representations in this environment explicitly. This setting reflects
the conditions in which real agents (i.e. rolling robots) find themselves,
where they may be placed in a new kind of environment and must adapt without
any teacher to tell them how this environment works. This setting is
challenging because it requires solving an exploration problem in addition to a
model building and representation learning problem. We test several existing
RL-based exploration methods on this benchmark and find that an agent using
unsupervised contrastive learning for representation learning, and
impact-driven learning for exploration, achieved the best results. However, all
models still fall short in sample efficiency when transferring to the
downstream tasks. We expect that OPEn will encourage the development of novel
rolling robot agents that can build reusable mental models of the world that
facilitate many tasks.

negative:
Model-free reinforcement learning has been successfully applied to a range of
challenging problems, and has recently been extended to handle large neural
network policies and value functions. However, the sample complexity of
model-free algorithms, particularly when using high-dimensional function
approximators, tends to limit their applicability to physical systems. In this
paper, we explore algorithms and representations to reduce the sample
complexity of deep reinforcement learning for continuous control tasks. We
propose two complementary techniques for improving the efficiency of such
algorithms. First, we derive a continuous variant of the Q-learning algorithm,
which we call normalized adantage functions (NAF), as an alternative to the
more commonly used policy gradient and actor-critic methods. NAF representation
allows us to apply Q-learning with experience replay to continuous tasks, and
substantially improves performance on a set of simulated robotic control tasks.
To further improve the efficiency of our approach, we explore the use of
learned models for accelerating model-free reinforcement learning. We show that
iteratively refitted local linear models are especially effective for this, and
demonstrate substantially faster learning on domains where such models are
applicable.

negative:
We present a reinforcement learning based framework for human-centered
collaborative systems. The framework is proactive and balances the benefits of
timely actions with the risk of taking improper actions by minimizing the total
time spent to complete the task. The framework is learned end-to-end in an
unsupervised fashion addressing the perception uncertainties and decision
making in an integrated manner. The framework is shown to provide more fluent
coordination between human and robot partners on an example task of packaging
compared to alternatives for which perception and decision-making systems are
learned independently, using supervised learning. The foremost benefit of the
proposed approach is that it allows for fast adaptation to new human partners
and tasks since tedious annotation of motion data is avoided and the learning
is performed on-line.

negative:
In recent years deep neural networks have been successfully applied to the
domains of reinforcement learning
\cite{bengio2009learning,krizhevsky2012imagenet,hinton2006reducing}. Deep
reinforcement learning \cite{mnih2015human} is reported to have the advantage
of learning effective policies directly from high-dimensional sensory inputs
over traditional agents. However, within the scope of the literature, there is
no fundamental change or improvement on the existing training framework. Here
we propose a novel training framework that is conceptually comprehensible and
potentially easy to be generalized to all feasible algorithms for reinforcement
learning. We employ Monte-carlo sampling to achieve raw data inputs, and train
them in batch to achieve Markov decision process sequences and synchronously
update the network parameters instead of experience replay. This training
framework proves to optimize the unbiased approximation of loss function whose
estimation exactly matches the real probability distribution data inputs
follow, and thus have overwhelming advantages of sample efficiency and
convergence rate over existing deep reinforcement learning after evaluating it
on both discrete action spaces and continuous control problems. Besides, we
propose several algorithms embedded with our new framework to deal with typical
discrete and continuous scenarios. These algorithms prove to be far more
efficient than their original versions under the framework of deep
reinforcement learning, and provide examples for existing and future algorithms
to generalize to our new framework.

negative:
Intelligent assistants that follow commands or answer simple questions, such
as Siri and Google search, are among the most economically important
applications of AI. Future conversational AI assistants promise even greater
capabilities and a better user experience through a deeper understanding of the
domain, the user, or the user's purposes. But what domain and what methods are
best suited to researching and realizing this promise? In this article we argue
for the domain of voice document editing and for the methods of model-based
reinforcement learning. The primary advantages of voice document editing are
that the domain is tightly scoped and that it provides something for the
conversation to be about (the document) that is delimited and fully accessible
to the intelligent assistant. The advantages of reinforcement learning in
general are that its methods are designed to learn from interaction without
explicit instruction and that it formalizes the purposes of the assistant.
Model-based reinforcement learning is needed in order to genuinely understand
the domain of discourse and thereby work efficiently with the user to achieve
their goals. Together, voice document editing and model-based reinforcement
learning comprise a promising research direction for achieving conversational
AI.

negative:
Measuring user satisfaction level is a challenging task, and a critical
component in developing large-scale conversational agent systems serving the
needs of real users. An widely used approach to tackle this is to collect human
annotation data and use them for evaluation or modeling. Human annotation based
approaches are easier to control, but hard to scale. A novel alternative
approach is to collect user's direct feedback via a feedback elicitation system
embedded to the conversational agent system, and use the collected user
feedback to train a machine-learned model for generalization. User feedback is
the best proxy for user satisfaction, but is not available for some ineligible
intents and certain situations. Thus, these two types of approaches are
complementary to each other. In this work, we tackle the user satisfaction
assessment problem with a hybrid approach that fuses explicit user feedback,
user satisfaction predictions inferred by two machine-learned models, one
trained on user feedback data and the other human annotation data. The hybrid
approach is based on a waterfall policy, and the experimental results with
Amazon Alexa's large-scale datasets show significant improvements in inferring
user satisfaction. A detailed hybrid architecture, an in-depth analysis on user
feedback data, and an algorithm that generates data sets to properly simulate
the live traffic are presented in this paper.

negative:
Reinforcement learning provides a powerful and flexible framework for
automated acquisition of robotic motion skills. However, applying reinforcement
learning requires a sufficiently detailed representation of the state,
including the configuration of task-relevant objects. We present an approach
that automates state-space construction by learning a state representation
directly from camera images. Our method uses a deep spatial autoencoder to
acquire a set of feature points that describe the environment for the current
task, such as the positions of objects, and then learns a motion skill with
these feature points using an efficient reinforcement learning method based on
local linear models. The resulting controller reacts continuously to the
learned feature points, allowing the robot to dynamically manipulate objects in
the world with closed-loop control. We demonstrate our method with a PR2 robot
on tasks that include pushing a free-standing toy block, picking up a bag of
rice using a spatula, and hanging a loop of rope on a hook at various
positions. In each task, our method automatically learns to track task-relevant
objects and manipulate their configuration with the robot's arm.

negative:
We consider the problem of imitation learning from a finite set of expert
trajectories, without access to reinforcement signals. The classical approach
of extracting the expert's reward function via inverse reinforcement learning,
followed by reinforcement learning is indirect and may be computationally
expensive. Recent generative adversarial methods based on matching the policy
distribution between the expert and the agent could be unstable during
training. We propose a new framework for imitation learning by estimating the
support of the expert policy to compute a fixed reward function, which allows
us to re-frame imitation learning within the standard reinforcement learning
setting. We demonstrate the efficacy of our reward function on both discrete
and continuous domains, achieving comparable or better performance than the
state of the art under different reinforcement learning algorithms.

negative:
The study of unsupervised learning can be generally divided into two
categories: imitation learning and reinforcement learning. In imitation
learning the machine learns by mimicking the behavior of an expert system
whereas in reinforcement learning the machine learns via direct environment
feedback. Traditional deep reinforcement learning takes a significant time
before the machine starts to converge to an optimal policy. This paper proposes
Augmented Q-Imitation-Learning, a method by which deep reinforcement learning
convergence can be accelerated by applying Q-imitation-learning as the initial
training process in traditional Deep Q-learning.

negative:
Consider learning a policy from example expert behavior, without interaction
with the expert or access to reinforcement signal. One approach is to recover
the expert's cost function with inverse reinforcement learning, then extract a
policy from that cost function with reinforcement learning. This approach is
indirect and can be slow. We propose a new general framework for directly
extracting a policy from data, as if it were obtained by reinforcement learning
following inverse reinforcement learning. We show that a certain instantiation
of our framework draws an analogy between imitation learning and generative
adversarial networks, from which we derive a model-free imitation learning
algorithm that obtains significant performance gains over existing model-free
methods in imitating complex behaviors in large, high-dimensional environments.

negative:
Artificial Intelligence has been developed for decades with the achievement
of great progress. Recently, deep learning shows its ability to solve many real
world problems, e.g. image classification and detection, natural language
processing, playing GO. Theoretically speaking, an artificial neural network
can fit any function and reinforcement learning can learn from any delayed
reward. But in solving real world tasks, we still need to spend a lot of effort
to adjust algorithms to fit task unique features. This paper proposes that the
reason of this phenomenon is the sparse feedback feature of the nature, and a
single algorithm, no matter how we improve it, can only solve dense feedback
tasks or specific sparse feedback tasks. This paper first analyses how sparse
feedback affects algorithm perfomance, and then proposes a pattern that
explains how to accumulate knowledge to solve sparse feedback problems.

negative:
Deep reinforcement learning augments the reinforcement learning framework and
utilizes the powerful representation of deep neural networks. Recent works have
demonstrated the remarkable successes of deep reinforcement learning in various
domains including finance, medicine, healthcare, video games, robotics, and
computer vision. In this work, we provide a detailed review of recent and
state-of-the-art research advances of deep reinforcement learning in computer
vision. We start with comprehending the theories of deep learning,
reinforcement learning, and deep reinforcement learning. We then propose a
categorization of deep reinforcement learning methodologies and discuss their
advantages and limitations. In particular, we divide deep reinforcement
learning into seven main categories according to their applications in computer
vision, i.e. (i)landmark localization (ii) object detection; (iii) object
tracking; (iv) registration on both 2D image and 3D image volumetric data (v)
image segmentation; (vi) videos analysis; and (vii) other applications. Each of
these categories is further analyzed with reinforcement learning techniques,
network design, and performance. Moreover, we provide a comprehensive analysis
of the existing publicly available datasets and examine source code
availability. Finally, we present some open issues and discuss future research
directions on deep reinforcement learning in computer vision

negative:
Recent approaches in literature have exploited the multi-modal information in
documents (text, layout, image) to serve specific downstream document tasks.
However, they are limited by their - (i) inability to learn cross-modal
representations across text, layout and image dimensions for documents and (ii)
inability to process multi-page documents. Pre-training techniques have been
shown in Natural Language Processing (NLP) domain to learn generic textual
representations from large unlabelled datasets, applicable to various
downstream NLP tasks. In this paper, we propose a multi-task learning-based
framework that utilizes a combination of self-supervised and supervised
pre-training tasks to learn a generic document representation applicable to
various downstream document tasks. Specifically, we introduce Document Topic
Modelling and Document Shuffle Prediction as novel pre-training tasks to learn
rich image representations along with the text and layout representations for
documents. We utilize the Longformer network architecture as the backbone to
encode the multi-modal information from multi-page documents in an end-to-end
fashion. We showcase the applicability of our pre-training framework on a
variety of different real-world document tasks such as document classification,
document information extraction, and document retrieval. We evaluate our
framework on different standard document datasets and conduct exhaustive
experiments to compare performance against various ablations of our framework
and state-of-the-art baselines.

negative:
Reinforcement learning has recently experienced increased prominence in the
machine learning community. There are many approaches to solving reinforcement
learning problems with new techniques developed constantly. When solving
problems using reinforcement learning, there are various difficult challenges
to overcome. To ensure progress in the field, benchmarks are important for
testing new algorithms and comparing with other approaches. The reproducibility
of results for fair comparison is therefore vital in ensuring that improvements
are accurately judged. This paper provides an overview of different
contributions to reinforcement learning benchmarking and discusses how they can
assist researchers to address the challenges facing reinforcement learning. The
contributions discussed are the most used and recent in the literature. The
paper discusses the contributions in terms of implementation, tasks and
provided algorithm implementations with benchmarks. The survey aims to bring
attention to the wide range of reinforcement learning benchmarking tasks
available and to encourage research to take place in a standardised manner.
Additionally, this survey acts as an overview for researchers not familiar with
the different tasks that can be used to develop and test new reinforcement
learning algorithms.

negative:
Reinforcement learning is one of the core components in designing an
artificial intelligent system emphasizing real-time response. Reinforcement
learning influences the system to take actions within an arbitrary environment
either having previous knowledge about the environment model or not. In this
paper, we present a comprehensive study on Reinforcement Learning focusing on
various dimensions including challenges, the recent development of different
state-of-the-art techniques, and future directions. The fundamental objective
of this paper is to provide a framework for the presentation of available
methods of reinforcement learning that is informative enough and simple to
follow for the new researchers and academics in this domain considering the
latest concerns. First, we illustrated the core techniques of reinforcement
learning in an easily understandable and comparable way. Finally, we analyzed
and depicted the recent developments in reinforcement learning approaches. My
analysis pointed out that most of the models focused on tuning policy values
rather than tuning other things in a particular state of reasoning.

negative:
One of the major prerequisites for any deep learning approach is the
availability of large-scale training data. When dealing with scanned document
images in real world scenarios, the principal information of its content is
stored in the layout itself. In this work, we have proposed an automated deep
generative model using Graph Neural Networks (GNNs) to generate synthetic data
with highly variable and plausible document layouts that can be used to train
document interpretation systems, in this case, specially in digital mailroom
applications. It is also the first graph-based approach for document layout
generation task experimented on administrative document images, in this case,
invoices.

negative:
Cross-lingual document representations enable language understanding in
multilingual contexts and allow transfer learning from high-resource to
low-resource languages at the document level. Recently large pre-trained
language models such as BERT, XLM and XLM-RoBERTa have achieved great success
when fine-tuned on sentence-level downstream tasks. It is tempting to apply
these cross-lingual models to document representation learning. However, there
are two challenges: (1) these models impose high costs on long document
processing and thus many of them have strict length limit; (2) model
fine-tuning requires extra data and computational resources, which is not
practical in resource-limited settings. In this work, we address these
challenges by proposing unsupervised Language-Agnostic Weighted Document
Representations (LAWDR). We study the geometry of pre-trained sentence
embeddings and leverage it to derive document representations without
fine-tuning. Evaluated on cross-lingual document alignment, LAWDR demonstrates
comparable performance to state-of-the-art models on benchmark datasets.

negative:
How to leverage cross-document interactions to improve ranking performance is
an important topic in information retrieval (IR) research. However, this topic
has not been well-studied in the learning-to-rank setting and most of the
existing work still treats each document independently while scoring. The
recent development of deep learning shows strength in modeling complex
relationships across sequences and sets. It thus motivates us to study how to
leverage cross-document interactions for learning-to-rank in the deep learning
framework. In this paper, we formally define the permutation-equivariance
requirement for a scoring function that captures cross-document interactions.
We then propose a self-attention based document interaction network and show
that it satisfies the permutation-equivariant requirement, and can generate
scores for document sets of varying sizes. Our proposed methods can
automatically learn to capture document interactions without any auxiliary
information, and can scale across large document sets. We conduct experiments
on three ranking datasets: the benchmark Web30k, a Gmail search, and a Google
Drive Quick Access dataset. Experimental results show that our proposed methods
are both more effective and efficient than baselines.

negative:
In this paper, we focus on the problem of unsupervised image-sentence
matching. Existing research explores to utilize document-level structural
information to sample positive and negative instances for model training.
Although the approach achieves positive results, it introduces a sampling bias
and fails to distinguish instances with high semantic similarity. To alleviate
the bias, we propose a new sampling strategy to select additional
intra-document image-sentence pairs as positive or negative samples.
Furthermore, to recognize the complex pattern in intra-document samples, we
propose a Transformer based model to capture fine-grained features and
implicitly construct a graph for each document, where concepts in a document
are introduced to bridge the representation learning of images and sentences in
the context of a document. Experimental results show the effectiveness of our
approach to alleviate the bias and learn well-aligned multimodal
representations.

negative:
Self-supervised metric learning has been a successful approach for learning a
distance from an unlabeled dataset. The resulting distance is broadly useful
for improving various distance-based downstream tasks, even when no information
from downstream tasks is utilized in the metric learning stage. To gain
insights into this approach, we develop a statistical framework to
theoretically study how self-supervised metric learning can benefit downstream
tasks in the context of multi-view data. Under this framework, we show that the
target distance of metric learning satisfies several desired properties for the
downstream tasks. On the other hand, our investigation suggests the target
distance can be further improved by moderating each direction's weights. In
addition, our analysis precisely characterizes the improvement by
self-supervised metric learning on four commonly used downstream tasks: sample
identification, two-sample testing, $k$-means clustering, and $k$-nearest
neighbor classification. When the distance is estimated from an unlabeled
dataset, we establish the upper bound on distance estimation's accuracy and the
number of samples sufficient for downstream task improvement. Finally,
numerical experiments are presented to support the theoretical results in the
paper.

negative:
Model selection is treated as a standard performance boosting step in many
machine learning applications. Once all other properties of a learning problem
are fixed, the model is selected by grid search on a held-out validation set.
This is strictly inapplicable to active learning. Within the standardized
workflow, the acquisition function is chosen among available heuristics a
priori, and its success is observed only after the labeling budget is already
exhausted. More importantly, none of the earlier studies report a unique
consistently successful acquisition heuristic to the extent to stand out as the
unique best choice. We present a method to break this vicious circle by
defining the acquisition function as a learning predictor and training it by
reinforcement feedback collected from each labeling round. As active learning
is a scarce data regime, we bootstrap from a well-known heuristic that filters
the bulk of data points on which all heuristics would agree, and learn a policy
to warp the top portion of this ranking in the most beneficial way for the
character of a specific data distribution. Our system consists of a Bayesian
neural net, the predictor, a bootstrap acquisition function, a probabilistic
state definition, and another Bayesian policy network that can effectively
incorporate this input distribution. We observe on three benchmark data sets
that our method always manages to either invent a new superior acquisition
function or to adapt itself to the a priori unknown best performing heuristic
for each specific data set.

negative:
In this work, a region-based Deep Convolutional Neural Network framework is
proposed for document structure learning. The contribution of this work
involves efficient training of region based classifiers and effective
ensembling for document image classification. A primary level of `inter-domain'
transfer learning is used by exporting weights from a pre-trained VGG16
architecture on the ImageNet dataset to train a document classifier on whole
document images. Exploiting the nature of region based influence modelling, a
secondary level of `intra-domain' transfer learning is used for rapid training
of deep learning models for image segments. Finally, stacked generalization
based ensembling is utilized for combining the predictions of the base deep
neural network models. The proposed method achieves state-of-the-art accuracy
of 92.2% on the popular RVL-CDIP document image dataset, exceeding benchmarks
set by existing algorithms.

negative:
While significant research advances have been made in the field of deep
reinforcement learning, there have been no concrete adversarial attack
strategies in literature tailored for studying the vulnerability of deep
reinforcement learning algorithms to membership inference attacks. In such
attacking systems, the adversary targets the set of collected input data on
which the deep reinforcement learning algorithm has been trained. To address
this gap, we propose an adversarial attack framework designed for testing the
vulnerability of a state-of-the-art deep reinforcement learning algorithm to a
membership inference attack. In particular, we design a series of experiments
to investigate the impact of temporal correlation, which naturally exists in
reinforcement learning training data, on the probability of information
leakage. Moreover, we compare the performance of \emph{collective} and
\emph{individual} membership attacks against the deep reinforcement learning
algorithm. Experimental results show that the proposed adversarial attack
framework is surprisingly effective at inferring data with an accuracy
exceeding $84\%$ in individual and $97\%$ in collective modes in three
different continuous control Mujoco tasks, which raises serious privacy
concerns in this regard. Finally, we show that the learning state of the
reinforcement learning algorithm influences the level of privacy breaches
significantly.

negative:
Document classification is a challenging task with important applications.
The deep learning approaches to the problem have gained much attention
recently. Despite the progress, the proposed models do not incorporate the
knowledge of the document structure in the architecture efficiently and not
take into account the contexting importance of words and sentences. In this
paper, we propose a new approach based on a combination of convolutional neural
networks, gated recurrent units, and attention mechanisms for document
classification tasks. The main contribution of this work is the use of
convolution layers to extract more meaningful, generalizable and abstract
features by the hierarchical representation. The proposed method in this paper
improves the results of the current attention-based approaches for document
classification.

negative:
Novel contexts may often arise in complex querying scenarios such as in
evidence-based medicine (EBM) involving biomedical literature, that may not
explicitly refer to entities or canonical concept forms occurring in any fact-
or rule-based knowledge source such as an ontology like the UMLS. Moreover,
hidden associations between candidate concepts meaningful in the current
context, may not exist within a single document, but within the collection, via
alternate lexical forms. Therefore, inspired by the recent success of
sequence-to-sequence neural models in delivering the state-of-the-art in a wide
range of NLP tasks, we develop a novel sequence-to-set framework with neural
attention for learning document representations that can effect term transfer
within the corpus, for semantically tagging a large collection of documents. We
demonstrate that our proposed method can be effective in both a supervised
multi-label classification setup for text categorization, as well as in a
unique unsupervised setting with no human-annotated document labels that uses
no external knowledge resources and only corpus-derived term statistics to
drive the training. Further, we show that semi-supervised training using our
architecture on large amounts of unlabeled data can augment performance on the
text categorization task when limited labeled data is available. Our approach
to generate document encodings employing our sequence-to-set models for
inference of semantic tags, gives to the best of our knowledge, the
state-of-the-art for both, the unsupervised query expansion task for the TREC
CDS 2016 challenge dataset when evaluated on an Okapi BM25--based document
retrieval system; and also over the MLTM baseline (Soleimani et al, 2016), for
both supervised and semi-supervised multi-label prediction tasks on the
del.icio.us and Ohsumed datasets. We will make our code and data publicly
available.

negative:
Rapid increase of digitized document give birth to high demand of document
image retrieval. While conventional document image retrieval approaches depend
on complex OCR-based text recognition and text similarity detection, this paper
proposes a new content-based approach, in which more attention is paid to
features extraction and fusion. In the proposed approach, multiple features of
document images are extracted by different CNN models. After that, the
extracted CNN features are reduced and fused into weighted average feature.
Finally, the document images are ranked based on feature similarity to a
provided query image. Experimental procedure is performed on a group of
document images that transformed from academic papers, which contain both
English and Chinese document, the results show that the proposed approach has
good ability to retrieve document images with similar text content, and the
fusion of CNN features can effectively improve the retrieval accuracy.

negative:
We consider the framework of Document Modeling, which lays the formal basis
for representing the document lifecycle in Business Process Management systems.
We formulate document models in the scope of the logic-based Semantic Modeling
language and study the question whether transactions given by a document model
terminate on any input. We show that in general this problem is undecidable and
formulate sufficient conditions, which guarantee decidability and polynomial
boundedness of effects of transactions.

negative:
This paper presents a new state-of-the-art for document image classification
and retrieval, using features learned by deep convolutional neural networks
(CNNs). In object and scene analysis, deep neural nets are capable of learning
a hierarchical chain of abstraction from pixel inputs to concise and
descriptive representations. The current work explores this capacity in the
realm of document analysis, and confirms that this representation strategy is
superior to a variety of popular hand-crafted alternatives. Experiments also
show that (i) features extracted from CNNs are robust to compression, (ii) CNNs
trained on non-document images transfer well to document analysis tasks, and
(iii) enforcing region-specific feature-learning is unnecessary given
sufficient training data. This work also makes available a new labelled subset
of the IIT-CDIP collection, containing 400,000 document images across 16
categories, useful for training new CNNs for document analysis.

negative:
The quality of a document is affected by various factors, including
grammaticality, readability, stylistics, and expertise depth, making the task
of document quality assessment a complex one. In this paper, we explore this
task in the context of assessing the quality of Wikipedia articles and academic
papers. Observing that the visual rendering of a document can capture implicit
quality indicators that are not present in the document text --- such as
images, font choices, and visual layout --- we propose a joint model that
combines the text content with a visual rendering of the document for document
quality assessment. Experimental results over two datasets reveal that textual
and visual features are complementary, achieving state-of-the-art results.

negative:
Document date is essential for many important tasks, such as document
retrieval, summarization, event detection, etc. While existing approaches for
these tasks assume accurate knowledge of the document date, this is not always
available, especially for arbitrary documents from the Web. Document Dating is
a challenging problem which requires inference over the temporal structure of
the document. Prior document dating systems have largely relied on handcrafted
features while ignoring such document internal structures. In this paper, we
propose NeuralDater, a Graph Convolutional Network (GCN) based document dating
approach which jointly exploits syntactic and temporal graph structures of
document in a principled way. To the best of our knowledge, this is the first
application of deep learning for the problem of document dating. Through
extensive experiments on real-world datasets, we find that NeuralDater
significantly outperforms state-of-the-art baseline by 19% absolute (45%
relative) accuracy points.