query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : In my reinforcement learning (RL) research, I encountered a significant challenge: the inability of a policy learned in one environment to adapt or generalize to another. I am seeking an efficient method to transfer policies between environments without the need to train a new policy from scratch each time the environment changes. Currently, I am considering the use of natural language to describe the entire dynamics of the environment or to outline the anticipated changes in the environment. To implement this, I require a language model capable of encoding these natural language descriptions. The resulting encoded latent representations should aid in facilitating policy adaptation to different environments. It appears crucial to align the text's meaning with the environment's dynamics, including transition probability and reward functions. Ultimately, I aim to assess the effectiveness of this approach in terms of transfer and multi-task scenarios across various environments and compare it with existing models. I am interested in any literature related to the use of textual descriptions in RL and any existing datasets suitable for evaluating my proposed methodology.

positive:
Obtaining policies that can generalise to new environments in reinforcement
learning is challenging. In this work, we demonstrate that language
understanding via a reading policy learner is a promising vehicle for
generalisation to new environments. We propose a grounded policy learning
problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason
over a language goal, relevant dynamics described in a document, and
environment observations. We procedurally generate environment dynamics and
corresponding language descriptions of the dynamics, such that agents must read
to understand new environment dynamics instead of memorising any particular
information. In addition, we propose txt2$\pi$, a model that captures three-way
interactions between the goal, document, and observations. On RTFM, txt2$\pi$
generalises to new environments with dynamics not seen during training via
reading. Furthermore, our model outperforms baselines such as FiLM and
language-conditioned CNNs on RTFM. Through curriculum learning, txt2$\pi$
produces policies that excel on complex RTFM tasks requiring several reasoning
and coreference steps.

positive:
Despite recent breakthroughs in reinforcement learning (RL) and imitation
learning (IL), existing algorithms fail to generalize beyond the training
environments. In reality, humans can adapt to new tasks quickly by leveraging
prior knowledge about the world such as language descriptions. To facilitate
the research on language-guided agents with domain adaption, we propose a novel
zero-shot compositional policy learning task, where the environments are
characterized as a composition of different attributes. Since there are no
public environments supporting this study, we introduce a new research platform
BabyAI++ in which the dynamics of environments are disentangled from visual
appearance. At each episode, BabyAI++ provides varied vision-dynamics
combinations along with corresponding descriptive texts. To evaluate the
adaption capability of learned agents, a set of vision-dynamics pairings are
held-out for testing on BabyAI++. Unsurprisingly, we find that current
language-guided RL/IL techniques overfit to the training environments and
suffer from a huge performance drop when facing unseen combinations. In
response, we propose a multi-modal fusion method with an attention mechanism to
perform visual language-grounding. Extensive experiments show strong evidence
that language grounding is able to improve the generalization of agents across
environments with varied dynamics.

positive:
Deep Reinforcement Learning (RL) is mainly studied in a setting where the
training and the testing environments are similar. But in many practical
applications, these environments may differ. For instance, in control systems,
the robot(s) on which a policy is learned might differ from the robot(s) on
which a policy will run. It can be caused by different internal factors (e.g.,
calibration issues, system attrition, defective modules) or also by external
changes (e.g., weather conditions). There is a need to develop RL methods that
generalize well to variations of the training conditions. In this article, we
consider the simplest yet hard to tackle generalization setting where the test
environment is unknown at train time, forcing the agent to adapt to the
system's new dynamics. This online adaptation process can be computationally
expensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there
is just a single train environment. To do so, we propose an approach where we
learn a subspace of policies within the parameter space. This subspace contains
an infinite number of policies that are trained to solve the training
environment while having different parameter values. As a consequence, two
policies in that subspace process information differently and exhibit different
behaviors when facing variations of the train environment. Our experiments
carried out over a large variety of benchmarks compare our approach with
baselines, including diversity-based methods. In comparison, our approach is
simple to tune, does not need any extra component (e.g., discriminator) and
learns policies able to gather a high reward on unseen environments.

positive:
Reinforcement Learning has shown success in a number of complex virtual
environments. However, many challenges still exist towards solving problems
with natural language as a core component. Interactive Fiction Games (or Text
Games) are one such problem type that offer a set of partially observable
environments where natural language is required as part of the reinforcement
learning solutions.
  Therefore, this survey's aim is to assist in the development of new Text Game
problem settings and solutions for Reinforcement Learning informed by natural
language. Specifically, this survey summarises: 1) the challenges introduced in
Text Game Reinforcement Learning problems, 2) the generation tools for
evaluating Text Games and the subsequent environments generated and, 3) the
agent architectures currently applied are compared to provide a systematic
review of benchmark methodologies and opportunities for future researchers.

positive:
Deep reinforcement learning (RL) has made it possible to solve complex
robotics problems using neural networks as function approximators. However, the
policies trained on stationary environments suffer in terms of generalization
when transferred from one environment to another. In this work, we use Robust
Markov Decision Processes (RMDP) to train the drone control policy, which
combines ideas from Robust Control and RL. It opts for pessimistic optimization
to handle potential gaps between policy transfer from one environment to
another. The trained control policy is tested on the task of quadcopter
positional control. RL agents were trained in a MuJoCo simulator. During
testing, different environment parameters (unseen during the training) were
used to validate the robustness of the trained policy for transfer from one
environment to another. The robust policy outperformed the standard agents in
these environments, suggesting that the added robustness increases generality
and can adapt to non-stationary environments.
  Codes: https://github.com/adipandas/gym_multirotor

positive:
The ability to transfer a policy from one environment to another is a
promising avenue for efficient robot learning in realistic settings where task
supervision is not available. This can allow us to take advantage of
environments well suited for training, such as simulators or laboratories, to
learn a policy for a real robot in a home or office. To succeed, such policy
transfer must overcome both the visual domain gap (e.g. different illumination
or background) and the dynamics domain gap (e.g. different robot calibration or
modelling error) between source and target environments. However, prior policy
transfer approaches either cannot handle a large domain gap or can only address
one type of domain gap at a time. In this paper, we propose a novel policy
transfer method with iterative "environment grounding", IDAPT, that alternates
between (1) directly minimizing both visual and dynamics domain gaps by
grounding the source environment in the target environment domains, and (2)
training a policy on the grounded source environment. This iterative training
progressively aligns the domains between the two environments and adapts the
policy to the target environment. Once trained, the policy can be directly
executed on the target environment. The empirical results on locomotion and
robotic manipulation tasks demonstrate that our approach can effectively
transfer a policy across visual and dynamics domain gaps with minimal
supervision and interaction with the target environment. Videos and code are
available at https://clvrai.com/idapt .

positive:
Deep reinforcement learning could be used to learn dexterous robotic policies
but it is challenging to transfer them to new robots with vastly different
hardware properties. It is also prohibitively expensive to learn a new policy
from scratch for each robot hardware due to the high sample complexity of
modern state-of-the-art algorithms. We propose a novel approach called
\textit{Hardware Conditioned Policies} where we train a universal policy
conditioned on a vector representation of robot hardware. We considered robots
in simulation with varied dynamics, kinematic structure, kinematic lengths and
degrees-of-freedom. First, we use the kinematic structure directly as the
hardware encoding and show great zero-shot transfer to completely novel robots
not seen during training. For robots with lower zero-shot success rate, we also
demonstrate that fine-tuning the policy network is significantly more
sample-efficient than training a model from scratch. In tasks where knowing the
agent dynamics is important for success, we learn an embedding for robot
hardware and show that policies conditioned on the encoding of hardware tend to
generalize and transfer well. The code and videos are available on the project
webpage: https://sites.google.com/view/robot-transfer-hcp.

positive:
Transfer and adaptation to new unknown environmental dynamics is a key
challenge for reinforcement learning (RL). An even greater challenge is
performing near-optimally in a single attempt at test time, possibly without
access to dense rewards, which is not addressed by current methods that require
multiple experience rollouts for adaptation. To achieve single episode transfer
in a family of environments with related dynamics, we propose a general
algorithm that optimizes a probe and an inference model to rapidly estimate
underlying latent variables of test dynamics, which are then immediately used
as input to a universal control policy. This modular approach enables
integration of state-of-the-art algorithms for variational inference or RL.
Moreover, our approach does not require access to rewards at test time,
allowing it to perform in settings where existing adaptive approaches cannot.
In diverse experimental domains with a single episode test constraint, our
method significantly outperforms existing adaptive approaches and shows
favorable performance against baselines for robust transfer.

positive:
In this work we present a technique to use natural language to help
reinforcement learning generalize to unseen environments. This technique uses
neural machine translation, specifically the use of encoder-decoder networks,
to learn associations between natural language behavior descriptions and
state-action information. We then use this learned model to guide agent
exploration using a modified version of policy shaping to make it more
effective at learning in unseen environments. We evaluate this technique using
the popular arcade game, Frogger, under ideal and non-ideal conditions. This
evaluation shows that our modified policy shaping algorithm improves over a
Q-learning agent as well as a baseline version of policy shaping.

positive:
The ability to adapt to changes in environmental contingencies is an
important challenge in reinforcement learning. Indeed, transferring previously
acquired knowledge to environments with unseen structural properties can
greatly enhance the flexibility and efficiency by which novel optimal policies
may be constructed. In this work, we study the problem of transfer learning
under changes in the environment dynamics. In this study, we apply causal
reasoning in the offline reinforcement learning setting to transfer a learned
policy to new environments. Specifically, we use the Decision Transformer (DT)
architecture to distill a new policy on the new environment. The DT is trained
on data collected by performing policy rollouts on factual and counterfactual
simulations from the source environment. We show that this mechanism can
bootstrap a successful policy on the target environment while retaining most of
the reward.

positive:
The ability to act in multiple environments and transfer previous knowledge
to new situations can be considered a critical aspect of any intelligent agent.
Towards this goal, we define a novel method of multitask and transfer learning
that enables an autonomous agent to learn how to behave in multiple tasks
simultaneously, and then generalize its knowledge to new domains. This method,
termed "Actor-Mimic", exploits the use of deep reinforcement learning and model
compression techniques to train a single policy network that learns how to act
in a set of distinct tasks by using the guidance of several expert teachers. We
then show that the representations learnt by the deep policy network are
capable of generalizing to new tasks with no prior expert guidance, speeding up
learning in novel environments. Although our method can in general be applied
to a wide range of problems, we use Atari games as a testing environment to
demonstrate these methods.

positive:
We investigate the use of natural language to drive the generalization of
control policies and introduce the new multi-task environment Messenger with
free-form text manuals describing the environment dynamics. Unlike previous
work, Messenger does not assume prior knowledge connecting text and state
observations $-$ the control policy must simultaneously ground the game manual
to entity symbols and dynamics in the environment. We develop a new model, EMMA
(Entity Mapper with Multi-modal Attention) which uses an entity-conditioned
attention module that allows for selective focus over relevant descriptions in
the manual for each entity in the environment. EMMA is end-to-end
differentiable and learns a latent grounding of entities and dynamics from text
to observations using only environment rewards. EMMA achieves successful
zero-shot generalization to unseen games with new dynamics, obtaining a 40%
higher win rate compared to multiple baselines. However, win rate on the
hardest stage of Messenger remains low (10%), demonstrating the need for
additional work in this direction.

positive:
Recent breakthroughs in Go play and strategic games have witnessed the great
potential of reinforcement learning in intelligently scheduling in uncertain
environment, but some bottlenecks are also encountered when we generalize this
paradigm to universal complex tasks. Among them, the low efficiency of data
utilization in model-free reinforcement algorithms is of great concern. In
contrast, the model-based reinforcement learning algorithms can reveal
underlying dynamics in learning environments and seldom suffer the data
utilization problem. To address the problem, a model-based reinforcement
learning algorithm with attention mechanism embedded is proposed as an
extension of World Models in this paper. We learn the environment model through
Mixture Density Network Recurrent Network(MDN-RNN) for agents to interact, with
combinations of variational auto-encoder(VAE) and attention incorporated in
state value estimates during the process of learning policy. In this way, agent
can learn optimal policies through less interactions with actual environment,
and final experiments demonstrate the effectiveness of our model in control
problem.

positive:
The ability to quickly solve a wide range of real-world tasks requires a
commonsense understanding of the world. Yet, how to best extract such knowledge
from natural language corpora and integrate it with reinforcement learning (RL)
agents remains an open challenge. This is partly due to the lack of lightweight
simulation environments that sufficiently reflect the semantics of the real
world and provide knowledge sources grounded with respect to observations in an
RL environment. To better enable research on agents making use of commonsense
knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2.
This lightweight environment is fast to run and built upon entities and
relations inspired by real-world semantics. We evaluate several representation
learning methods on this new benchmark and propose a new method for integrating
knowledge graphs with an RL agent.

positive:
Efficient and robust policy transfer remains a key challenge for
reinforcement learning to become viable for real-wold robotics. Policy transfer
through warm initialization, imitation, or interacting over a large set of
agents with randomized instances, have been commonly applied to solve a variety
of Reinforcement Learning tasks. However, this seems far from how skill
transfer happens in the biological world: Humans and animals are able to
quickly adapt the learned behaviors between similar tasks and learn new skills
when presented with new situations. Here we seek to answer the question: Will
learning to combine adaptation and exploration lead to a more efficient
transfer of policies between domains? We introduce a principled mechanism that
can "Adapt-to-Learn", that is adapt the source policy to learn to solve a
target task with significant transition differences and uncertainties. We show
that the presented method learns to seamlessly combine learning from adaptation
and exploration and leads to a robust policy transfer algorithm with
significantly reduced sample complexity in transferring skills between related
tasks.

positive:
Machine Learning models become increasingly proficient in complex tasks.
However, even for experts in the field, it can be difficult to understand what
the model learned. This hampers trust and acceptance, and it obstructs the
possibility to correct the model. There is therefore a need for transparency of
machine learning models. The development of transparent classification models
has received much attention, but there are few developments for achieving
transparent Reinforcement Learning (RL) models. In this study we propose a
method that enables a RL agent to explain its behavior in terms of the expected
consequences of state transitions and outcomes. First, we define a translation
of states and actions to a description that is easier to understand for human
users. Second, we developed a procedure that enables the agent to obtain the
consequences of a single action, as well as its entire policy. The method
calculates contrasts between the consequences of a policy derived from a user
query, and of the learned policy of the agent. Third, a format for generating
explanations was constructed. A pilot survey study was conducted to explore
preferences of users for different explanation properties. Results indicate
that human users tend to favor explanations about policy rather than about
single actions.

negative:
We present VisualHints, a novel environment for multimodal reinforcement
learning (RL) involving text-based interactions along with visual hints
(obtained from the environment). Real-life problems often demand that agents
interact with the environment using both natural language information and
visual perception towards solving a goal. However, most traditional RL
environments either solve pure vision-based tasks like Atari games or
video-based robotic manipulation; or entirely use natural language as a mode of
interaction, like Text-based games and dialog systems. In this work, we aim to
bridge this gap and unify these two approaches in a single environment for
multimodal RL. We introduce an extension of the TextWorld cooking environment
with the addition of visual clues interspersed throughout the environment. The
goal is to force an RL agent to use both text and visual features to predict
natural language action commands for solving the final task of cooking a meal.
We enable variations and difficulties in our environment to emulate various
interactive real-world scenarios. We present a baseline multimodal agent for
solving such problems using CNN-based feature extraction from visual hints and
LSTMs for textual feature extraction. We believe that our proposed
visual-lingual environment will facilitate novel problem settings for the RL
community.

negative:
Efficiently adapting to new environments and changes in dynamics is critical
for agents to successfully operate in the real world. Reinforcement learning
(RL) based approaches typically rely on external reward feedback for
adaptation. However, in many scenarios this reward signal might not be readily
available for the target task, or the difference between the environments can
be implicit and only observable from the dynamics. To this end, we introduce a
method that allows for self-adaptation of learned policies: No-Reward Meta
Learning (NoRML). NoRML extends Model Agnostic Meta Learning (MAML) for RL and
uses observable dynamics of the environment instead of an explicit reward
function in MAML's finetune step. Our method has a more expressive update step
than MAML, while maintaining MAML's gradient based foundation. Additionally, in
order to allow more targeted exploration, we implement an extension to MAML
that effectively disconnects the meta-policy parameters from the fine-tuned
policies' parameters. We first study our method on a number of synthetic
control problems and then validate our method on common benchmark environments,
showing that NoRML outperforms MAML when the dynamics change between tasks.

negative:
Being able to transfer existing skills to new situations is a key capability
when training robots to operate in unpredictable real-world environments. A
successful transfer algorithm should not only minimize the number of samples
that the robot needs to collect in the new environment, but also prevent the
robot from damaging itself or the surrounding environment during the transfer
process. In this work, we introduce a policy transfer algorithm for adapting
robot motor skills to novel scenarios while minimizing serious failures. Our
algorithm trains two control policies in the training environment: a task
policy that is optimized to complete the task of interest, and a protective
policy that is dedicated to keep the robot from unsafe events (e.g. falling to
the ground). To decide which policy to use during execution, we learn a safety
estimator model in the training environment that estimates a continuous safety
level of the robot. When used with a set of thresholds, the safety estimator
becomes a classifier for switching between the protective policy and the task
policy. We evaluate our approach on four simulated robot locomotion problems
and a 2D navigation problem and show that our method can achieve successful
transfer to notably different environments while taking the robot's safety into
consideration.

negative:
A key challenge in reinforcement learning (RL) is environment generalization:
a policy trained to solve a task in one environment often fails to solve the
same task in a slightly different test environment. A common approach to
improve inter-environment transfer is to learn policies that are invariant to
the distribution of testing environments. However, we argue that instead of
being invariant, the policy should identify the specific nuances of an
environment and exploit them to achieve better performance. In this work, we
propose the 'Environment-Probing' Interaction (EPI) policy, a policy that
probes a new environment to extract an implicit understanding of that
environment's behavior. Once this environment-specific information is obtained,
it is used as an additional input to a task-specific policy that can now
perform environment-conditioned actions to solve a task. To learn these
EPI-policies, we present a reward function based on transition predictability.
Specifically, a higher reward is given if the trajectory generated by the
EPI-policy can be used to better predict transitions. We experimentally show
that EPI-conditioned task-specific policies significantly outperform commonly
used policy generalization methods on novel testing environments.

negative:
One practical challenge in reinforcement learning (RL) is how to make quick
adaptations when faced with new environments. In this paper, we propose a
principled framework for adaptive RL, called \textit{AdaRL}, that adapts
reliably and efficiently to changes across domains with a few samples from the
target domain, even in partially observable environments. Specifically, we
leverage a parsimonious graphical representation that characterizes structural
relationships over variables in the RL system. Such graphical representations
provide a compact way to encode what and where the changes across domains are,
and furthermore inform us with a minimal set of changes that one has to
consider for the purpose of policy adaptation. We show that by explicitly
leveraging this compact representation to encode changes, we can efficiently
adapt the policy to the target domain, in which only a few samples are needed
and further policy optimization is avoided. We illustrate the efficacy of AdaRL
through a series of experiments that vary factors in the observation,
transition, and reward functions for Cartpole and Atari games.

negative:
A wide range of reinforcement learning (RL) problems - including robustness,
transfer learning, unsupervised RL, and emergent complexity - require
specifying a distribution of tasks or environments in which a policy will be
trained. However, creating a useful distribution of environments is error
prone, and takes a significant amount of developer time and effort. We propose
Unsupervised Environment Design (UED) as an alternative paradigm, where
developers provide environments with unknown parameters, and these parameters
are used to automatically produce a distribution over valid, solvable
environments. Existing approaches to automatically generating environments
suffer from common failure modes: domain randomization cannot generate
structure or adapt the difficulty of the environment to the agent's learning
progress, and minimax adversarial training leads to worst-case environments
that are often unsolvable. To generate structured, solvable environments for
our protagonist agent, we introduce a second, antagonist agent that is allied
with the environment-generating adversary. The adversary is motivated to
generate environments which maximize regret, defined as the difference between
the protagonist and antagonist agent's return. We call our technique
Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our
experiments demonstrate that PAIRED produces a natural curriculum of
increasingly complex environments, and PAIRED agents achieve higher zero-shot
transfer performance when tested in highly novel environments.

negative:
To be successful in real-world tasks, Reinforcement Learning (RL) needs to
exploit the compositional, relational, and hierarchical structure of the world,
and learn to transfer it to the task at hand. Recent advances in representation
learning for language make it possible to build models that acquire world
knowledge from text corpora and integrate this knowledge into downstream
decision making problems. We thus argue that the time is right to investigate a
tight integration of natural language understanding into RL in particular. We
survey the state of the field, including work on instruction following, text
games, and learning from textual domain knowledge. Finally, we call for the
development of new environments as well as further investigation into the
potential uses of recent Natural Language Processing (NLP) techniques for such
tasks.

negative:
A grand goal in AI is to build a robot that can accurately navigate based on
natural language instructions, which requires the agent to perceive the scene,
understand and ground language, and act in the real-world environment. One key
challenge here is to learn to navigate in new environments that are unseen
during training. Most of the existing approaches perform dramatically worse in
unseen environments as compared to seen ones. In this paper, we present a
generalizable navigational agent. Our agent is trained in two stages. The first
stage is training via mixed imitation and reinforcement learning, combining the
benefits from both off-policy and on-policy optimization. The second stage is
fine-tuning via newly-introduced 'unseen' triplets (environment, path,
instruction). To generate these unseen triplets, we propose a simple but
effective 'environmental dropout' method to mimic unseen environments, which
overcomes the problem of limited seen environment variability. Next, we apply
semi-supervised learning (via back-translation) on these dropped-out
environments to generate new paths and instructions. Empirically, we show that
our agent is substantially better at generalizability when fine-tuned with
these triplets, outperforming the state-of-art approaches by a large margin on
the private unseen test set of the Room-to-Room task, and achieving the top
rank on the leaderboard.

negative:
Despite ample motivation from costly exploration and limited trajectory data,
rapidly adapting to new environments with few-shot reinforcement learning (RL)
can remain a challenging task, especially with respect to personalized
settings. Here, we consider the problem of recommending optimal policies to a
set of multiple entities each with potentially different characteristics, such
that individual entities may parameterize distinct environments with unique
transition dynamics. Inspired by existing literature in meta-learning, we
extend previous work by focusing on the notion that certain environments are
more similar to each other than others in personalized settings, and propose a
model-free meta-learning algorithm that prioritizes past experiences by
relevance during gradient-based adaptation. Our algorithm involves
characterizing past policy divergence through methods in inverse reinforcement
learning, and we illustrate how such metrics are able to effectively
distinguish past policy parameters by the environment they were deployed in,
leading to more effective fast adaptation during test time. To study
personalization more effectively we introduce a navigation testbed to
specifically incorporate environment diversity across training episodes, and
demonstrate that our approach outperforms meta-learning alternatives with
respect to few-shot reinforcement learning in personalized settings.

negative:
Reinforcement learning would enjoy better success on real-world problems if
domain knowledge could be imparted to the algorithm by the modelers. Most
problems have both hidden state and unknown dynamics. Partially observable
Markov decision processes (POMDPs) allow for the modeling of both.
Unfortunately, they do not provide a natural framework in which to specify
knowledge about the domain dynamics. The designer must either admit to knowing
nothing about the dynamics or completely specify the dynamics (thereby turning
it into a planning problem). We propose a new framework called a partially
known Markov decision process (PKMDP) which allows the designer to specify
known dynamics while still leaving portions of the environment s dynamics
unknown.The model represents NOT ONLY the environment dynamics but also the
agents knowledge of the dynamics. We present a reinforcement learning algorithm
for this model based on importance sampling. The algorithm incorporates
planning based on the known dynamics and learning about the unknown dynamics.
Our results clearly demonstrate the ability to add domain knowledge and the
resulting benefits for learning.

negative:
Standard RL algorithms assume fixed environment dynamics and require a
significant amount of interaction to adapt to new environments. We introduce
Policy-Dynamics Value Functions (PD-VF), a novel approach for rapidly adapting
to dynamics different from those previously seen in training. PD-VF explicitly
estimates the cumulative reward in a space of policies and environments. An
ensemble of conventional RL policies is used to gather experience on training
environments, from which embeddings of both policies and environments can be
learned. Then, a value function conditioned on both embeddings is trained. At
test time, a few actions are sufficient to infer the environment embedding,
enabling a policy to be selected by maximizing the learned value function
(which requires no additional environment interaction). We show that our method
can rapidly adapt to new dynamics on a set of MuJoCo domains. Code available at
https://github.com/rraileanu/policy-dynamics-value-functions.

negative:
A central capability of a long-lived reinforcement learning (RL) agent is to
incrementally adapt its behavior as its environment changes, and to
incrementally build upon previous experiences to facilitate future learning in
real-world scenarios. In this paper, we propose LifeLong Incremental
Reinforcement Learning (LLIRL), a new incremental algorithm for efficient
lifelong adaptation to dynamic environments. We develop and maintain a library
that contains an infinite mixture of parameterized environment models, which is
equivalent to clustering environment parameters in a latent space. The prior
distribution over the mixture is formulated as a Chinese restaurant process
(CRP), which incrementally instantiates new environment models without any
external information to signal environmental changes in advance. During
lifelong learning, we employ the expectation maximization (EM) algorithm with
online Bayesian inference to update the mixture in a fully incremental manner.
In EM, the E-step involves estimating the posterior expectation of
environment-to-cluster assignments, while the M-step updates the environment
parameters for future learning. This method allows for all environment models
to be adapted as necessary, with new models instantiated for environmental
changes and old models retrieved when previously seen environments are
encountered again. Experiments demonstrate that LLIRL outperforms relevant
existing methods, and enables effective incremental adaptation to various
dynamic environments for lifelong learning.

negative:
Reinforcement learning (RL), particularly in sparse reward settings, often
requires prohibitively large numbers of interactions with the environment,
thereby limiting its applicability to complex problems. To address this,
several prior approaches have used natural language to guide the agent's
exploration. However, these approaches typically operate on structured
representations of the environment, and/or assume some structure in the natural
language commands. In this work, we propose a model that directly maps pixels
to rewards, given a free-form natural language description of the task, which
can then be used for policy learning. Our experiments on the Meta-World robot
manipulation domain show that language-based rewards significantly improves the
sample efficiency of policy learning, both in sparse and dense reward settings.

negative:
In this paper we target the problem of transferring policies across multiple
environments with different dynamics parameters and motor noise variations, by
introducing a framework that decouples the processes of policy learning and
system identification. Efficiently transferring learned policies to an unknown
environment with changes in dynamics configurations in the presence of motor
noise is very important for operating robots in the real world, and our work is
a novel attempt in that direction. We introduce MANGA: Method Agnostic
Neural-policy Generalization and Adaptation, that trains dynamics conditioned
policies and efficiently learns to estimate the dynamics parameters of the
environment given off-policy state-transition rollouts in the environment. Our
scheme is agnostic to the type of training method used - both reinforcement
learning (RL) and imitation learning (IL) strategies can be used. We
demonstrate the effectiveness of our approach by experimenting with four
different MuJoCo agents and comparing against previously proposed transfer
baselines.

negative:
Reinforcement Learning methods are capable of solving complex problems, but
resulting policies might perform poorly in environments that are even slightly
different. In robotics especially, training and deployment conditions often
vary and data collection is expensive, making retraining undesirable.
Simulation training allows for feasible training times, but on the other hand
suffers from a reality-gap when applied in real-world settings. This raises the
need of efficient adaptation of policies acting in new environments. We
consider this as a problem of transferring knowledge within a family of similar
Markov decision processes.
  For this purpose we assume that Q-functions are generated by some
low-dimensional latent variable. Given such a Q-function, we can find a master
policy that can adapt given different values of this latent variable. Our
method learns both the generative mapping and an approximate posterior of the
latent variables, enabling identification of policies for new tasks by
searching only in the latent space, rather than the space of all policies. The
low-dimensional space, and master policy found by our method enables policies
to quickly adapt to new environments. We demonstrate the method on both a
pendulum swing-up task in simulation, and for simulation-to-real transfer on a
pushing task.

negative:
This paper presents a comprehensive literature review on applications of deep
reinforcement learning in communications and networking. Modern networks, e.g.,
Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become
more decentralized and autonomous. In such networks, network entities need to
make decisions locally to maximize the network performance under uncertainty of
network environment. Reinforcement learning has been efficiently used to enable
the network entities to obtain the optimal policy including, e.g., decisions or
actions, given their states when the state and action spaces are small.
However, in complex and large-scale networks, the state and action spaces are
usually large, and the reinforcement learning may not be able to find the
optimal policy in reasonable time. Therefore, deep reinforcement learning, a
combination of reinforcement learning with deep learning, has been developed to
overcome the shortcomings. In this survey, we first give a tutorial of deep
reinforcement learning from fundamental concepts to advanced models. Then, we
review deep reinforcement learning approaches proposed to address emerging
issues in communications and networking. The issues include dynamic network
access, data rate control, wireless caching, data offloading, network security,
and connectivity preservation which are all important to next generation
networks such as 5G and beyond. Furthermore, we present applications of deep
reinforcement learning for traffic routing, resource sharing, and data
collection. Finally, we highlight important challenges, open issues, and future
research directions of applying deep reinforcement learning.

negative:
Reinforcement learning (RL) is a technique to learn the control policy for an
agent that interacts with a stochastic environment. In any given state, the
agent takes some action, and the environment determines the probability
distribution over the next state as well as gives the agent some reward. Most
RL algorithms typically assume that the environment satisfies Markov
assumptions (i.e. the probability distribution over the next state depends only
on the current state). In this paper, we propose a model-based RL technique for
a system that has non-Markovian dynamics. Such environments are common in many
real-world applications such as in human physiology, biological systems,
material science, and population dynamics. Model-based RL (MBRL) techniques
typically try to simultaneously learn a model of the environment from the data,
as well as try to identify an optimal policy for the learned model. We propose
a technique where the non-Markovianity of the system is modeled through a
fractional dynamical system. We show that we can quantify the difference in the
performance of an MBRL algorithm that uses bounded horizon model predictive
control from the optimal policy. Finally, we demonstrate our proposed framework
on a pharmacokinetic model of human blood glucose dynamics and show that our
fractional models can capture distant correlations on real-world datasets.

negative:
Recently, text world games have been proposed to enable artificial agents to
understand and reason about real-world scenarios. These text-based games are
challenging for artificial agents, as it requires an understanding of and
interaction using natural language in a partially observable environment.
Agents observe the environment via textual descriptions designed to be
challenging enough for even human players. Past approaches have not paid enough
attention to the language understanding capability of the proposed agents.
Typically, these approaches train from scratch, an agent that learns both
textual representations and the gameplay online during training using a
temporal loss function. Given the sample-inefficiency of RL approaches, it is
inefficient to learn rich enough textual representations to be able to
understand and reason using the textual observation in such a complicated game
environment setting. In this paper, we improve the semantic understanding of
the agent by proposing a simple RL with LM framework where we use
transformer-based language models with Deep RL models. We perform a detailed
study of our framework to demonstrate how our model outperforms all existing
agents on the popular game, Zork1, to achieve a score of 44.7, which is 1.6
higher than the state-of-the-art model. Overall, our proposed approach
outperforms 4 games out of the 14 text-based games, while performing comparable
to the state-of-the-art models on the remaining games.

negative:
Complex, multi-task problems have proven to be difficult to solve efficiently
in a sparse-reward reinforcement learning setting. In order to be sample
efficient, multi-task learning requires reuse and sharing of low-level
policies. To facilitate the automatic decomposition of hierarchical tasks, we
propose the use of step-by-step human demonstrations in the form of natural
language instructions and action trajectories. We introduce a dataset of such
demonstrations in a crafting-based grid world. Our model consists of a
high-level language generator and low-level policy, conditioned on language. We
find that human demonstrations help solve the most complex tasks. We also find
that incorporating natural language allows the model to generalize to unseen
tasks in a zero-shot setting and to learn quickly from a few demonstrations.
Generalization is not only reflected in the actions of the agent, but also in
the generated natural language instructions in unseen tasks. Our approach also
gives our trained agent interpretable behaviors because it is able to generate
a sequence of high-level descriptions of its actions.

negative:
Reinforcement learning (RL) algorithms typically start tabula rasa, without
any prior knowledge of the environment, and without any prior skills. This
however often leads to low sample efficiency, requiring a large amount of
interaction with the environment. This is especially true in a lifelong
learning setting, in which the agent needs to continually extend its
capabilities. In this paper, we examine how a pre-trained task-independent
language model can make a goal-conditional RL agent more sample efficient. We
do this by facilitating transfer learning between different related tasks. We
experimentally demonstrate our approach on a set of object navigation tasks.

negative:
This paper introduces a novel architecture for reinforcement learning with
deep neural networks designed to handle state and action spaces characterized
by natural language, as found in text-based games. Termed a deep reinforcement
relevance network (DRRN), the architecture represents action and state spaces
with separate embedding vectors, which are combined with an interaction
function to approximate the Q-function in reinforcement learning. We evaluate
the DRRN on two popular text games, showing superior performance over other
deep Q-learning architectures. Experiments with paraphrased action descriptions
show that the model is extracting meaning rather than simply memorizing strings
of text.

negative:
Recent work has described neural-network-based agents that are trained with
reinforcement learning (RL) to execute language-like commands in simulated
worlds, as a step towards an intelligent agent or robot that can be instructed
by human users. However, the optimisation of multi-goal motor policies via deep
RL from scratch requires many episodes of experience. Consequently,
instruction-following with deep RL typically involves language generated from
templates (by an environment simulator), which does not reflect the varied or
ambiguous expressions of real users. Here, we propose a conceptually simple
method for training instruction-following agents with deep RL that are robust
to natural human instructions. By applying our method with a state-of-the-art
pre-trained text-based language model (BERT), on tasks requiring agents to
identify and position everyday objects relative to other objects in a
naturalistic 3D simulated room, we demonstrate substantially-above-chance
zero-shot transfer from synthetic template commands to natural instructions
given by humans. Our approach is a general recipe for training any deep
RL-based system to interface with human users, and bridges the gap between two
research directions of notable recent success: agent-centric motor behavior and
text-based representation learning.

negative:
Reinforcement learning (RL) has been used in a range of simulated real-world
tasks, e.g., sensor coordination, traffic light control, and on-demand mobility
services. However, real world deployments are rare, as RL struggles with
dynamic nature of real world environments, requiring time for learning a task
and adapting to changes in the environment. Transfer Learning (TL) can help
lower these adaptation times. In particular, there is a significant potential
of applying TL in multi-agent RL systems, where multiple agents can share
knowledge with each other, as well as with new agents that join the system. To
obtain the most from inter-agent transfer, transfer roles (i.e., determining
which agents act as sources and which as targets), as well as relevant transfer
content parameters (e.g., transfer size) should be selected dynamically in each
particular situation. As a first step towards fully dynamic transfers, in this
paper we investigate the impact of TL transfer parameters with fixed source and
target roles. Specifically, we label every agent-environment interaction with
agent's epistemic confidence, and we filter the shared examples using varying
threshold levels and sample sizes. We investigate impact of these parameters in
two scenarios, a standard predator-prey RL benchmark and a simulation of a
ride-sharing system with 200 vehicle agents and 10,000 ride-requests.

negative:
The capability of a reinforcement learning (RL) agent heavily depends on the
diversity of the learning scenarios generated by the environment. Generation of
diverse realistic scenarios is challenging for real-time strategy (RTS)
environments. The RTS environments are characterized by intelligent
entities/non-RL agents cooperating and competing with the RL agents with large
state and action spaces over a long period of time, resulting in an infinite
space of feasible, but not necessarily realistic, scenarios involving complex
interaction among different RL and non-RL agents. Yet, most of the existing
simulators rely on randomly generating the environments based on predefined
settings/layouts and offer limited flexibility and control over the environment
dynamics for researchers to generate diverse, realistic scenarios as per their
demand. To address this issue, for the first time, we formally introduce the
benefits of adopting an existing formal scenario specification language,
SCENIC, to assist researchers to model and generate diverse scenarios in an RTS
environment in a flexible, systematic, and programmatic manner. To showcase the
benefits, we interfaced SCENIC to an existing RTS environment Google Research
Football(GRF) simulator and introduced a benchmark consisting of 32 realistic
scenarios, encoded in SCENIC, to train RL agents and testing their
generalization capabilities. We also show how researchers/RL practitioners can
incorporate their domain knowledge to expedite the training process by
intuitively modeling stochastic programmatic policies with SCENIC.

negative:
Robots that are trained to perform a task in a fixed environment often fail
when facing unexpected changes to the environment due to a lack of exploration.
We propose a principled way to adapt the policy for better exploration in
changing sparse-reward environments. Unlike previous works which explicitly
model environmental changes, we analyze the relationship between the value
function and the optimal exploration for a Gaussian-parameterized policy and
show that our theory leads to an effective strategy for adjusting the variance
of the policy, enabling fast adapt to changes in a variety of sparse-reward
environments.

negative:
Deep reinforcement learning has over the past few years shown great potential
in learning near-optimal control in complex simulated environments with little
visible information. Rainbow (Q-Learning) and PPO (Policy Optimisation) have
shown outstanding performance in a variety of tasks, including Atari 2600,
MuJoCo, and Roboschool test suite. While these algorithms are fundamentally
different, both suffer from high variance, low sample efficiency, and
hyperparameter sensitivity that in practice, make these algorithms a no-go for
critical operations in the industry.
  On the other hand, model-based reinforcement learning focuses on learning the
transition dynamics between states in an environment. If these environment
dynamics are adequately learned, a model-based approach is perhaps the most
sample efficient method for learning agents to act in an environment optimally.
The traits of model-based reinforcement are ideal for real-world environments
where sampling is slow and for mission-critical operations. In the warehouse
industry, there is an increasing motivation to minimise time and to maximise
production. Currently, autonomous agents act suboptimally using handcrafted
policies for significant portions of the state-space.
  In this paper, we present The Dreaming Variational Autoencoder v2 (DVAE-2), a
model-based reinforcement learning algorithm that increases sample efficiency,
hence enable algorithms with low sample efficiency function better in
real-world environments. We introduce Deep Warehouse, a simulated environment
for industry-near testing of autonomous agents in grid-based warehouses.
Finally, we illustrate that DVAE-2 improves the sample efficiency for the Deep
Warehouse compared to model-free methods.

negative:
Many real-world systems problems require reasoning about the long term
consequences of actions taken to configure and manage the system. These
problems with delayed and often sequentially aggregated reward, are often
inherently reinforcement learning problems and present the opportunity to
leverage the recent substantial advances in deep reinforcement learning.
However, in some cases, it is not clear why deep reinforcement learning is a
good fit for the problem. Sometimes, it does not perform better than the
state-of-the-art solutions. And in other cases, random search or greedy
algorithms could outperform deep reinforcement learning. In this paper, we
review, discuss, and evaluate the recent trends of using deep reinforcement
learning in system optimization. We propose a set of essential metrics to guide
future works in evaluating the efficacy of using deep reinforcement learning in
system optimization. Our evaluation includes challenges, the types of problems,
their formulation in the deep reinforcement learning setting, embedding, the
model used, efficiency, and robustness. We conclude with a discussion on open
challenges and potential directions for pushing further the integration of
reinforcement learning in system optimization.

negative:
Autonomous Vehicles (AVs) are required to operate safely and efficiently in
dynamic environments. For this, the AVs equipped with Joint
Radar-Communications (JRC) functions can enhance the driving safety by
utilizing both radar detection and data communication functions. However,
optimizing the performance of the AV system with two different functions under
uncertainty and dynamic of surrounding environments is very challenging. In
this work, we first propose an intelligent optimization framework based on the
Markov Decision Process (MDP) to help the AV make optimal decisions in
selecting JRC operation functions under the dynamic and uncertainty of the
surrounding environment. We then develop an effective learning algorithm
leveraging recent advances of deep reinforcement learning techniques to find
the optimal policy for the AV without requiring any prior information about
surrounding environment. Furthermore, to make our proposed framework more
scalable, we develop a Transfer Learning (TL) mechanism that enables the AV to
leverage valuable experiences for accelerating the training process when it
moves to a new environment. Extensive simulations show that the proposed
transferable deep reinforcement learning framework reduces the obstacle miss
detection probability by the AV up to 67% compared to other conventional deep
reinforcement learning approaches.

negative:
Explicit engineering of reward functions for given environments has been a
major hindrance to reinforcement learning methods. While Inverse Reinforcement
Learning (IRL) is a solution to recover reward functions from demonstrations
only, these learned rewards are generally heavily \textit{entangled} with the
dynamics of the environment and therefore not portable or \emph{robust} to
changing environments. Modern adversarial methods have yielded some success in
reducing reward entanglement in the IRL setting. In this work, we leverage one
such method, Adversarial Inverse Reinforcement Learning (AIRL), to propose an
algorithm that learns hierarchical disentangled rewards with a policy over
options. We show that this method has the ability to learn \emph{generalizable}
policies and reward functions in complex transfer learning tasks, while
yielding results in continuous control benchmarks that are comparable to those
of the state-of-the-art methods.

negative:
In order for robots to perform mission-critical tasks, it is essential that
they are able to quickly adapt to changes in their environment as well as to
injuries and or other bodily changes. Deep reinforcement learning has been
shown to be successful in training robot control policies for operation in
complex environments. However, existing methods typically employ only a single
policy. This can limit the adaptability since a large environmental
modification might require a completely different behavior compared to the
learning environment. To solve this problem, we propose Map-based Multi-Policy
Reinforcement Learning (MMPRL), which aims to search and store multiple
policies that encode different behavioral features while maximizing the
expected reward in advance of the environment change. Thanks to these policies,
which are stored into a multi-dimensional discrete map according to its
behavioral feature, adaptation can be performed within reasonable time without
retraining the robot. An appropriate pre-trained policy from the map can be
recalled using Bayesian optimization. Our experiments show that MMPRL enables
robots to quickly adapt to large changes without requiring any prior knowledge
on the type of injuries that could occur. A highlight of the learned behaviors
can be found here: https://youtu.be/QwInbilXNOE .

negative:
One of the central challenges faced by a reinforcement learning (RL) agent is
to effectively learn a (near-)optimal policy in environments with large state
spaces having sparse and noisy feedback signals. In real-world applications, an
expert with additional domain knowledge can help in speeding up the learning
process via \emph{shaping the environment}, i.e., making the environment more
learner-friendly. A popular paradigm in literature is \emph{potential-based
reward shaping}, where the environment's reward function is augmented with
additional local rewards using a potential function. However, the applicability
of potential-based reward shaping is limited in settings where (i) the state
space is very large, and it is challenging to compute an appropriate potential
function, (ii) the feedback signals are noisy, and even with shaped rewards the
agent could be trapped in local optima, and (iii) changing the rewards alone is
not sufficient, and effective shaping requires changing the dynamics. We
address these limitations of potential-based shaping methods and propose a
novel framework of \emph{environment shaping using state abstraction}. Our key
idea is to compress the environment's large state space with noisy signals to
an abstracted space, and to use this abstraction in creating smoother and more
effective feedback signals for the agent. We study the theoretical
underpinnings of our abstraction-based environment shaping, and show that the
agent's policy learnt in the shaped environment preserves near-optimal behavior
in the original environment.

negative:
In this paper, we present a new approach to Transfer Learning (TL) in
Reinforcement Learning (RL) for cross-domain tasks. Many of the available
techniques approach the transfer architecture as a method of speeding up the
target task learning. We propose to adapt and reuse the mapped source task
optimal-policy directly in related domains. We show the optimal policy from a
related source task can be near optimal in target domain provided an adaptive
policy accounts for the model error between target and source. The main benefit
of this policy augmentation is generalizing policies across multiple related
domains without having to re-learn the new tasks. Our results show that this
architecture leads to better sample efficiency in the transfer, reducing sample
complexity of target task learning to target apprentice learning.

negative:
Existing work in language grounding typically study single environments. How
do we build unified models that apply across multiple environments? We propose
the multi-environment Symbolic Interactive Language Grounding benchmark (SILG),
which unifies a collection of diverse grounded language learning environments
under a common interface. SILG consists of grid-world environments that require
generalization to new dynamics, entities, and partially observed worlds (RTFM,
Messenger, NetHack), as well as symbolic counterparts of visual worlds that
require interpreting rich natural language with respect to complex scenes
(ALFWorld, Touchdown). Together, these environments provide diverse grounding
challenges in richness of observation space, action space, language
specification, and plan complexity. In addition, we propose the first shared
model architecture for RL on these environments, and evaluate recent advances
such as egocentric local convolution, recurrent state-tracking, entity-centric
attention, and pretrained LM using SILG. Our shared architecture achieves
comparable performance to environment-specific architectures. Moreover, we find
that many recent modelling advances do not result in significant gains on
environments other than the one they were designed for. This highlights the
need for a multi-environment benchmark. Finally, the best models significantly
underperform humans on SILG, which suggests ample room for future work. We hope
SILG enables the community to quickly identify new methodologies for language
grounding that generalize to a diverse set of environments and their associated
challenges.

negative:
We consider problems of making sequences of decisions to accomplish tasks,
interacting via the medium of language. These problems are often tackled with
reinforcement learning approaches. We find that these models do not generalize
well when applied to novel task domains. However, the large amount of
computation necessary to adequately train and explore the search space of
sequential decision making, under a reinforcement learning paradigm, precludes
the inclusion of large contextualized language models, which might otherwise
enable the desired generalization ability. We introduce a teacher-student
imitation learning methodology and a means of converting a reinforcement
learning model into a natural language understanding model. Together, these
methodologies enable the introduction of contextualized language models into
the sequential decision making problem space. We show that models can learn
faster and generalize more, leveraging both the imitation learning and the
reformulation. Our models exceed teacher performance on various held-out
decision problems, by up to 7% on in-domain problems and 24% on out-of-domain
problems.

negative:
Recent advances in reinforcement learning have shown its potential to tackle
complex real-life tasks. However, as the dimensionality of the task increases,
reinforcement learning methods tend to struggle. To overcome this, we explore
methods for representing the semantic information embedded in the state. While
previous methods focused on information in its raw form (e.g., raw visual
input), we propose to represent the state using natural language. Language can
represent complex scenarios and concepts, making it a favorable candidate for
representation. Empirical evidence, within the domain of ViZDoom, suggests that
natural language based agents are more robust, converge faster and perform
better than vision based agents, showing the benefit of using natural language
representations for reinforcement learning.

negative:
Reinforcement learning provides a powerful and general framework for decision
making and control, but its application in practice is often hindered by the
need for extensive feature and reward engineering. Deep reinforcement learning
methods can remove the need for explicit engineering of policy or value
features, but still require a manually specified reward function. Inverse
reinforcement learning holds the promise of automatic reward acquisition, but
has proven exceptionally difficult to apply to large, high-dimensional problems
with unknown dynamics. In this work, we propose adverserial inverse
reinforcement learning (AIRL), a practical and scalable inverse reinforcement
learning algorithm based on an adversarial reward learning formulation. We
demonstrate that AIRL is able to recover reward functions that are robust to
changes in dynamics, enabling us to learn policies even under significant
variation in the environment seen during training. Our experiments show that
AIRL greatly outperforms prior methods in these transfer settings.

negative:
The objective of a reinforcement learning agent is to behave so as to
maximise the sum of a suitable scalar function of state: the reward. These
rewards are typically given and immutable. In this paper, we instead consider
the proposition that the reward function itself can be a good locus of learned
knowledge. To investigate this, we propose a scalable meta-gradient framework
for learning useful intrinsic reward functions across multiple lifetimes of
experience. Through several proof-of-concept experiments, we show that it is
feasible to learn and capture knowledge about long-term exploration and
exploitation into a reward function. Furthermore, we show that unlike policy
transfer methods that capture "how" the agent should behave, the learned reward
functions can generalise to other kinds of agents and to changes in the
dynamics of the environment by capturing "what" the agent should strive to do.

negative:
In this work we apply deep reinforcement learning to the problems of
navigating a three-dimensional environment and inferring the locations of human
speaker audio sources within, in the case where the only available information
is the raw sound from the environment, as a simulated human listener placed in
the environment would hear it. For this purpose we create two virtual
environments using the Unity game engine, one presenting an audio-based
navigation problem and one presenting an audio source localization problem. We
also create an autonomous agent based on PPO online reinforcement learning
algorithm and attempt to train it to solve these environments. Our experiments
show that our agent achieves adequate performance and generalization ability in
both environments, measured by quantitative metrics, even when a limited amount
of training data are available or the environment parameters shift in ways not
encountered during training. We also show that a degree of agent knowledge
transfer is possible between the environments.

negative:
To further improve the learning efficiency and performance of reinforcement
learning (RL), in this paper we propose a novel uncertainty-aware model-based
RL (UA-MBRL) framework, and then implement and validate it in autonomous
driving under various task scenarios. First, an action-conditioned ensemble
model with the ability of uncertainty assessment is established as the virtual
environment model. Then, a novel uncertainty-aware model-based RL framework is
developed based on the adaptive truncation approach, providing virtual
interactions between the agent and environment model, and improving RL's
training efficiency and performance. The developed algorithms are then
implemented in end-to-end autonomous vehicle control tasks, validated and
compared with state-of-the-art methods under various driving scenarios. The
validation results suggest that the proposed UA-MBRL method surpasses the
existing model-based and model-free RL approaches, in terms of learning
efficiency and achieved performance. The results also demonstrate the good
ability of the proposed method with respect to the adaptiveness and robustness,
under various autonomous driving scenarios.

negative:
In this paper, we consider the source of Deep Reinforcement Learning (DRL)'s
sample complexity, asking how much derives from the requirement of learning
useful representations of environment states and how much is due to the sample
complexity of learning a policy. While for DRL agents, the distinction between
representation and policy may not be clear, we seek new insight through a set
of transfer learning experiments. In each experiment, we retain some fraction
of layers trained on either the same game or a related game, comparing the
benefits of transfer learning to learning a policy from scratch. Interestingly,
we find that benefits due to transfer are highly variable in general and
non-symmetric across pairs of tasks. Our experiments suggest that perhaps
transfer from simpler environments can boost performance on more complex
downstream tasks and that the requirements of learning a useful representation
can range from negligible to the majority of the sample complexity, based on
the environment. Furthermore, we find that fine-tuning generally outperforms
training with the transferred layers frozen, confirming an insight first noted
in the classification setting.

negative:
Reinforcement learning from large-scale offline datasets provides us with the
ability to learn policies without potentially unsafe or impractical
exploration. Significant progress has been made in the past few years in
dealing with the challenge of correcting for differing behavior between the
data collection and learned policies. However, little attention has been paid
to potentially changing dynamics when transferring a policy to the online
setting, where performance can be up to 90% reduced for existing methods. In
this paper we address this problem with Augmented World Models (AugWM). We
augment a learned dynamics model with simple transformations that seek to
capture potential changes in physical properties of the robot, leading to more
robust policies. We not only train our policy in this new setting, but also
provide it with the sampled augmentation as a context, allowing it to adapt to
changes in the environment. At test time we learn the context in a
self-supervised fashion by approximating the augmentation which corresponds to
the new environment. We rigorously evaluate our approach on over 100 different
changed dynamics settings, and show that this simple approach can significantly
improve the zero-shot generalization of a recent state-of-the-art baseline,
often achieving successful policies where the baseline fails.

negative:
Sample-efficient generalisation of reinforcement learning approaches have
always been a challenge, especially, for complex scenes with many components.
In this work, we introduce Plug and Play Markov Decision Processes, an
object-based representation that allows zero-shot integration of new objects
from known object classes. This is achieved by representing the global
transition dynamics as a union of local transition functions, each with respect
to one active object in the scene. Transition dynamics from an object class can
be pre-learnt and thus would be ready to use in a new environment. Each active
object is also endowed with its reward function. Since there is no central
reward function, addition or removal of objects can be handled efficiently by
only updating the reward functions of objects involved. A new transfer learning
mechanism is also proposed to adapt reward function in such cases. Experiments
show that our representation can achieve sample-efficiency in a variety of
set-ups.

negative:
In recent years some researchers have explored the use of reinforcement
learning (RL) algorithms as key components in the solution of various natural
language processing tasks. For instance, some of these algorithms leveraging
deep neural learning have found their way into conversational systems. This
paper reviews the state of the art of RL methods for their possible use for
different problems of natural language processing, focusing primarily on
conversational systems, mainly due to their growing relevance. We provide
detailed descriptions of the problems as well as discussions of why RL is
well-suited to solve them. Also, we analyze the advantages and limitations of
these methods. Finally, we elaborate on promising research directions in
natural language processing that might benefit from reinforcement learning.

negative:
A suitable state representation is a fundamental part of the learning process
in Reinforcement Learning. In various tasks, the state can either be described
by natural language or be natural language itself. This survey outlines the
strategies used in the literature to build natural language state
representations. We appeal for more linguistically interpretable and grounded
representations, careful justification of design decisions and evaluation of
the effectiveness of different approaches.

negative:
Humans generally use natural language to communicate task requirements to
each other. Ideally, natural language should also be usable for communicating
goals to autonomous machines (e.g., robots) to minimize friction in task
specification. However, understanding and mapping natural language goals to
sequences of states and actions is challenging. Specifically, existing work
along these lines has encountered difficulty in generalizing learned policies
to new natural language goals and environments. In this paper, we propose a
novel adversarial inverse reinforcement learning algorithm to learn a
language-conditioned policy and reward function. To improve generalization of
the learned policy and reward function, we use a variational goal generator to
relabel trajectories and sample diverse goals during training. Our algorithm
outperforms multiple baselines by a large margin on a vision-based natural
language instruction following dataset (Room-2-Room), demonstrating a promising
advance in enabling the use of natural language instructions in specifying
agent goals.

negative:
Recent work has shown that deep reinforcement-learning agents can learn to
follow language-like instructions from infrequent environment rewards. However,
this places on environment designers the onus of designing language-conditional
reward functions which may not be easily or tractably implemented as the
complexity of the environment and the language scales. To overcome this
limitation, we present a framework within which instruction-conditional RL
agents are trained using rewards obtained not from the environment, but from
reward models which are jointly trained from expert examples. As reward models
improve, they learn to accurately reward agents for completing tasks for
environment configurations---and for instructions---not present amongst the
expert data. This framework effectively separates the representation of what
instructions require from how they can be executed. In a simple grid world, it
enables an agent to learn a range of commands requiring interaction with blocks
and understanding of spatial relations and underspecified abstract
arrangements. We further show the method allows our agent to adapt to changes
in the environment without requiring new expert examples.

negative:
The ability to transfer in reinforcement learning is key towards building an
agent of general artificial intelligence. In this paper, we consider the
problem of learning to simultaneously transfer across both environments (ENV)
and tasks (TASK), probably more importantly, by learning from only sparse (ENV,
TASK) pairs out of all the possible combinations. We propose a novel
compositional neural network architecture which depicts a meta rule for
composing policies from the environment and task embeddings. Notably, one of
the main challenges is to learn the embeddings jointly with the meta rule. We
further propose new training methods to disentangle the embeddings, making them
both distinctive signatures of the environments and tasks and effective
building blocks for composing the policies. Experiments on GridWorld and Thor,
of which the agent takes as input an egocentric view, show that our approach
gives rise to high success rates on all the (ENV, TASK) pairs after learning
from only 40% of them.

negative:
Reinforcement learning (RL) can automate a wide variety of robotic skills,
but learning each new skill requires considerable real-world data collection
and manual representation engineering to design policy classes or features.
Using deep reinforcement learning to train general purpose neural network
policies alleviates some of the burden of manual representation engineering by
using expressive policy classes, but exacerbates the challenge of data
collection, since such methods tend to be less efficient than RL with
low-dimensional, hand-designed representations. Transfer learning can mitigate
this problem by enabling us to transfer information from one skill to another
and even from one robot to another. We show that neural network policies can be
decomposed into "task-specific" and "robot-specific" modules, where the
task-specific modules are shared across robots, and the robot-specific modules
are shared across all tasks on that robot. This allows for sharing task
information, such as perception, between robots and sharing robot information,
such as dynamics and kinematics, between tasks. We exploit this decomposition
to train mix-and-match modules that can solve new robot-task combinations that
were not seen during training. Using a novel neural network architecture, we
demonstrate the effectiveness of our transfer method for enabling zero-shot
generalization with a variety of robots and tasks in simulation for both visual
and non-visual tasks.

negative:
Learning policies for complex tasks that require multiple different skills is
a major challenge in reinforcement learning (RL). It is also a requirement for
its deployment in real-world scenarios. This paper proposes a novel framework
for efficient multi-task reinforcement learning. Our framework trains agents to
employ hierarchical policies that decide when to use a previously learned
policy and when to learn a new skill. This enables agents to continually
acquire new skills during different stages of training. Each learned task
corresponds to a human language description. Because agents can only access
previously learned skills through these descriptions, the agent can always
provide a human-interpretable description of its choices. In order to help the
agent learn the complex temporal dependencies necessary for the hierarchical
policy, we provide it with a stochastic temporal grammar that modulates when to
rely on previously learned skills and when to execute new skills. We validate
our approach on Minecraft games designed to explicitly test the ability to
reuse previously learned skills while simultaneously learning new skills.

negative:
In most real world scenarios, a policy trained by reinforcement learning in
one environment needs to be deployed in another, potentially quite different
environment. However, generalization across different environments is known to
be hard. A natural solution would be to keep training after deployment in the
new environment, but this cannot be done if the new environment offers no
reward signal. Our work explores the use of self-supervision to allow the
policy to continue training after deployment without using any rewards. While
previous methods explicitly anticipate changes in the new environment, we
assume no prior knowledge of those changes yet still obtain significant
improvements. Empirical evaluations are performed on diverse simulation
environments from DeepMind Control suite and ViZDoom, as well as real robotic
manipulation tasks in continuously changing environments, taking observations
from an uncalibrated camera. Our method improves generalization in 31 out of 36
environments across various tasks and outperforms domain randomization on a
majority of environments.

negative:
Robots need to be able to work in multiple different environments. Even when
performing similar tasks, different behaviour should be deployed to best fit
the current environment. In this paper, We propose a new approach to
navigation, where it is treated as a multi-task learning problem. This enables
the robot to learn to behave differently in visual navigation tasks for
different environments while also learning shared expertise across
environments. We evaluated our approach in both simulated environments as well
as real-world data. Our method allows our system to converge with a 26%
reduction in training time, while also increasing accuracy.

negative:
Traditional model-based RL relies on hand-specified or learned models of
transition dynamics of the environment. These methods are sample efficient and
facilitate learning in the real world but fail to generalize to subtle
variations in the underlying dynamics, e.g., due to differences in mass,
friction, or actuators across robotic agents or across time. We propose using
variational inference to learn an explicit latent representation of unknown
environment properties that accelerates learning and facilitates generalization
on novel environments at test time. We use Online Bayesian Inference of these
learned latents to rapidly adapt online to changes in environments without
retaining large replay buffers of recent data. Combined with a neural network
ensemble that models dynamics and captures uncertainty over dynamics, our
approach demonstrates positive transfer during training and online adaptation
on the continuous control task HalfCheetah.

negative:
In typical reinforcement learning (RL), the environment is assumed given and
the goal of the learning is to identify an optimal policy for the agent taking
actions through its interactions with the environment. In this paper, we extend
this setting by considering the environment is not given, but controllable and
learnable through its interaction with the agent at the same time. This
extension is motivated by environment design scenarios in the real-world,
including game design, shopping space design and traffic signal design.
Theoretically, we find a dual Markov decision process (MDP) w.r.t. the
environment to that w.r.t. the agent, and derive a policy gradient solution to
optimizing the parametrized environment. Furthermore, discontinuous
environments are addressed by a proposed general generative framework. Our
experiments on a Maze game design task show the effectiveness of the proposed
algorithms in generating diverse and challenging Mazes against various agent
settings.

negative:
Reinforcement learning is a learning paradigm for solving sequential
decision-making problems. Recent years have witnessed remarkable progress in
reinforcement learning upon the fast development of deep neural networks. Along
with the promising prospects of reinforcement learning in numerous domains such
as robotics and game-playing, transfer learning has arisen to tackle various
challenges faced by reinforcement learning, by transferring knowledge from
external expertise to facilitate the efficiency and effectiveness of the
learning process. In this survey, we systematically investigate the recent
progress of transfer learning approaches in the context of deep reinforcement
learning. Specifically, we provide a framework for categorizing the
state-of-the-art transfer learning approaches, under which we analyze their
goals, methodologies, compatible reinforcement learning backbones, and
practical applications. We also draw connections between transfer learning and
other relevant topics from the reinforcement learning perspective and explore
their potential challenges that await future research progress.

negative:
Agents trained with deep reinforcement learning algorithms are capable of
performing highly complex tasks including locomotion in continuous
environments. We investigate transferring the learning acquired in one task to
a set of previously unseen tasks. Generalization and overfitting in deep
reinforcement learning are not commonly addressed in current transfer learning
research. Conducting a comparative analysis without an intermediate
regularization step results in underperforming benchmarks and inaccurate
algorithm comparisons due to rudimentary assessments. In this study, we propose
regularization techniques in deep reinforcement learning for continuous control
through the application of sample elimination, early stopping and maximum
entropy regularized adversarial learning. First, the importance of the
inclusion of training iteration number to the hyperparameters in deep transfer
reinforcement learning will be discussed. Because source task performance is
not indicative of the generalization capacity of the algorithm, we start by
acknowledging the training iteration number as a hyperparameter. In line with
this, we introduce an additional step of resorting to earlier snapshots of
policy parameters to prevent overfitting to the source task. Then, to generate
robust policies, we discard the samples that lead to overfitting via a method
we call strict clipping. Furthermore, we increase the generalization capacity
in widely used transfer learning benchmarks by using maximum entropy
regularization, different critic methods, and curriculum learning in an
adversarial setup. Subsequently, we propose maximum entropy adversarial
reinforcement learning to increase the domain randomization. Finally, we
evaluate the robustness of these methods on simulated robots in target
environments where the morphology of the robot, gravity, and tangential
friction coefficient of the environment are altered.

negative:
Over its lifetime, a reinforcement learning agent is often tasked with
different tasks. How to efficiently adapt a previously learned control policy
from one task to another, remains an open research question. In this paper, we
investigate how instructions formulated in natural language can enable faster
and more effective task adaptation. This can serve as the basis for developing
language instructed skills, which can be used in a lifelong learning setting.
Our method is capable of assessing, given a set of developed base control
policies, which policy will adapt best to a new unseen task.

negative:
While reinforcement learning (RL) has been successful in natural language
processing (NLP) domains such as dialogue generation and text-based games, it
typically faces the problem of sparse rewards that leads to slow or no
convergence. Traditional methods that use text descriptions to extract only a
state representation ignore the feedback inherently present in them. In
text-based games, for example, descriptions like "Good Job! You ate the food}"
indicate progress, and descriptions like "You entered a new room" indicate
exploration. Positive and negative cues like these can be converted to rewards
through sentiment analysis. This technique converts the sparse reward problem
into a dense one, which is easier to solve. Furthermore, this can enable
reinforcement learning without rewards, in which the agent learns entirely from
these intrinsic sentiment rewards. This framework is similar to intrinsic
motivation, where the environment does not necessarily provide the rewards, but
the agent analyzes and realizes them by itself. We find that providing dense
rewards in text-based games using sentiment analysis improves performance under
some conditions.

negative:
Agents trained by reinforcement learning (RL) often fail to generalize beyond
the environment they were trained in, even when presented with new scenarios
that seem similar to the training environment. We study the query complexity
required to train RL agents that generalize to multiple environments.
Intuitively, tractable generalization is only possible when the environments
are similar or close in some sense. To capture this, we introduce Weak
Proximity, a natural structural condition that requires the environments to
have highly similar transition and reward functions and share a policy
providing optimal value. Despite such shared structure, we prove that tractable
generalization is impossible in the worst case. This holds even when each
individual environment can be efficiently solved to obtain an optimal linear
policy, and when the agent possesses a generative model. Our lower bound
applies to the more complex task of representation learning for the purpose of
efficient generalization to multiple environments. On the positive side, we
introduce Strong Proximity, a strengthened condition which we prove is
sufficient for efficient generalization.

negative:
Reinforcement learning is an active research area with a vast number of
applications in robotics, and the RoboCup competition is an interesting
environment for studying and evaluating reinforcement learning methods. A known
difficulty in applying reinforcement learning to robotics is the high number of
experience samples required, being the use of simulated environments for
training the agents followed by transfer learning to real-world (sim-to-real) a
viable path. This article introduces an open-source simulator for the IEEE Very
Small Size Soccer and the Small Size League optimized for reinforcement
learning experiments. We also propose a framework for creating OpenAI Gym
environments with a set of benchmarks tasks for evaluating single-agent and
multi-agent robot soccer skills. We then demonstrate the learning capabilities
of two state-of-the-art reinforcement learning methods as well as their
limitations in certain scenarios introduced in this framework. We believe this
will make it easier for more teams to compete in these categories using
end-to-end reinforcement learning approaches and further develop this research
area.

negative:
We propose a novel approach to address one aspect of the non-stationarity
problem in multi-agent reinforcement learning (RL), where the other agents may
alter their policies due to environment changes during execution. This violates
the Markov assumption that governs most single-agent RL methods and is one of
the key challenges in multi-agent RL. To tackle this, we propose to train
multiple policies for each agent and postpone the selection of the best policy
at execution time. Specifically, we model the environment non-stationarity with
a finite set of scenarios and train policies fitting each scenario. In addition
to multiple policies, each agent also learns a policy predictor to determine
which policy is the best with its local information. By doing so, each agent is
able to adapt its policy when the environment changes and consequentially the
other agents alter their policies during execution. We empirically evaluated
our method on a variety of common benchmark problems proposed for multi-agent
deep RL in the literature. Our experimental results show that the agents
trained by our algorithm have better adaptiveness in changing environments and
outperform the state-of-the-art methods in all the tested environments.

negative:
Deep reinforcement learning augments the reinforcement learning framework and
utilizes the powerful representation of deep neural networks. Recent works have
demonstrated the remarkable successes of deep reinforcement learning in various
domains including finance, medicine, healthcare, video games, robotics, and
computer vision. In this work, we provide a detailed review of recent and
state-of-the-art research advances of deep reinforcement learning in computer
vision. We start with comprehending the theories of deep learning,
reinforcement learning, and deep reinforcement learning. We then propose a
categorization of deep reinforcement learning methodologies and discuss their
advantages and limitations. In particular, we divide deep reinforcement
learning into seven main categories according to their applications in computer
vision, i.e. (i)landmark localization (ii) object detection; (iii) object
tracking; (iv) registration on both 2D image and 3D image volumetric data (v)
image segmentation; (vi) videos analysis; and (vii) other applications. Each of
these categories is further analyzed with reinforcement learning techniques,
network design, and performance. Moreover, we provide a comprehensive analysis
of the existing publicly available datasets and examine source code
availability. Finally, we present some open issues and discuss future research
directions on deep reinforcement learning in computer vision

negative:
This paper provides a roadmap that explores the question of how to imbue
learning agents with the ability to understand and generate contextually
relevant natural language in service of achieving a goal. We hypothesize that
two key components in creating such agents are interactivity and environment
grounding, shown to be vital parts of language learning in humans, and posit
that interactive narratives should be the environments of choice for such
training these agents. These games are simulations in which an agent interacts
with the world through natural language -- "perceiving", "acting upon", and
"talking to" the world using textual descriptions, commands, and dialogue --
and as such exist at the intersection of natural language processing,
storytelling, and sequential decision making. We discuss the unique challenges
a text games' puzzle-like structure combined with natural language
state-and-action spaces provides: knowledge representation, commonsense
reasoning, and exploration. Beyond the challenges described so far, progress in
the realm of interactive narratives can be applied in adjacent problem domains.
These applications provide interesting challenges of their own as well as
extensions to those discussed so far. We describe three of them in detail: (1)
evaluating AI system's commonsense understanding by automatically creating
interactive narratives; (2) adapting abstract text-based policies to include
other modalities such as vision; and (3) enabling multi-agent and human-AI
collaboration in shared, situated worlds.

negative:
Transfer in reinforcement learning refers to the notion that generalization
should occur not only within a task but also across tasks. We propose a
transfer framework for the scenario where the reward function changes between
tasks but the environment's dynamics remain the same. Our approach rests on two
key ideas: "successor features", a value function representation that decouples
the dynamics of the environment from the rewards, and "generalized policy
improvement", a generalization of dynamic programming's policy improvement
operation that considers a set of policies rather than a single one. Put
together, the two ideas lead to an approach that integrates seamlessly within
the reinforcement learning framework and allows the free exchange of
information across tasks. The proposed method also provides performance
guarantees for the transferred policy even before any learning has taken place.
We derive two theorems that set our approach in firm theoretical ground and
present experiments that show that it successfully promotes transfer in
practice, significantly outperforming alternative methods in a sequence of
navigation tasks and in the control of a simulated robotic arm.

negative:
We seek to align agent behavior with a user's objectives in a reinforcement
learning setting with unknown dynamics, an unknown reward function, and unknown
unsafe states. The user knows the rewards and unsafe states, but querying the
user is expensive. To address this challenge, we propose an algorithm that
safely and interactively learns a model of the user's reward function. We start
with a generative model of initial states and a forward dynamics model trained
on off-policy data. Our method uses these models to synthesize hypothetical
behaviors, asks the user to label the behaviors with rewards, and trains a
neural network to predict the rewards. The key idea is to actively synthesize
the hypothetical behaviors from scratch by maximizing tractable proxies for the
value of information, without interacting with the environment. We call this
method reward query synthesis via trajectory optimization (ReQueST). We
evaluate ReQueST with simulated users on a state-based 2D navigation task and
the image-based Car Racing video game. The results show that ReQueST
significantly outperforms prior methods in learning reward models that transfer
to new environments with different initial state distributions. Moreover,
ReQueST safely trains the reward model to detect unsafe states, and corrects
reward hacking before deploying the agent.

negative:
In this paper, we consider the task of learning control policies for
text-based games. In these games, all interactions in the virtual world are
through text and the underlying state is not observed. The resulting language
barrier makes such environments challenging for automatic game players. We
employ a deep reinforcement learning framework to jointly learn state
representations and action policies using game rewards as feedback. This
framework enables us to map text descriptions into vector representations that
capture the semantics of the game states. We evaluate our approach on two game
worlds, comparing against baselines using bag-of-words and bag-of-bigrams for
state representations. Our algorithm outperforms the baselines on both worlds
demonstrating the importance of learning expressive representations.

negative:
How can we train a dialog model to produce better conversations by learning
from human feedback, without the risk of humans teaching it harmful chat
behaviors? We start by hosting models online, and gather human feedback from
real-time, open-ended conversations, which we then use to train and improve the
models using offline reinforcement learning (RL). We identify implicit
conversational cues including language similarity, elicitation of laughter,
sentiment, and more, which indicate positive human feedback, and embed these in
multiple reward functions. A well-known challenge is that learning an RL policy
in an offline setting usually fails due to the lack of ability to explore and
the tendency to make over-optimistic estimates of future reward. These problems
become even harder when using RL for language models, which can easily have a
20,000 action vocabulary and many possible reward functions. We solve the
challenge by developing a novel class of offline RL algorithms. These
algorithms use KL-control to penalize divergence from a pre-trained prior
language model, and use a new strategy to make the algorithm pessimistic,
instead of optimistic, in the face of uncertainty. We test the resulting dialog
model with ratings from 80 users in an open-domain setting and find it achieves
significant improvements over existing deep offline RL approaches. The novel
offline RL method is viable for improving any existing generative dialog model
using a static dataset of human feedback.

negative:
This paper gives a detailed review of reinforcement learning in combinatorial
optimization, introduces the history of combinatorial optimization starting in
the 1960s, and compares it with the reinforcement learning algorithms in recent
years. We explicitly look at a famous combinatorial problem known as the
Traveling Salesperson Problem (TSP). We compare the approach of the modern
reinforcement learning algorithms on TSP with an approach published in 1970.
Then, we discuss the similarities between these algorithms and how the approach
of reinforcement learning changes due to the evolution of machine learning
techniques and computing power. We also mention the deep learning approach on
the TSP, which is named Deep Reinforcement Learning. We argue that deep
learning is a generic approach that can be integrated with traditional
reinforcement learning algorithms and optimize the outcomes of the TSP.

negative:
We consider the challenge of policy simplification and verification in the
context of policies learned through reinforcement learning (RL) in continuous
environments. In well-behaved settings, RL algorithms have convergence
guarantees in the limit. While these guarantees are valuable, they are
insufficient for safety-critical applications. Furthermore, they are lost when
applying advanced techniques such as deep-RL. To recover guarantees when
applying advanced RL algorithms to more complex environments with (i)
reachability, (ii) safety-constrained reachability, or (iii) discounted-reward
objectives, we build upon the DeepMDP framework introduced by Gelada et al. to
derive new bisimulation bounds between the unknown environment and a learned
discrete latent model of it. Our bisimulation bounds enable the application of
formal methods for Markov decision processes. Finally, we show how one can use
a policy obtained via state-of-the-art RL to efficiently train a variational
autoencoder that yields a discrete latent model with provably approximately
correct bisimulation guarantees. Additionally, we obtain a distilled version of
the policy for the latent model.

negative:
To achieve general artificial intelligence, reinforcement learning (RL)
agents should learn not only to optimize returns for one specific task but also
to constantly build more complex skills and scaffold their knowledge about the
world, without forgetting what has already been learned. In this paper, we
discuss the desired characteristics of environments that can support the
training and evaluation of lifelong reinforcement learning agents, review
existing environments from this perspective, and propose recommendations for
devising suitable environments in the future.

negative:
The goal of reinforcement learning (RL) is to let an agent learn an optimal
control policy in an unknown environment so that future expected rewards are
maximized. The model-free RL approach directly learns the policy based on data
samples. Although using many samples tends to improve the accuracy of policy
learning, collecting a large number of samples is often expensive in practice.
On the other hand, the model-based RL approach first estimates the transition
model of the environment and then learns the policy based on the estimated
transition model. Thus, if the transition model is accurately learned from a
small amount of data, the model-based approach can perform better than the
model-free approach. In this paper, we propose a novel model-based RL method by
combining a recently proposed model-free policy search method called policy
gradients with parameter-based exploration and the state-of-the-art transition
model estimator called least-squares conditional density estimation. Through
experiments, we demonstrate the practical usefulness of the proposed method.

negative:
It is a long-standing challenge to enable an intelligent agent to learn in
one environment and generalize to an unseen environment without further data
collection and finetuning. In this paper, we consider a zero shot
generalization problem setup that complies with biological intelligent agents'
learning and generalization processes. The agent is first presented with
previous experiences in the training environment, along with task description
in the form of trajectory-level sparse rewards. Later when it is placed in the
new testing environment, it is asked to perform the task without any
interaction with the testing environment. We find this setting natural for
biological creatures and at the same time, challenging for previous methods.
Behavior cloning, state-of-art RL along with other zero-shot learning methods
perform poorly on this benchmark. Given a set of experiences in the training
environment, our method learns a neural function that decomposes the sparse
reward into particular regions in a contingency-aware observation as a per step
reward. Based on such decomposed rewards, we further learn a dynamics model and
use Model Predictive Control (MPC) to obtain a policy. Since the rewards are
decomposed to finer-granularity observations, they are naturally generalizable
to new environments that are composed of similar basic elements. We demonstrate
our method on a wide range of environments, including a classic video game --
Super Mario Bros, as well as a robotic continuous control task. Please refer to
the project page for more visualized results.

negative:
Human intelligence has the remarkable ability to adapt to new tasks and
environments quickly. Starting from a very young age, humans acquire new skills
and learn how to solve new tasks either by imitating the behavior of others or
by following provided natural language instructions. To facilitate research in
this direction, we propose IGLU: Interactive Grounded Language Understanding in
a Collaborative Environment. The primary goal of the competition is to approach
the problem of how to build interactive agents that learn to solve a task while
provided with grounded natural language instructions in a collaborative
environment. Understanding the complexity of the challenge, we split it into
sub-tasks to make it feasible for participants.
  This research challenge is naturally related, but not limited, to two fields
of study that are highly relevant to the NeurIPS community: Natural Language
Understanding and Generation (NLU/G) and Reinforcement Learning (RL).
Therefore, the suggested challenge can bring two communities together to
approach one of the important challenges in AI. Another important aspect of the
challenge is the dedication to perform a human-in-the-loop evaluation as a
final evaluation for the agents developed by contestants.

negative:
Language creates a compact representation of the world and allows the
description of unlimited situations and objectives through compositionality.
While these characterizations may foster instructing, conditioning or
structuring interactive agent behavior, it remains an open-problem to correctly
relate language understanding and reinforcement learning in even simple
instruction following scenarios. This joint learning problem is alleviated
through expert demonstrations, auxiliary losses, or neural inductive biases. In
this paper, we propose an orthogonal approach called Hindsight Generation for
Experience Replay (HIGhER) that extends the Hindsight Experience Replay (HER)
approach to the language-conditioned policy setting. Whenever the agent does
not fulfill its instruction, HIGhER learns to output a new directive that
matches the agent trajectory, and it relabels the episode with a positive
reward. To do so, HIGhER learns to map a state into an instruction by using
past successful trajectories, which removes the need to have external expert
interventions to relabel episodes as in vanilla HER. We show the efficiency of
our approach in the BabyAI environment, and demonstrate how it complements
other instruction following methods.

negative:
Natural language is perhaps the most flexible and intuitive way for humans to
communicate tasks to a robot. Prior work in imitation learning typically
requires each task be specified with a task id or goal image -- something that
is often impractical in open-world environments. On the other hand, previous
approaches in instruction following allow agent behavior to be guided by
language, but typically assume structure in the observations, actuators, or
language that limit their applicability to complex settings like robotics. In
this work, we present a method for incorporating free-form natural language
conditioning into imitation learning. Our approach learns perception from
pixels, natural language understanding, and multitask continuous control
end-to-end as a single neural network. Unlike prior work in imitation learning,
our method is able to incorporate unlabeled and unstructured demonstration data
(i.e. no task or language labels). We show this dramatically improves language
conditioned performance, while reducing the cost of language annotation to less
than 1% of total data. At test time, a single language conditioned visuomotor
policy trained with our method can perform a wide variety of robotic
manipulation skills in a 3D environment, specified only with natural language
descriptions of each task (e.g. "open the drawer...now pick up the block...now
press the green button..."). To scale up the number of instructions an agent
can follow, we propose combining text conditioned policies with large
pretrained neural language models. We find this allows a policy to be robust to
many out-of-distribution synonym instructions, without requiring new
demonstrations. See videos of a human typing live text commands to our agent at
language-play.github.io

negative:
This paper studies the problem of autonomous exploration under localization
uncertainty for a mobile robot with 3D range sensing. We present a framework
for self-learning a high-performance exploration policy in a single simulation
environment, and transferring it to other environments, which may be physical
or virtual. Recent work in transfer learning achieves encouraging performance
by domain adaptation and domain randomization to expose an agent to scenarios
that fill the inherent gaps in sim2sim and sim2real approaches. However, it is
inefficient to train an agent in environments with randomized conditions to
learn the important features of its current state. An agent can use domain
knowledge provided by human experts to learn efficiently. We propose a novel
approach that uses graph neural networks in conjunction with deep reinforcement
learning, enabling decision-making over graphs containing relevant exploration
information provided by human experts to predict a robot's optimal sensing
action in belief space. The policy, which is trained only in a single
simulation environment, offers a real-time, scalable, and transferable
decision-making strategy, resulting in zero-shot transfer to other simulation
environments and even real-world environments.

negative:
Exploration is essential for reinforcement learning (RL). To face the
challenges of exploration, we consider a reward-free RL framework that
completely separates exploration from exploitation and brings new challenges
for exploration algorithms. In the exploration phase, the agent learns an
exploratory policy by interacting with a reward-free environment and collects a
dataset of transitions by executing the policy. In the planning phase, the
agent computes a good policy for any reward function based on the dataset
without further interacting with the environment. This framework is suitable
for the meta RL setting where there are many reward functions of interest. In
the exploration phase, we propose to maximize the Renyi entropy over the
state-action space and justify this objective theoretically. The success of
using Renyi entropy as the objective results from its encouragement to explore
the hard-to-reach state-actions. We further deduce a policy gradient
formulation for this objective and design a practical exploration algorithm
that can deal with complex environments. In the planning phase, we solve for
good policies given arbitrary reward functions using a batch RL algorithm.
Empirically, we show that our exploration algorithm is effective and sample
efficient, and results in superior policies for arbitrary reward functions in
the planning phase.

negative:
The named concepts and compositional operators present in natural language
provide a rich source of information about the kinds of abstractions humans use
to navigate the world. Can this linguistic background knowledge improve the
generality and efficiency of learned classifiers and control policies? This
paper aims to show that using the space of natural language strings as a
parameter space is an effective way to capture natural task structure. In a
pretraining phase, we learn a language interpretation model that transforms
inputs (e.g. images) into outputs (e.g. labels) given natural language
descriptions. To learn a new concept (e.g. a classifier), we search directly in
the space of descriptions to minimize the interpreter's loss on training
examples. Crucially, our models do not require language data to learn these
concepts: language is used only in pretraining to impose structure on
subsequent learning. Results on image classification, text editing, and
reinforcement learning show that, in all settings, models with a linguistic
parameterization outperform those without.

negative:
Deep reinforcement learning is revolutionizing the artificial intelligence
field. Currently, it serves as a good starting point for constructing
intelligent autonomous systems which offer a better knowledge of the visual
world. It is possible to scale deep reinforcement learning with the use of deep
learning and do amazing tasks such as use of pixels in playing video games. In
this paper, key concepts of deep reinforcement learning including reward
function, differences between reinforcement learning and supervised learning
and models for implementation of reinforcement are discussed. Key challenges
related to the implementation of reinforcement learning in conversational AI
domain are identified as well as discussed in detail. Various conversational
models which are based on deep reinforcement learning (as well as deep
learning) are also discussed. In summary, this paper discusses key aspects of
deep reinforcement learning which are crucial for designing an efficient
conversational AI.

negative:
Reinforcement learning has seen great advancements in the past five years.
The successful introduction of deep learning in place of more traditional
methods allowed reinforcement learning to scale to very complex domains
achieving super-human performance in environments like the game of Go or
numerous video games. Despite great successes in multiple domains, these new
methods suffer from their own issues that make them often inapplicable to the
real world problems. Extreme lack of data efficiency, together with huge
variance and difficulty in enforcing safety constraints, is one of the three
most prominent issues in the field. Usually, millions of data points sampled
from the environment are necessary for these algorithms to converge to
acceptable policies.
  This thesis proposes novel Generative Adversarial Imaginative Reinforcement
Learning algorithm. It takes advantage of the recent introduction of highly
effective generative adversarial models, and Markov property that underpins
reinforcement learning setting, to model dynamics of the real environment
within the internal imagination module. Rollouts from the imagination are then
used to artificially simulate the real environment in a standard reinforcement
learning process to avoid, often expensive and dangerous, trial and error in
the real environment. Experimental results show that the proposed algorithm
more economically utilises experience from the real environment than the
current state-of-the-art Rainbow DQN algorithm, and thus makes an important
step towards sample efficient deep reinforcement learning.

negative:
Teaching an agent to navigate in an unseen 3D environment is a challenging
task, even in the event of simulated environments. To generalize to unseen
environments, an agent needs to be robust to low-level variations (e.g. color,
texture, object changes), and also high-level variations (e.g. layout changes
of the environment). To improve overall generalization, all types of variations
in the environment have to be taken under consideration via different level of
data augmentation steps. To this end, we propose House3D, a rich, extensible
and efficient environment that contains 45,622 human-designed 3D scenes of
visually realistic houses, ranging from single-room studios to multi-storied
houses, equipped with a diverse set of fully labeled 3D objects, textures and
scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in
House3D opens the door towards scene-level augmentation, while the label-rich
nature of House3D enables us to inject pixel- & task-level augmentations such
as domain randomization (Toubin et. al.) and multi-task training. Using a
subset of houses in House3D, we show that reinforcement learning agents trained
with an enhancement of different levels of augmentations perform much better in
unseen environments than our baselines with raw RGB input by over 8% in terms
of navigation success rate. House3D is publicly available at
http://github.com/facebookresearch/House3D.

negative:
Imitation learning and instruction-following are two common approaches to
communicate a user's intent to a learning agent. However, as the complexity of
tasks grows, it could be beneficial to use both demonstrations and language to
communicate with an agent. In this work, we propose a novel setting where an
agent is given both a demonstration and a description, and must combine
information from both the modalities. Specifically, given a demonstration for a
task (the source task), and a natural language description of the differences
between the demonstrated task and a related but different task (the target
task), our goal is to train an agent to complete the target task in a zero-shot
setting, that is, without any demonstrations for the target task. To this end,
we introduce Language-Aided Reward and Value Adaptation (LARVA) which, given a
source demonstration and a linguistic description of how the target task
differs, learns to output a reward / value function that accurately describes
the target task. Our experiments show that on a diverse set of adaptations, our
approach is able to complete more than 95% of target tasks when using
template-based descriptions, and more than 70% when using free-form natural
language.

negative:
Reward learning enables the application of reinforcement learning (RL) to
tasks where reward is defined by human judgment, building a model of reward by
asking humans questions. Most work on reward learning has used simulated
environments, but complex information about values is often expressed in
natural language, and we believe reward learning for language is a key to
making RL practical and safe for real-world tasks. In this paper, we build on
advances in generative pretraining of language models to apply reward learning
to four natural language tasks: continuing text with positive sentiment or
physically descriptive language, and summarization tasks on the TL;DR and
CNN/Daily Mail datasets. For stylistic continuation we achieve good results
with only 5,000 comparisons evaluated by humans. For summarization, models
trained with 60,000 comparisons copy whole sentences from the input but skip
irrelevant preamble; this leads to reasonable ROUGE scores and very good
performance according to our human labelers, but may be exploiting the fact
that labelers rely on simple heuristics.

negative:
Several recent works have been dedicated to unsupervised reinforcement
learning in a single environment, in which a policy is first pre-trained with
unsupervised interactions, and then fine-tuned towards the optimal policy for
several downstream supervised tasks defined over the same environment. Along
this line, we address the problem of unsupervised reinforcement learning in a
class of multiple environments, in which the policy is pre-trained with
interactions from the whole class, and then fine-tuned for several tasks in any
environment of the class. Notably, the problem is inherently multi-objective as
we can trade off the pre-training objective between environments in many ways.
In this work, we foster an exploration strategy that is sensitive to the most
adverse cases within the class. Hence, we cast the exploration problem as the
maximization of the mean of a critical percentile of the state visitation
entropy induced by the exploration strategy over the class of environments.
Then, we present a policy gradient algorithm, $\alpha$MEPOL, to optimize the
introduced objective through mediated interactions with the class. Finally, we
empirically demonstrate the ability of the algorithm in learning to explore
challenging classes of continuous environments and we show that reinforcement
learning greatly benefits from the pre-trained exploration strategy w.r.t.
learning from scratch.

negative:
Reinforcement learning is one of the core components in designing an
artificial intelligent system emphasizing real-time response. Reinforcement
learning influences the system to take actions within an arbitrary environment
either having previous knowledge about the environment model or not. In this
paper, we present a comprehensive study on Reinforcement Learning focusing on
various dimensions including challenges, the recent development of different
state-of-the-art techniques, and future directions. The fundamental objective
of this paper is to provide a framework for the presentation of available
methods of reinforcement learning that is informative enough and simple to
follow for the new researchers and academics in this domain considering the
latest concerns. First, we illustrated the core techniques of reinforcement
learning in an easily understandable and comparable way. Finally, we analyzed
and depicted the recent developments in reinforcement learning approaches. My
analysis pointed out that most of the models focused on tuning policy values
rather than tuning other things in a particular state of reasoning.

negative:
Due to its nature of learning from dynamic interactions and planning for
long-run performance, reinforcement learning (RL) recently has received much
attention in interactive recommender systems (IRSs). IRSs usually face the
large discrete action space problem, which makes most of the existing RL-based
recommendation methods inefficient. Moreover, data sparsity is another
challenging problem that most IRSs are confronted with. While the textual
information like reviews and descriptions is less sensitive to sparsity,
existing RL-based recommendation methods either neglect or are not suitable for
incorporating textual information. To address these two problems, in this
paper, we propose a Text-based Deep Deterministic Policy Gradient framework
(TDDPG-Rec) for IRSs. Specifically, we leverage textual information to map
items and users into a feature space, which greatly alleviates the sparsity
problem. Moreover, we design an effective method to construct an action
candidate set. By the policy vector dynamically learned from TDDPG-Rec that
expresses the user's preference, we can select actions from the candidate set
effectively. Through experiments on three public datasets, we demonstrate that
TDDPG-Rec achieves state-of-the-art performance over several baselines in a
time-efficient manner.

negative:
This paper describes an application of reinforcement learning to the mention
detection task. We define a novel action-based formulation for the mention
detection task, in which a model can flexibly revise past labeling decisions by
grouping together tokens and assigning partial mention labels. We devise a
method to create mention-level episodes and we train a model by rewarding
correctly labeled complete mentions, irrespective of the inner structure
created. The model yields results which are on par with a competitive
supervised counterpart while being more flexible in terms of achieving targeted
behavior through reward modeling and generating internal mention structure,
especially on longer mentions.

negative:
Computer simulation provides an automatic and safe way for training robotic
control policies to achieve complex tasks such as locomotion. However, a policy
trained in simulation usually does not transfer directly to the real hardware
due to the differences between the two environments. Transfer learning using
domain randomization is a promising approach, but it usually assumes that the
target environment is close to the distribution of the training environments,
thus relying heavily on accurate system identification. In this paper, we
present a different approach that leverages domain randomization for
transferring control policies to unknown environments. The key idea that,
instead of learning a single policy in the simulation, we simultaneously learn
a family of policies that exhibit different behaviors. When tested in the
target environment, we directly search for the best policy in the family based
on the task performance, without the need to identify the dynamic parameters.
We evaluate our method on five simulated robotic control problems with
different discrepancies in the training and testing environment and demonstrate
that our method can overcome larger modeling errors compared to training a
robust policy or an adaptive policy.

negative:
We consider the problem of imitation learning from a finite set of expert
trajectories, without access to reinforcement signals. The classical approach
of extracting the expert's reward function via inverse reinforcement learning,
followed by reinforcement learning is indirect and may be computationally
expensive. Recent generative adversarial methods based on matching the policy
distribution between the expert and the agent could be unstable during
training. We propose a new framework for imitation learning by estimating the
support of the expert policy to compute a fixed reward function, which allows
us to re-frame imitation learning within the standard reinforcement learning
setting. We demonstrate the efficacy of our reward function on both discrete
and continuous domains, achieving comparable or better performance than the
state of the art under different reinforcement learning algorithms.

negative:
The speed and accuracy with which robots are able to interpret natural
language is fundamental to realizing effective human-robot interaction. A great
deal of attention has been paid to developing models and approximate inference
algorithms that improve the efficiency of language understanding. However,
existing methods still attempt to reason over a representation of the
environment that is flat and unnecessarily detailed, which limits scalability.
An open problem is then to develop methods capable of producing the most
compact environment model sufficient for accurate and efficient natural
language understanding. We propose a model that leverages environment-related
information encoded within instructions to identify the subset of observations
and perceptual classifiers necessary to perceive a succinct,
instruction-specific environment representation. The framework uses three
probabilistic graphical models trained from a corpus of annotated instructions
to infer salient scene semantics, perceptual classifiers, and grounded symbols.
Experimental results on two robots operating in different environments
demonstrate that by exploiting the content and the structure of the
instructions, our method learns compact environment representations that
significantly improve the efficiency of natural language symbol grounding.

negative:
Reinforcement learning (RL) can enable task-oriented dialogue systems to
steer the conversation towards successful task completion. In an end-to-end
setting, a response can be constructed in a word-level sequential decision
making process with the entire system vocabulary as action space. Policies
trained in such a fashion do not require expert-defined action spaces, but they
have to deal with large action spaces and long trajectories, making RL
impractical. Using the latent space of a variational model as action space
alleviates this problem. However, current approaches use an uninformed prior
for training and optimize the latent distribution solely on the context. It is
therefore unclear whether the latent representation truly encodes the
characteristics of different actions. In this paper, we explore three ways of
leveraging an auxiliary task to shape the latent variable distribution: via
pre-training, to obtain an informed prior, and via multitask learning. We
choose response auto-encoding as the auxiliary task, as this captures the
generative factors of dialogue responses while requiring low computational cost
and neither additional data nor labels. Our approach yields a more
action-characterized latent representations which support end-to-end dialogue
policy optimization and achieves state-of-the-art success rates. These results
warrant a more wide-spread use of RL in end-to-end dialogue models.

negative:
Reinforcement Learning (RL) is a powerful framework to address the
discrepancy between loss functions used during training and the final
evaluation metrics to be used at test time. When applied to neural Machine
Translation (MT), it minimises the mismatch between the cross-entropy loss and
non-differentiable evaluation metrics like BLEU. However, the suitability of
these metrics as reward function at training time is questionable: they tend to
be sparse and biased towards the specific words used in the reference texts. We
propose to address this problem by making models less reliant on such metrics
in two ways: (a) with an entropy-regularised RL method that does not only
maximise a reward function but also explore the action space to avoid peaky
distributions; (b) with a novel RL method that explores a dynamic unsupervised
reward function to balance between exploration and exploitation. We base our
proposals on the Soft Actor-Critic (SAC) framework, adapting the off-policy
maximum entropy model for language generation applications such as MT. We
demonstrate that SAC with BLEU reward tends to overfit less to the training
data and performs better on out-of-domain data. We also show that our dynamic
unsupervised reward can lead to better translation of ambiguous words.

negative:
Most Knowledge Graphs (KGs) contain textual descriptions of entities in
various natural languages. These descriptions of entities provide valuable
information that may not be explicitly represented in the structured part of
the KG. Based on this fact, some link prediction methods which make use of the
information presented in the textual descriptions of entities have been
proposed to learn representations of (monolingual) KGs. However, these methods
use entity descriptions in only one language and ignore the fact that
descriptions given in different languages may provide complementary information
and thereby also additional semantics. In this position paper, the problem of
effectively leveraging multilingual entity descriptions for the purpose of link
prediction in KGs will be discussed along with potential solutions to the
problem.

negative:
This work presents an exploration and imitation-learning-based agent capable
of state-of-the-art performance in playing text-based computer games.
Text-based computer games describe their world to the player through natural
language and expect the player to interact with the game using text. These
games are of interest as they can be seen as a testbed for language
understanding, problem-solving, and language generation by artificial agents.
Moreover, they provide a learning environment in which these skills can be
acquired through interactions with an environment rather than using fixed
corpora. One aspect that makes these games particularly challenging for
learning agents is the combinatorially large action space. Existing methods for
solving text-based games are limited to games that are either very simple or
have an action space restricted to a predetermined set of admissible actions.
In this work, we propose to use the exploration approach of Go-Explore for
solving text-based games. More specifically, in an initial exploration phase,
we first extract trajectories with high rewards, after which we train a policy
to solve the game by imitating these trajectories. Our experiments show that
this approach outperforms existing solutions in solving text-based games, and
it is more sample efficient in terms of the number of interactions with the
environment. Moreover, we show that the learned policy can generalize better
than existing solutions to unseen games without using any restriction on the
action space.

negative:
Reinforcement learning has recently experienced increased prominence in the
machine learning community. There are many approaches to solving reinforcement
learning problems with new techniques developed constantly. When solving
problems using reinforcement learning, there are various difficult challenges
to overcome. To ensure progress in the field, benchmarks are important for
testing new algorithms and comparing with other approaches. The reproducibility
of results for fair comparison is therefore vital in ensuring that improvements
are accurately judged. This paper provides an overview of different
contributions to reinforcement learning benchmarking and discusses how they can
assist researchers to address the challenges facing reinforcement learning. The
contributions discussed are the most used and recent in the literature. The
paper discusses the contributions in terms of implementation, tasks and
provided algorithm implementations with benchmarks. The survey aims to bring
attention to the wide range of reinforcement learning benchmarking tasks
available and to encourage research to take place in a standardised manner.
Additionally, this survey acts as an overview for researchers not familiar with
the different tasks that can be used to develop and test new reinforcement
learning algorithms.

negative:
Deep reinforcement learning is poised to revolutionise the field of AI and
represents a step towards building autonomous systems with a higher level
understanding of the visual world. Currently, deep learning is enabling
reinforcement learning to scale to problems that were previously intractable,
such as learning to play video games directly from pixels. Deep reinforcement
learning algorithms are also applied to robotics, allowing control policies for
robots to be learned directly from camera inputs in the real world. In this
survey, we begin with an introduction to the general field of reinforcement
learning, then progress to the main streams of value-based and policy-based
methods. Our survey will cover central algorithms in deep reinforcement
learning, including the deep $Q$-network, trust region policy optimisation, and
asynchronous advantage actor-critic. In parallel, we highlight the unique
advantages of deep neural networks, focusing on visual understanding via
reinforcement learning. To conclude, we describe several current areas of
research within the field.

negative:
Text Worlds are virtual environments for embodied agents that, unlike 2D or
3D environments, are rendered exclusively using textual descriptions. These
environments offer an alternative to higher-fidelity 3D environments due to
their low barrier to entry, providing the ability to study semantics,
compositional inference, and other high-level tasks with rich high-level action
spaces while controlling for perceptual input. This systematic survey outlines
recent developments in tooling, environments, and agent modeling for Text
Worlds, while examining recent trends in knowledge graphs, common sense
reasoning, transfer learning of Text World performance to higher-fidelity
environments, as well as near-term development targets that, once achieved,
make Text Worlds an attractive general research paradigm for natural language
processing.

negative:
Several approaches have recently been proposed for learning decentralized
deep multiagent policies that coordinate via a differentiable communication
channel. While these policies are effective for many tasks, interpretation of
their induced communication strategies has remained a challenge. Here we
propose to interpret agents' messages by translating them. Unlike in typical
machine translation problems, we have no parallel data to learn from. Instead
we develop a translation model based on the insight that agent messages and
natural language strings mean the same thing if they induce the same belief
about the world in a listener. We present theoretical guarantees and empirical
evidence that our approach preserves both the semantics and pragmatics of
messages by ensuring that players communicating through a translation layer do
not suffer a substantial loss in reward relative to players with a common
language.

negative:
Recent work has shown that recurrent neural networks can be trained to
separate individual speakers in a sound mixture with high fidelity. Here we
explore convolutional neural network models as an alternative and show that
they achieve state-of-the-art results with an order of magnitude fewer
parameters. We also characterize and compare the robustness and ability of
these different approaches to generalize under three different test conditions:
longer time sequences, the addition of intermittent noise, and different
datasets not seen during training. For the last condition, we create a new
dataset, RealTalkLibri, to test source separation in real-world environments.
We show that the acoustics of the environment have significant impact on the
structure of the waveform and the overall performance of neural network models,
with the convolutional model showing superior ability to generalize to new
environments. The code for our study is available at
https://github.com/ShariqM/source_separation.

negative:
Human infants are able to acquire natural language seemingly easily at an
early age. Their language learning seems to occur simultaneously with learning
other cognitive functions as well as with playful interactions with the
environment and caregivers. From a neuroscientific perspective, natural
language is embodied, grounded in most, if not all, sensory and sensorimotor
modalities, and acquired by means of crossmodal integration. However,
characterising the underlying mechanisms in the brain is difficult and
explaining the grounding of language in crossmodal perception and action
remains challenging. In this paper, we present a neurocognitive model for
language grounding which reflects bio-inspired mechanisms such as an implicit
adaptation of timescales as well as end-to-end multimodal abstraction. It
addresses developmental robotic interaction and extends its learning
capabilities using larger-scale knowledge-based data. In our scenario, we
utilise the humanoid robot NICO in obtaining the EMIL data collection, in which
the cognitive robot interacts with objects in a children's playground
environment while receiving linguistic labels from a caregiver. The model
analysis shows that crossmodally integrated representations are sufficient for
acquiring language merely from sensory input through interaction with objects
in an environment. The representations self-organise hierarchically and embed
temporal and spatial information through composition and decomposition. This
model can also provide the basis for further crossmodal integration of
perceptually grounded cognitive representations.

negative:
We address the problem of solving mixed random linear equations. We have
unlabeled observations coming from multiple linear regressions, and each
observation corresponds to exactly one of the regression models. The goal is to
learn the linear regressors from the observations. Classically, Alternating
Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used
to solve this problem. AM iteratively alternates between the estimation of
labels and solving the regression problems with the estimated labels.
Empirically, it is observed that, for a large variety of non-convex problems
including mixed linear regression, AM converges at a much faster rate compared
to gradient based algorithms. However, the existing theory suggests similar
rate of convergence for AM and gradient based methods, failing to capture this
empirical behavior. In this paper, we close this gap between theory and
practice for the special case of a mixture of $2$ linear regressions. We show
that, provided initialized properly, AM enjoys a \emph{super-linear} rate of
convergence in certain parameter regimes. To the best of our knowledge, this is
the first work that theoretically establishes such rate for AM. Hence, if we
want to recover the unknown regressors upto an error (in $\ell_2$ norm) of
$\epsilon$, AM only takes $\mathcal{O}(\log \log (1/\epsilon))$ iterations.
Furthermore, we compare AM with a gradient based heuristic algorithm
empirically and show that AM dominates in iteration complexity as well as
wall-clock time.