query: The query consists of users’ needs, leading to several research questions that span a paragraph. Each candidate passage is an abstract from a scientific paper. The objective of this information retrieval task is to identify the abstract that most effectively meets the user’s needs in the query : I am interested in conducting a study to examine the behavior and learning capabilities of both human and artificial intelligence (AI) agents within a simulated psychology laboratory environment. This environment must be computerized to allow AI agents to interact within it. I require a pre-existing platform or a method to construct one that enables the implementation of psychological experiments. This platform must be compatible with both human subjects and AI agents, necessitating a computerized system with a visual user interface. The platform should include an Application Programming Interface (API) that allows for the creation of customized tasks and experiments. Essential built-in functionalities should include but not be limited to object detection, object tracking, and change detection. Upon acquiring or constructing such a platform, I aim to explore how reinforcement learning (RL) agents perceive and interpret visual information. I am also interested in investigating how their learning capabilities can be enhanced through the integration of simpler models, such as a basic central vision model. I hope that this platform and the subsequent research it facilitates can contribute to the advancement of deep RL and strengthen its ties with cognitive science.

positive:
The capability to learn and adapt to changes in the driving environment is
crucial for developing autonomous driving systems that are scalable beyond
geo-fenced operational design domains. Deep Reinforcement Learning (RL)
provides a promising and scalable framework for developing adaptive learning
based solutions. Deep RL methods usually model the problem as a (Partially
Observable) Markov Decision Process in which an agent acts in a stationary
environment to learn an optimal behavior policy. However, driving involves
complex interaction between multiple, intelligent (artificial or human) agents
in a highly non-stationary environment. In this paper, we propose the use of
Partially Observable Markov Games(POSG) for formulating the connected
autonomous driving problems with realistic assumptions. We provide a taxonomy
of multi-agent learning environments based on the nature of tasks, nature of
agents and the nature of the environment to help in categorizing various
autonomous driving problems that can be addressed under the proposed
formulation. As our main contributions, we provide MACAD-Gym, a Multi-Agent
Connected, Autonomous Driving agent learning platform for furthering research
in this direction. Our MACAD-Gym platform provides an extensible set of
Connected Autonomous Driving (CAD) simulation environments that enable the
research and development of Deep RL- based integrated sensing, perception,
planning and control algorithms for CAD systems with unlimited operational
design domain under realistic, multi-agent settings. We also share the
MACAD-Agents that were trained successfully using the MACAD-Gym platform to
learn control policies for multiple vehicle agents in a partially observable,
stop-sign controlled, 3-way urban intersection environment with raw (camera)
sensor observations.

positive:
Reinforcement learning (RL) is a popular machine learning paradigm for game
playing, robotics control, and other sequential decision tasks. However, RL
agents often have long learning times with high data requirements because they
begin by acting randomly. In order to better learn in complex tasks, this
article argues that an external teacher can often significantly help the RL
agent learn.
  OpenAI Gym is a common framework for RL research, including a large number of
standard environments and agents, making RL research significantly more
accessible. This article introduces our new open-source RL framework, the Human
Input Parsing Platform for Openai Gym (HIPPO Gym), and the design decisions
that went into its creation. The goal of this platform is to facilitate
human-RL research, again lowering the bar so that more researchers can quickly
investigate different ways that human teachers could assist RL agents,
including learning from demonstrations, learning from feedback, or curriculum
learning.

negative:
We introduce AndroidEnv, an open-source platform for Reinforcement Learning
(RL) research built on top of the Android ecosystem. AndroidEnv allows RL
agents to interact with a wide variety of apps and services commonly used by
humans through a universal touchscreen interface. Since agents train on a
realistic simulation of an Android device, they have the potential to be
deployed on real devices. In this report, we give an overview of the
environment, highlighting the significant features it provides for research,
and we present an empirical evaluation of some popular reinforcement learning
agents on a set of tasks built on this platform.

negative:
Reinforcement Learning AI commonly uses reward/penalty signals that are
objective and explicit in an environment -- e.g. game score, completion time,
etc. -- in order to learn the optimal strategy for task performance. However,
Human-AI interaction for such AI agents should include additional reinforcement
that is implicit and subjective -- e.g. human preferences for certain AI
behavior -- in order to adapt the AI behavior to idiosyncratic human
preferences. Such adaptations would mirror naturally occurring processes that
increase trust and comfort during social interactions. Here, we show how a
hybrid brain-computer-interface (hBCI), which detects an individual's level of
interest in objects/events in a virtual environment, can be used to adapt the
behavior of a Deep Reinforcement Learning AI agent that is controlling a
virtual autonomous vehicle. Specifically, we show that the AI learns a driving
strategy that maintains a safe distance from a lead vehicle, and most novelly,
preferentially slows the vehicle when the human passengers of the vehicle
encounter objects of interest. This adaptation affords an additional 20\%
viewing time for subjectively interesting objects. This is the first
demonstration of how an hBCI can be used to provide implicit reinforcement to
an AI agent in a way that incorporates user preferences into the control
system.

negative:
Reinforcement learning (RL) agents are often designed specifically for a
particular problem and they generally have uninterpretable working processes.
Statistical methods-based agent algorithms can be improved in terms of
generalizability and interpretability using symbolic Artificial Intelligence
(AI) tools such as logic programming. In this study, we present a model-free RL
architecture that is supported with explicit relational representations of the
environmental objects. For the first time, we use the PrediNet network
architecture in a dynamic decision-making problem rather than image-based
tasks, and Multi-Head Dot-Product Attention Network (MHDPA) as a baseline for
performance comparisons. We tested two networks in two environments ---i.e.,
the baseline Box-World environment and our novel environment,
Relational-Grid-World (RGW). With the procedurally generated RGW environment,
which is complex in terms of visual perceptions and combinatorial selections,
it is easy to measure the relational representation performance of the RL
agents. The experiments were carried out using different configurations of the
environment so that the presented module and the environment were compared with
the baselines. We reached similar policy optimization performance results with
the PrediNet architecture and MHDPA; additionally, we achieved to extract the
propositional representation explicitly ---which makes the agent's statistical
policy logic more interpretable and tractable. This flexibility in the agent's
policy provides convenience for designing non-task-specific agent
architectures. The main contributions of this study are two-fold ---an RL agent
that can explicitly perform relational reasoning, and a new environment that
measures the relational reasoning capabilities of RL agents.

negative:
Human-robot collaboration is an essential research topic in artificial
intelligence (AI), enabling researchers to devise cognitive AI systems and
affords an intuitive means for users to interact with the robot. Of note,
communication plays a central role. To date, prior studies in embodied agent
navigation have only demonstrated that human languages facilitate communication
by instructions in natural languages. Nevertheless, a plethora of other forms
of communication is left unexplored. In fact, human communication originated in
gestures and oftentimes is delivered through multimodal cues, e.g. "go there"
with a pointing gesture. To bridge the gap and fill in the missing dimension of
communication in embodied agent navigation, we propose investigating the
effects of using gestures as the communicative interface instead of verbal
cues. Specifically, we develop a VR-based 3D simulation environment, named
Ges-THOR, based on AI2-THOR platform. In this virtual environment, a human
player is placed in the same virtual scene and shepherds the artificial agent
using only gestures. The agent is tasked to solve the navigation problem guided
by natural gestures with unknown semantics; we do not use any predefined
gestures due to the diversity and versatile nature of human gestures. We argue
that learning the semantics of natural gestures is mutually beneficial to
learning the navigation task--learn to communicate and communicate to learn. In
a series of experiments, we demonstrate that human gesture cues, even without
predefined semantics, improve the object-goal navigation for an embodied agent,
outperforming various state-of-the-art methods.

negative:
We are increasingly surrounded by artificially intelligent technology that
takes decisions and executes actions on our behalf. This creates a pressing
need for general means to communicate with, instruct and guide artificial
agents, with human language the most compelling means for such communication.
To achieve this in a scalable fashion, agents must be able to relate language
to the world and to actions; that is, their understanding of language must be
grounded and embodied. However, learning grounded language is a notoriously
challenging problem in artificial intelligence research. Here we present an
agent that learns to interpret language in a simulated 3D environment where it
is rewarded for the successful execution of written instructions. Trained via a
combination of reinforcement and unsupervised learning, and beginning with
minimal prior knowledge, the agent learns to relate linguistic symbols to
emergent perceptual representations of its physical surroundings and to
pertinent sequences of actions. The agent's comprehension of language extends
beyond its prior experience, enabling it to apply familiar language to
unfamiliar situations and to interpret entirely novel instructions. Moreover,
the speed with which this agent learns new words increases as its semantic
knowledge grows. This facility for generalising and bootstrapping semantic
knowledge indicates the potential of the present approach for reconciling
ambiguous natural language with the complexity of the physical world.

negative:
Cognitive Psychology and related disciplines have identified several critical
mechanisms that enable intelligent biological agents to learn to solve complex
problems. There exists pressing evidence that the cognitive mechanisms that
enable problem-solving skills in these species build on hierarchical mental
representations. Among the most promising computational approaches to provide
comparable learning-based problem-solving abilities for artificial agents and
robots is hierarchical reinforcement learning. However, so far the existing
computational approaches have not been able to equip artificial agents with
problem-solving abilities that are comparable to intelligent animals, including
human and non-human primates, crows, or octopuses. Here, we first survey the
literature in Cognitive Psychology, and related disciplines, and find that many
important mental mechanisms involve compositional abstraction, curiosity, and
forward models. We then relate these insights with contemporary hierarchical
reinforcement learning methods, and identify the key machine intelligence
approaches that realise these mechanisms. As our main result, we show that all
important cognitive mechanisms have been implemented independently in isolated
computational architectures, and there is simply a lack of approaches that
integrate them appropriately. We expect our results to guide the development of
more sophisticated cognitively inspired hierarchical methods, so that future
artificial agents achieve a problem-solving performance on the level of
intelligent animals.

negative:
Human beings, even small children, quickly become adept at figuring out how
to use applications on their mobile devices. Learning to use a new app is often
achieved via trial-and-error, accelerated by transfer of knowledge from past
experiences with like apps. The prospect of building a smarter smartphone - one
that can learn how to achieve tasks using mobile apps - is tantalizing. In this
paper we explore the use of Reinforcement Learning (RL) with the goal of
advancing this aspiration. We introduce an RL-based framework for learning to
accomplish tasks in mobile apps. RL agents are provided with states derived
from the underlying representation of on-screen elements, and rewards that are
based on progress made in the task. Agents can interact with screen elements by
tapping or typing. Our experimental results, over a number of mobile apps, show
that RL agents can learn to accomplish multi-step tasks, as well as achieve
modest generalization across different apps. More generally, we develop a
platform which addresses several engineering challenges to enable an effective
RL training environment. Our AppBuddy platform is compatible with OpenAI Gym
and includes a suite of mobile apps and benchmark tasks that supports a
diversity of RL research in the mobile app setting.

negative:
Social learning is a key component of human and animal intelligence. By
taking cues from the behavior of experts in their environment, social learners
can acquire sophisticated behavior and rapidly adapt to new circumstances. This
paper investigates whether independent reinforcement learning (RL) agents in a
multi-agent environment can learn to use social learning to improve their
performance. We find that in most circumstances, vanilla model-free RL agents
do not use social learning. We analyze the reasons for this deficiency, and
show that by imposing constraints on the training environment and introducing a
model-based auxiliary loss we are able to obtain generalized social learning
policies which enable agents to: i) discover complex skills that are not
learned from single-agent training, and ii) adapt online to novel environments
by taking cues from experts present in the new environment. In contrast, agents
trained with model-free RL or imitation learning generalize poorly and do not
succeed in the transfer tasks. By mixing multi-agent and solo training, we can
obtain agents that use social learning to gain skills that they can deploy when
alone, even out-performing agents trained alone from the start.

negative:
This paper presents TrolleyMod v1.0, an open-source platform based on the
CARLA simulator for the collection of ethical decision-making data for
autonomous vehicles. This platform is designed to facilitate experiments aiming
to observe and record human decisions and actions in high-fidelity simulations
of ethical dilemmas that occur in the context of driving. Targeting experiments
in the class of trolley problems, TrolleyMod provides a seamless approach to
creating new experimental settings and environments with the realistic
physics-engine and the high-quality graphical capabilities of CARLA and the
Unreal Engine. Also, TrolleyMod provides a straightforward interface between
the CARLA environment and Python to enable the implementation of custom
controllers, such as deep reinforcement learning agents. The results of such
experiments can be used for sociological analyses, as well as the training and
tuning of value-aligned autonomous vehicles based on social values that are
inferred from observations.

negative:
Navigating and understanding the real world remains a key challenge in
machine learning and inspires a great variety of research in areas such as
language grounding, planning, navigation and computer vision. We propose an
instruction-following task that requires all of the above, and which combines
the practicality of simulated environments with the challenges of ambiguous,
noisy real world data. StreetNav is built on top of Google Street View and
provides visually accurate environments representing real places. Agents are
given driving instructions which they must learn to interpret in order to
successfully navigate in this environment. Since humans equipped with driving
instructions can readily navigate in previously unseen cities, we set a high
bar and test our trained agents for similar cognitive capabilities. Although
deep reinforcement learning (RL) methods are frequently evaluated only on data
that closely follow the training distribution, our dataset extends to multiple
cities and has a clean train/test separation. This allows for thorough testing
of generalisation ability. This paper presents the StreetNav environment and
tasks, models that establish strong baselines, and extensive analysis of the
task and the trained agents.

negative:
We introduce a large scale crowdsourced text adventure game as a research
platform for studying grounded dialogue. In it, agents can perceive, emote, and
act whilst conducting dialogue with other agents. Models and humans can both
act as characters within the game. We describe the results of training
state-of-the-art generative and retrieval models in this setting. We show that
in addition to using past dialogue, these models are able to effectively use
the state of the underlying world to condition their predictions. In
particular, we show that grounding on the details of the local environment,
including location descriptions, and the objects (and their affordances) and
characters (and their previous actions) present within it allows better
predictions of agent behavior and dialogue. We analyze the ingredients
necessary for successful grounding in this setting, and how each of these
factors relate to agents that can talk and act successfully.

negative:
Recent advances in artificial intelligence have been driven by the presence
of increasingly realistic and complex simulated environments. However, many of
the existing environments provide either unrealistic visuals, inaccurate
physics, low task complexity, restricted agent perspective, or a limited
capacity for interaction among artificial agents. Furthermore, many platforms
lack the ability to flexibly configure the simulation, making the simulated
environment a black-box from the perspective of the learning system. In this
work, we propose a novel taxonomy of existing simulation platforms and discuss
the highest level class of general platforms which enable the development of
learning environments that are rich in visual, physical, task, and social
complexity. We argue that modern game engines are uniquely suited to act as
general platforms and as a case study examine the Unity engine and open source
Unity ML-Agents Toolkit. We then survey the research enabled by Unity and the
Unity ML-Agents Toolkit, discussing the kinds of research a flexible,
interactive and easily configurable general platform can facilitate.

negative:
Artificial intelligence systems increasingly involve continual learning to
enable flexibility in general situations that are not encountered during system
training. Human interaction with autonomous systems is broadly studied, but
research has hitherto under-explored interactions that occur while the system
is actively learning, and can noticeably change its behaviour in minutes. In
this pilot study, we investigate how the interaction between a human and a
continually learning prediction agent develops as the agent develops
competency. Additionally, we compare two different agent architectures to
assess how representational choices in agent design affect the human-agent
interaction. We develop a virtual reality environment and a time-based
prediction task wherein learned predictions from a reinforcement learning (RL)
algorithm augment human predictions. We assess how a participant's performance
and behaviour in this task differs across agent types, using both quantitative
and qualitative analyses. Our findings suggest that human trust of the system
may be influenced by early interactions with the agent, and that trust in turn
affects strategic behaviour, but limitations of the pilot study rule out any
conclusive statement. We identify trust as a key feature of interaction to
focus on when considering RL-based technologies, and make several
recommendations for modification to this study in preparation for a
larger-scale investigation. A video summary of this paper can be found at
https://youtu.be/oVYJdnBqTwQ .

negative:
Multi-Agent Reinforcement Learning (MARL) methods find optimal policies for
agents that operate in the presence of other learning agents. Central to
achieving this is how the agents coordinate. One way to coordinate is by
learning to communicate with each other. Can the agents develop a language
while learning to perform a common task? In this paper, we formulate and study
a MARL problem where cooperative agents are connected to each other via a fixed
underlying network. These agents can communicate along the edges of this
network by exchanging discrete symbols. However, the semantics of these symbols
are not predefined and, during training, the agents are required to develop a
language that helps them in accomplishing their goals. We propose a method for
training these agents using emergent communication. We demonstrate the
applicability of the proposed framework by applying it to the problem of
managing traffic controllers, where we achieve state-of-the-art performance as
compared to a number of strong baselines. More importantly, we perform a
detailed analysis of the emergent communication to show, for instance, that the
developed language is grounded and demonstrate its relationship with the
underlying network topology. To the best of our knowledge, this is the only
work that performs an in depth analysis of emergent communication in a
networked MARL setting while being applicable to a broad class of problems.

negative:
While deep reinforcement learning (deep RL) agents are effective at
maximizing rewards, it is often unclear what strategies they use to do so. In
this paper, we take a step toward explaining deep RL agents through a case
study using Atari 2600 environments. In particular, we focus on using saliency
maps to understand how an agent learns and executes a policy. We introduce a
method for generating useful saliency maps and use it to show 1) what strong
agents attend to, 2) whether agents are making decisions for the right or wrong
reasons, and 3) how agents evolve during learning. We also test our method on
non-expert human subjects and find that it improves their ability to reason
about these agents. Overall, our results show that saliency information can
provide significant insight into an RL agent's decisions and learning behavior.

negative:
Applying Deep Reinforcement Learning (DRL) to Human-Robot Cooperation (HRC)
in dynamic control problems is promising yet challenging as the robot needs to
learn the dynamics of the controlled system and dynamics of the human partner.
In existing research, the robot powered by DRL adopts coupled observation of
the environment and the human partner to learn both dynamics simultaneously.
However, such a learning strategy is limited in terms of learning efficiency
and team performance. This work proposes a novel task decomposition method with
a hierarchical reward mechanism that enables the robot to learn the
hierarchical dynamic control task separately from learning the human partner's
behavior. The method is validated with a hierarchical control task in a
simulated environment with human subject experiments. Our method also provides
insight into the design of the learning strategy for HRC. The results show that
the robot should learn the task first to achieve higher team performance and
learn the human first to achieve higher learning efficiency.

negative:
Learning agents that are not only capable of taking tests, but also
innovating is becoming a hot topic in AI. One of the most promising paths
towards this vision is multi-agent learning, where agents act as the
environment for each other, and improving each agent means proposing new
problems for others. However, existing evaluation platforms are either not
compatible with multi-agent settings, or limited to a specific game. That is,
there is not yet a general evaluation platform for research on multi-agent
intelligence. To this end, we introduce Arena, a general evaluation platform
for multi-agent intelligence with 35 games of diverse logics and
representations. Furthermore, multi-agent intelligence is still at the stage
where many problems remain unexplored. Therefore, we provide a building toolkit
for researchers to easily invent and build novel multi-agent problems from the
provided game set based on a GUI-configurable social tree and five basic
multi-agent reward schemes. Finally, we provide Python implementations of five
state-of-the-art deep multi-agent reinforcement learning baselines. Along with
the baseline implementations, we release a set of 100 best agents/teams that we
can train with different training schemes for each game, as the base for
evaluating agents with population performance. As such, the research community
can perform comparisons under a stable and uniform standard. All the
implementations and accompanied tutorials have been open-sourced for the
community at https://sites.google.com/view/arena-unity/.

negative:
Reinforcement Learning (RL) agents have great successes in solving tasks with
large observation and action spaces from limited feedback. Still, training the
agents is data-intensive and there are no guarantees that the learned behavior
is safe and does not violate rules of the environment, which has limitations
for the practical deployment in real-world scenarios. This paper discusses the
engineering of reliable agents via the integration of deep RL with
constraint-based augmentation models to guide the RL agent towards safe
behavior. Within the constraints set, the RL agent is free to adapt and
explore, such that its effectiveness to solve the given problem is not
hindered. However, once the RL agent leaves the space defined by the
constraints, the outside models can provide guidance to still work reliably. We
discuss integration points for constraint guidance within the RL process and
perform experiments on two case studies: a strictly constrained card game and a
grid world environment with additional combinatorial subgoals. Our results show
that constraint-guidance does both provide reliability improvements and safer
behavior, as well as accelerated training.

negative:
We introduce MAgent, a platform to support research and development of
many-agent reinforcement learning. Unlike previous research platforms on single
or multi-agent reinforcement learning, MAgent focuses on supporting the tasks
and the applications that require hundreds to millions of agents. Within the
interactions among a population of agents, it enables not only the study of
learning algorithms for agents' optimal polices, but more importantly, the
observation and understanding of individual agent's behaviors and social
phenomena emerging from the AI society, including communication languages,
leaderships, altruism. MAgent is highly scalable and can host up to one million
agents on a single GPU server. MAgent also provides flexible configurations for
AI researchers to design their customized environments and agents. In this
demo, we present three environments designed on MAgent and show emerged
collective intelligence by learning from scratch.

negative:
A good dialogue agent should have the ability to interact with users by both
responding to questions and by asking questions, and importantly to learn from
both types of interaction. In this work, we explore this direction by designing
a simulator and a set of synthetic tasks in the movie domain that allow such
interactions between a learner and a teacher. We investigate how a learner can
benefit from asking questions in both offline and online reinforcement learning
settings, and demonstrate that the learner improves when asking questions.
Finally, real experiments with Mechanical Turk validate the approach. Our work
represents a first step in developing such end-to-end learned interactive
dialogue agents.

negative:
Humans and other intelligent animals evolved highly sophisticated perception
systems that combine multiple sensory modalities. On the other hand,
state-of-the-art artificial agents rely mostly on visual inputs or structured
low-dimensional observations provided by instrumented environments. Learning to
act based on combined visual and auditory inputs is still a new topic of
research that has not been explored beyond simple scenarios. To facilitate
progress in this area we introduce a new version of VizDoom simulator to create
a highly efficient learning environment that provides raw audio observations.
We study the performance of different model architectures in a series of tasks
that require the agent to recognize sounds and execute instructions given in
natural language. Finally, we train our agent to play the full game of Doom and
find that it can consistently defeat a traditional vision-based adversary. We
are currently in the process of merging the augmented simulator with the main
ViZDoom code repository. Video demonstrations and experiment code can be found
at https://sites.google.com/view/sound-rl.

negative:
Inferring other agents' mental states such as their knowledge, beliefs and
intentions is thought to be essential for effective interactions with other
agents. Recently, multiagent systems trained via deep reinforcement learning
have been shown to succeed in solving different tasks, but it remains unclear
how each agent modeled or represented other agents in their environment. In
this work we test whether deep reinforcement learning agents explicitly
represent other agents' intentions (their specific aims or goals) during a task
in which the agents had to coordinate the covering of different spots in a 2D
environment. In particular, we tracked over time the performance of a linear
decoder trained to predict the final goal of all agents from the hidden state
of each agent's neural network controller. We observed that the hidden layers
of agents represented explicit information about other agents' goals, i.e. the
target landmark they ended up covering. We also performed a series of
experiments, in which some agents were replaced by others with fixed goals, to
test the level of generalization of the trained agents. We noticed that during
the training phase the agents developed a differential preference for each
goal, which hindered generalization. To alleviate the above problem, we propose
simple changes to the MADDPG training algorithm which leads to better
generalization against unseen agents. We believe that training protocols
promoting more active intention reading mechanisms, e.g. by preventing simple
symmetry-breaking solutions, is a promising direction towards achieving a more
robust generalization in different cooperative and competitive tasks.

negative:
Behavior Trees are commonly used to model agents for robotics and games,
where constrained behaviors must be designed by human experts in order to
guarantee that these agents will execute a specific chain of actions given a
specific set of perceptions. In such application areas, learning is a desirable
feature to provide agents with the ability to adapt and improve interactions
with humans and environment, but often discarded due to its unreliability. In
this paper, we propose a framework that uses Reinforcement Learning nodes as
part of Behavior Trees to address the problem of adding learning capabilities
in constrained agents. We show how this framework relates to Options in
Hierarchical Reinforcement Learning, ensuring convergence of nested learning
nodes, and we empirically show that the learning nodes do not affect the
execution of other nodes in the tree.

negative:
As AI continues to advance, human-AI teams are inevitable. However, progress
in AI is routinely measured in isolation, without a human in the loop. It is
crucial to benchmark progress in AI, not just in isolation, but also in terms
of how it translates to helping humans perform certain tasks, i.e., the
performance of human-AI teams.
  In this work, we design a cooperative game - GuessWhich - to measure human-AI
team performance in the specific context of the AI being a visual
conversational agent. GuessWhich involves live interaction between the human
and the AI. The AI, which we call ALICE, is provided an image which is unseen
by the human. Following a brief description of the image, the human questions
ALICE about this secret image to identify it from a fixed pool of images.
  We measure performance of the human-ALICE team by the number of guesses it
takes the human to correctly identify the secret image after a fixed number of
dialog rounds with ALICE. We compare performance of the human-ALICE teams for
two versions of ALICE. Our human studies suggest a counterintuitive trend -
that while AI literature shows that one version outperforms the other when
paired with an AI questioner bot, we find that this improvement in AI-AI
performance does not translate to improved human-AI performance. This suggests
a mismatch between benchmarking of AI in isolation and in the context of
human-AI teams.

negative:
Deep reinforcement learning augments the reinforcement learning framework and
utilizes the powerful representation of deep neural networks. Recent works have
demonstrated the remarkable successes of deep reinforcement learning in various
domains including finance, medicine, healthcare, video games, robotics, and
computer vision. In this work, we provide a detailed review of recent and
state-of-the-art research advances of deep reinforcement learning in computer
vision. We start with comprehending the theories of deep learning,
reinforcement learning, and deep reinforcement learning. We then propose a
categorization of deep reinforcement learning methodologies and discuss their
advantages and limitations. In particular, we divide deep reinforcement
learning into seven main categories according to their applications in computer
vision, i.e. (i)landmark localization (ii) object detection; (iii) object
tracking; (iv) registration on both 2D image and 3D image volumetric data (v)
image segmentation; (vi) videos analysis; and (vii) other applications. Each of
these categories is further analyzed with reinforcement learning techniques,
network design, and performance. Moreover, we provide a comprehensive analysis
of the existing publicly available datasets and examine source code
availability. Finally, we present some open issues and discuss future research
directions on deep reinforcement learning in computer vision

negative:
Reinforcement learning (RL) agents in human-computer interactions
applications require repeated user interactions before they can perform well.
To address this "cold start" problem, we propose a novel approach of using
cognitive models to pre-train RL agents before they are applied to real users.
After briefly reviewing relevant cognitive models, we present our general
methodological approach, followed by two case studies from our previous and
ongoing projects. We hope this position paper stimulates conversations between
RL, HCI, and cognitive science researchers in order to explore the full
potential of the approach.

negative:
Evidence that visual communication preceded written language and provided a
basis for it goes back to prehistory, in forms such as cave and rock paintings
depicting traces of our distant ancestors. Emergent communication research has
sought to explore how agents can learn to communicate in order to
collaboratively solve tasks. Existing research has focused on language, with a
learned communication channel transmitting sequences of discrete tokens between
the agents. In this work, we explore a visual communication channel between
agents that are allowed to draw with simple strokes. Our agents are
parameterised by deep neural networks, and the drawing procedure is
differentiable, allowing for end-to-end training. In the framework of a
referential communication game, we demonstrate that agents can not only
successfully learn to communicate by drawing, but with appropriate inductive
biases, can do so in a fashion that humans can interpret. We hope to encourage
future research to consider visual communication as a more flexible and
directly interpretable alternative of training collaborative agents.

negative:
A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. This setting nevertheless integrates a
number of the central challenges of artificial intelligence (AI) research:
complex visual perception and goal-directed physical control, grounded language
comprehension and production, and multi-agent social interaction. To build
agents that can robustly interact with humans, we would ideally train them
while they interact with humans. However, this is presently impractical.
Therefore, we approximate the role of the human with another learned agent, and
use ideas from inverse reinforcement learning to reduce the disparities between
human-human and agent-agent interactive behaviour. Rigorously evaluating our
agents poses a great challenge, so we develop a variety of behavioural tests,
including evaluation by humans who watch videos of agents or interact directly
with them. These evaluations convincingly demonstrate that interactive training
and auxiliary losses improve agent behaviour beyond what is achieved by
supervised learning of actions alone. Further, we demonstrate that agent
capabilities generalise beyond literal experiences in the dataset. Finally, we
train evaluation models whose ratings of agents agree well with human
judgement, thus permitting the evaluation of new agent models without
additional effort. Taken together, our results in this virtual environment
provide evidence that large-scale human behavioural imitation is a promising
tool to create intelligent, interactive agents, and the challenge of reliably
evaluating such agents is possible to surmount.

negative:
Deep reinforcement learning has learned to play many games well, but failed
on others. To better characterize the modes and reasons of failure of deep
reinforcement learners, we test the widely used Asynchronous Actor-Critic (A2C)
algorithm on four deceptive games, which are specially designed to provide
challenges to game-playing agents. These games are implemented in the General
Video Game AI framework, which allows us to compare the behavior of
reinforcement learning-based agents with planning agents based on tree search.
We find that several of these games reliably deceive deep reinforcement
learners, and that the resulting behavior highlights the shortcomings of the
learning algorithm. The particular ways in which agents fail differ from how
planning-based agents fail, further illuminating the character of these
algorithms. We propose an initial typology of deceptions which could help us
better understand pitfalls and failure modes of (deep) reinforcement learning.

negative:
AI systems have seen dramatic advancement in recent years, bringing many
applications that pervade our everyday life. However, we are still mostly
seeing instances of narrow AI: many of these recent developments are typically
focused on a very limited set of competencies and goals, e.g., image
interpretation, natural language processing, classification, prediction, and
many others. Moreover, while these successes can be accredited to improved
algorithms and techniques, they are also tightly linked to the availability of
huge datasets and computational power. State-of-the-art AI still lacks many
capabilities that would naturally be included in a notion of (human)
intelligence.
  We argue that a better study of the mechanisms that allow humans to have
these capabilities can help us understand how to imbue AI systems with these
competencies. We focus especially on D. Kahneman's theory of thinking fast and
slow, and we propose a multi-agent AI architecture where incoming problems are
solved by either system 1 (or "fast") agents, that react by exploiting only
past experience, or by system 2 (or "slow") agents, that are deliberately
activated when there is the need to reason and search for optimal solutions
beyond what is expected from the system 1 agent. Both kinds of agents are
supported by a model of the world, containing domain knowledge about the
environment, and a model of "self", containing information about past actions
of the system and solvers' skills.

negative:
Advances in reinforcement learning (RL) have resulted in recent breakthroughs
in the application of artificial intelligence (AI) across many different
domains. An emerging landscape of development environments is making powerful
RL techniques more accessible for a growing community of researchers. However,
most existing frameworks do not directly address the problem of learning in
complex operating environments, such as dense urban settings or defense-related
scenarios, that incorporate distributed, heterogeneous teams of agents. To help
enable AI research for this important class of applications, we introduce the
AI Arena: a scalable framework with flexible abstractions for distributed
multi-agent reinforcement learning. The AI Arena extends the OpenAI Gym
interface to allow greater flexibility in learning control policies across
multiple agents with heterogeneous learning strategies and localized views of
the environment. To illustrate the utility of our framework, we present
experimental results that demonstrate performance gains due to a distributed
multi-agent learning approach over commonly-used RL techniques in several
different learning environments.

negative:
When deployed, AI agents will encounter problems that are beyond their
autonomous problem-solving capabilities. Leveraging human assistance can help
agents overcome their inherent limitations and robustly cope with unfamiliar
situations. We present a general interactive framework that enables an agent to
request and interpret rich, contextually useful information from an assistant
that has knowledge about the task and the environment. We demonstrate the
practicality of our framework on a simulated human-assisted navigation problem.
Aided with an assistance-requesting policy learned by our method, a navigation
agent achieves up to a 7x improvement in success rate on tasks that take place
in previously unseen environments, compared to fully autonomous behavior. We
show that the agent can take advantage of different types of information
depending on the context, and analyze the benefits and challenges of learning
the assistance-requesting policy when the assistant can recursively decompose
tasks into subtasks.

negative:
Deep reinforcement learning has the potential to train robots to perform
complex tasks in the real world without requiring accurate models of the robot
or its environment. A practical approach is to train agents in simulation, and
then transfer them to the real world. One popular method for achieving
transferability is to use domain randomisation, which involves randomly
perturbing various aspects of a simulated environment in order to make trained
agents robust to the reality gap. However, less work has gone into
understanding such agents - which are deployed in the real world - beyond task
performance. In this work we examine such agents, through qualitative and
quantitative comparisons between agents trained with and without visual domain
randomisation. We train agents for Fetch and Jaco robots on a visuomotor
control task and evaluate how well they generalise using different testing
conditions. Finally, we investigate the internals of the trained agents by
using a suite of interpretability techniques. Our results show that the primary
outcome of domain randomisation is more robust, entangled representations,
accompanied with larger weights with greater spatial structure; moreover, the
types of changes are heavily influenced by the task setup and presence of
additional proprioceptive inputs. Additionally, we demonstrate that our domain
randomised agents require higher sample complexity, can overfit and more
heavily rely on recurrent processing. Furthermore, even with an improved
saliency method introduced in this work, we show that qualitative studies may
not always correspond with quantitative measures, necessitating the combination
of inspection tools in order to provide sufficient insights into the behaviour
of trained agents.

negative:
Movement coordination in human ensembles has been studied little in the
current literature. In the existing experimental works, situations where all
subjects are connected with each other through direct visual and auditory
coupling, and social interaction affects their coordination, have been
investigated. Here, we study coordination in human ensembles via a novel
computer-based set-up that enables individuals to coordinate each other's
motion from a distance so as to minimize the influence of social interaction.
The proposed platform makes it possible to implement different visual
interaction patterns among the players, so that participants take into
consideration the motion of a designated subset of the others. This allows the
evaluation of the exclusive effects on coordination of the structure of
interconnections among the players and their own dynamics. Our set-up enables
also the deployment of virtual players to investigate dyadic interaction
between a human and a virtual agent, as well as group synchronization in mixed
teams of human and virtual agents. We use this novel set-up to study
coordination both in dyads and in groups over different structures of
interconnections, with and without virtual agents. We find that, in dual
interaction, virtual players manage to interact with participants in a
human-like fashion, thus confirming findings in previous work. We also observe
that, in group interaction, the level of coordination among humans in the
absence of direct visual and auditory coupling depends on the structure of
interconnections among participants. This confirms, as recently suggested in
the literature, that different coordination levels are achieved over diverse
visual pairings in the presence and in the absence of social interaction. We
present preliminary experimental results on the effect on group coordination of
deploying virtual computer agents in the human ensemble.

negative:
This paper presents the first two editions of Visual Doom AI Competition,
held in 2016 and 2017. The challenge was to create bots that compete in a
multi-player deathmatch in a first-person shooter (FPS) game, Doom. The bots
had to make their decisions based solely on visual information, i.e., a raw
screen buffer. To play well, the bots needed to understand their surroundings,
navigate, explore, and handle the opponents at the same time. These aspects,
together with the competitive multi-agent aspect of the game, make the
competition a unique platform for evaluating the state of the art reinforcement
learning algorithms. The paper discusses the rules, solutions, results, and
statistics that give insight into the agents' behaviors. Best-performing agents
are described in more detail. The results of the competition lead to the
conclusion that, although reinforcement learning can produce capable Doom bots,
they still are not yet able to successfully compete against humans in this
game. The paper also revisits the ViZDoom environment, which is a flexible,
easy to use, and efficient 3D platform for research for vision-based
reinforcement learning, based on a well-recognized first-person perspective
game Doom.

negative:
Even with impressive advances in application-specific models, we still lack
knowledge about how to build a model that can learn in a human-like way and do
multiple tasks. To learn in a human-like way, we need to provide a diverse
experience that is comparable to humans. In this paper, we introduce our
ongoing effort to build a simulated environment for developmental robotics
(SEDRo). SEDRo provides diverse human experiences ranging from those of a fetus
to a 12th-month-old. A series of simulated tests based on developmental
psychology will be used to evaluate the progress of a learning model. We
anticipate SEDRo to lower the cost of entry and facilitate research in the
developmental robotics community.

negative:
What might sound like the beginning of a joke has become an attractive
prospect for many cognitive scientists: the use of deep neural network models
(DNNs) as models of human behavior in perceptual and cognitive tasks. Although
DNNs have taken over machine learning, attempts to use them as models of human
behavior are still in the early stages. Can they become a versatile model class
in the cognitive scientist's toolbox? We first argue why DNNs have the
potential to be interesting models of human behavior. We then discuss how that
potential can be more fully realized. On the one hand, we argue that the cycle
of training, testing, and revising DNNs needs to be revisited through the lens
of the cognitive scientist's goals. Specifically, we argue that methods for
assessing the goodness of fit between DNN models and human behavior have to
date been impoverished. On the other hand, cognitive science might have to
start using more complex tasks (including richer stimulus spaces), but doing so
might be beneficial for DNN-independent reasons as well. Finally, we highlight
avenues where traditional cognitive process models and DNNs may show productive
synergy.

negative:
The measurement of time is central to intelligent behavior. We know that both
animals and artificial agents can successfully use temporal dependencies to
select actions. In artificial agents, little work has directly addressed (1)
which architectural components are necessary for successful development of this
ability, (2) how this timing ability comes to be represented in the units and
actions of the agent, and (3) whether the resulting behavior of the system
converges on solutions similar to those of biology. Here we studied interval
timing abilities in deep reinforcement learning agents trained end-to-end on an
interval reproduction paradigm inspired by experimental literature on
mechanisms of timing. We characterize the strategies developed by recurrent and
feedforward agents, which both succeed at temporal reproduction using distinct
mechanisms, some of which bear specific and intriguing similarities to
biological systems. These findings advance our understanding of how agents come
to represent time, and they highlight the value of experimentally inspired
approaches to characterizing agent abilities.

negative:
Building intelligent agents that can communicate with and learn from humans
in natural language is of great value. Supervised language learning is limited
by the ability of capturing mainly the statistics of training data, and is
hardly adaptive to new scenarios or flexible for acquiring new knowledge
without inefficient retraining or catastrophic forgetting. We highlight the
perspective that conversational interaction serves as a natural interface both
for language learning and for novel knowledge acquisition and propose a joint
imitation and reinforcement approach for grounded language learning through an
interactive conversational game. The agent trained with this approach is able
to actively acquire information by asking questions about novel objects and use
the just-learned knowledge in subsequent conversations in a one-shot fashion.
Results compared with other methods verified the effectiveness of the proposed
approach.

negative:
Interactive Machine Learning is concerned with creating systems that operate
in environments alongside humans to achieve a task. A typical use is to extend
or amplify the capabilities of a human in cognitive or physical ways, requiring
the machine to adapt to the users' intentions and preferences. Often, this
takes the form of a human operator providing some type of feedback to the user,
which can be explicit feedback, implicit feedback, or a combination of both.
Explicit feedback, such as through a mouse click, carries a high cognitive
load. The focus of this study is to extend the current state of the art in
interactive machine learning by demonstrating that agents can learn a human
user's behavior and adapt to preferences with a reduced amount of explicit
human feedback in a mixed feedback setting. The learning agent perceives a
value of its own behavior from hand gestures given via a spatial interface.
This feedback mechanism is termed Spatial Interface Valuing. This method is
evaluated experimentally in a simulated environment for a grasping task using a
robotic arm with variable grip settings. Preliminary results indicate that
learning agents using spatial interface valuing can learn a value function
mapping spatial gestures to expected future rewards much more quickly as
compared to those same agents just receiving explicit feedback, demonstrating
that an agent perceiving feedback from a human user via a spatial interface can
serve as an effective complement to existing approaches.

negative:
Future robots should follow human social norms in order to be useful and
accepted in human society. In this paper, we leverage already existing social
knowledge in human societies by capturing it in our framework through the
notion of social norms. We show how norms can be used to guide a reinforcement
learning agent towards achieving normative behavior and apply the same set of
norms over different domains. Thus, we are able to: (1) provide a way to
intuitively encode social knowledge (through norms); (2) guide learning towards
normative behaviors (through an automatic norm reward system); and (3) achieve
a transfer of learning by abstracting policies; Finally, (4) the method is not
dependent on a particular RL algorithm. We show how our approach can be seen as
a means to achieve abstract representation and learn procedural knowledge based
on the declarative semantics of norms and discuss possible implications of this
in some areas of cognitive science.

negative:
We present a method of endowing agents in an agent-based model (ABM) with
sophisticated cognitive capabilities and a naturally tunable level of
intelligence. Often, ABMs use random behavior or greedy algorithms for
maximizing objectives (such as a predator always chasing after the closest
prey). However, random behavior is too simplistic in many circumstances and
greedy algorithms, as well as classic AI planning techniques, can be brittle in
the context of the unpredictable and emergent situations in which agents may
find themselves. Our method, called agent-centric Monte Carlo cognition
(ACMCC), centers around using a separate agent-based model to represent the
agents' cognition. This model is then used by the agents in the primary model
to predict the outcomes of their actions, and thus guide their behavior. To
that end, we have implemented our method in the NetLogo agent-based modeling
platform, using the recently released LevelSpace extension, which we developed
to allow NetLogo models to interact with other NetLogo models. As an
illustrative example, we extend the Wolf Sheep Predation model (included with
NetLogo) by using ACMCC to guide animal behavior, and analyze the impact on
agent performance and model dynamics. We find that ACMCC provides a reliable
and understandable method of controlling agent intelligence, and has a large
impact on agent performance and model dynamics even at low settings.

negative:
Multi-agent reinforcement learning (MARL) extends (single-agent)
reinforcement learning (RL) by introducing additional agents and (potentially)
partial observability of the environment. Consequently, algorithms for solving
MARL problems incorporate various extensions beyond traditional RL methods,
such as a learned communication protocol between cooperative agents that
enables exchange of private information or adaptive modeling of opponents in
competitive settings. One popular algorithmic construct is a memory mechanism
such that an agent's decisions can depend not only upon the current state but
also upon the history of observed states and actions. In this paper, we study
how a memory mechanism can be useful in environments with different properties,
such as observability, internality and presence of a communication channel.
Using both prior work and new experiments, we show that a memory mechanism is
helpful when learning agents need to model other agents and/or when
communication is constrained in some way; however we must to be cautious of
agents achieving effective memoryfulness through other means.

negative:
We present Coevo, an online platform that allows both humans and artificial
agents to design shapes that solve different tasks. Our goal is to explore
common shared design tools that can be used by humans and artificial agents in
a context of creation. This approach can provide a better knowledge transfer
and interaction with artificial agents since a common language of design is
defined. In this paper, we outline the main components of this platform and
discuss the definition of a human-centered language to enhance human-AI
collaboration in co-creation scenarios.

negative:
Deep reinforcement learning (deep RL) has been successful in learning
sophisticated behaviors automatically; however, the learning process requires a
huge number of trials. In contrast, animals can learn new tasks in just a few
trials, benefiting from their prior knowledge about the world. This paper seeks
to bridge this gap. Rather than designing a "fast" reinforcement learning
algorithm, we propose to represent it as a recurrent neural network (RNN) and
learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in
the weights of the RNN, which are learned slowly through a general-purpose
("slow") RL algorithm. The RNN receives all information a typical RL algorithm
would receive, including observations, actions, rewards, and termination flags;
and it retains its state across episodes in a given Markov Decision Process
(MDP). The activations of the RNN store the state of the "fast" RL algorithm on
the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both
small-scale and large-scale problems. On the small-scale side, we train it to
solve randomly generated multi-arm bandit problems and finite MDPs. After
RL$^2$ is trained, its performance on new MDPs is close to human-designed
algorithms with optimality guarantees. On the large-scale side, we test RL$^2$
on a vision-based navigation task and show that it scales up to
high-dimensional problems.

negative:
My overarching research goal is to provide robots with perceptional abilities
that allow interactions with humans in a human-like manner. To develop these
perceptional abilities, I believe that it is useful to study the principles of
the human visual system. I use these principles to develop new computer vision
algorithms and validate their effectiveness in intelligent robotic systems. I
am enthusiastic about this approach as it offers the dual benefit of uncovering
principles inherent in the human visual system, as well as applying these
principles to its artificial counterpart. Fig. 1 contains a depiction of my
research.

negative:
For machine agents to successfully interact with humans in real-world
settings, they will need to develop an understanding of human mental life.
Intuitive psychology, the ability to reason about hidden mental variables that
drive observable actions, comes naturally to people: even pre-verbal infants
can tell agents from objects, expecting agents to act efficiently to achieve
goals given constraints. Despite recent interest in machine agents that reason
about other agents, it is not clear if such agents learn or hold the core
psychology principles that drive human reasoning. Inspired by cognitive
development studies on intuitive psychology, we present a benchmark consisting
of a large dataset of procedurally generated 3D animations, AGENT (Action,
Goal, Efficiency, coNstraint, uTility), structured around four scenarios (goal
preferences, action efficiency, unobserved constraints, and cost-reward
trade-offs) that probe key concepts of core intuitive psychology. We validate
AGENT with human-ratings, propose an evaluation protocol emphasizing
generalization, and compare two strong baselines built on Bayesian inverse
planning and a Theory of Mind neural network. Our results suggest that to pass
the designed tests of core intuitive psychology at human levels, a model must
acquire or have built-in representations of how agents plan, combining utility
computations and core knowledge of objects and physics.

negative:
In this paper, we argue that the future of Artificial Intelligence research
resides in two keywords: integration and embodiment. We support this claim by
analyzing the recent advances of the field. Regarding integration, we note that
the most impactful recent contributions have been made possible through the
integration of recent Machine Learning methods (based in particular on Deep
Learning and Recurrent Neural Networks) with more traditional ones (e.g.
Monte-Carlo tree search, goal babbling exploration or addressable memory
systems). Regarding embodiment, we note that the traditional benchmark tasks
(e.g. visual classification or board games) are becoming obsolete as
state-of-the-art learning algorithms approach or even surpass human performance
in most of them, having recently encouraged the development of first-person 3D
game platforms embedding realistic physics. Building upon this analysis, we
first propose an embodied cognitive architecture integrating heterogenous
sub-fields of Artificial Intelligence into a unified framework. We demonstrate
the utility of our approach by showing how major contributions of the field can
be expressed within the proposed framework. We then claim that benchmarking
environments need to reproduce ecologically-valid conditions for bootstrapping
the acquisition of increasingly complex cognitive skills through the concept of
a cognitive arms race between embodied agents.

negative:
Deep artificial neural networks, trained with labeled data sets are widely
used in numerous vision and robotics applications today. In terms of AI, these
are called reflex models, referring to the fact that they do not self-evolve or
actively adapt to environmental changes. As demand for intelligent robot
control expands to many high level tasks, reinforcement learning and state
based models play an increasingly important role. Herein, in computer vision
and robotics domain, we study a novel approach to add reinforcement controls
onto the image recognition reflex models to attain better overall performance,
specifically to a wider environment range beyond what is expected of the task
reflex models. Follow a common infrastructure with environment sensing and AI
based modeling of self-adaptive agents, we implement multiple types of AI
control agents. To the end, we provide comparative results of these agents with
baseline, and an insightful analysis of their benefit to improve overall image
recognition performance in real world.

negative:
Designing human-centered AI-driven applications require deep understandings
of how people develop mental models of AI. Currently, we have little knowledge
of this process and limited tools to study it. This paper presents the position
that AI-based games, particularly the player-AI interaction component, offer an
ideal domain to study the process in which mental models evolve. We present a
case study to illustrate the benefits of our approach for explainable AI.

negative:
Recent progress in artificial intelligence (AI) has renewed interest in
building systems that learn and think like people. Many advances have come from
using deep neural networks trained end-to-end in tasks such as object
recognition, video games, and board games, achieving performance that equals or
even beats humans in some respects. Despite their biological inspiration and
performance achievements, these systems differ from human intelligence in
crucial ways. We review progress in cognitive science suggesting that truly
human-like learning and thinking machines will have to reach beyond current
engineering trends in both what they learn, and how they learn it.
Specifically, we argue that these machines should (a) build causal models of
the world that support explanation and understanding, rather than merely
solving pattern recognition problems; (b) ground learning in intuitive theories
of physics and psychology, to support and enrich the knowledge that is learned;
and (c) harness compositionality and learning-to-learn to rapidly acquire and
generalize knowledge to new tasks and situations. We suggest concrete
challenges and promising routes towards these goals that can combine the
strengths of recent neural network advances with more structured cognitive
models.

negative:
The objective of a reinforcement learning agent is to behave so as to
maximise the sum of a suitable scalar function of state: the reward. These
rewards are typically given and immutable. In this paper, we instead consider
the proposition that the reward function itself can be a good locus of learned
knowledge. To investigate this, we propose a scalable meta-gradient framework
for learning useful intrinsic reward functions across multiple lifetimes of
experience. Through several proof-of-concept experiments, we show that it is
feasible to learn and capture knowledge about long-term exploration and
exploitation into a reward function. Furthermore, we show that unlike policy
transfer methods that capture "how" the agent should behave, the learned reward
functions can generalise to other kinds of agents and to changes in the
dynamics of the environment by capturing "what" the agent should strive to do.

negative:
Recently, there have been several high-profile achievements of agents
learning to play games against humans and beat them. In this paper, we study
the problem of training intelligent agents in service of game development.
Unlike the agents built to "beat the game", our agents aim to produce
human-like behavior to help with game evaluation and balancing. We discuss two
fundamental metrics based on which we measure the human-likeness of agents,
namely skill and style, which are multi-faceted concepts with practical
implications outlined in this paper. We report four case studies in which the
style and skill requirements inform the choice of algorithms and metrics used
to train agents; ranging from A* search to state-of-the-art deep reinforcement
learning. We, further, show that the learning potential of state-of-the-art
deep RL models does not seamlessly transfer from the benchmark environments to
target ones without heavily tuning their hyperparameters, leading to linear
scaling of the engineering efforts and computational cost with the number of
target domains.

negative:
We present a machine learning framework for multi-agent systems to learn both
the optimal policy for maximizing the rewards and the encoding of the high
dimensional visual observation. The encoding is useful for sharing local visual
observations with other agents under communication resource constraints. The
actor-encoder encodes the raw images and chooses an action based on local
observations and messages sent by the other agents. The machine learning agent
generates not only an actuator command to the physical device, but also a
communication message to the other agents. We formulate a reinforcement
learning problem, which extends the action space to consider the communication
action as well. The feasibility of the reinforcement learning framework is
demonstrated using a 3D simulation environment with two collaborating agents.
The environment provides realistic visual observations to be used and shared
between the two agents.

negative:
We propose an explainable reinforcement learning (XRL) framework that
analyzes an agent's history of interaction with the environment to extract
interestingness elements that help explain its behavior. The framework relies
on data readily available from standard RL algorithms, augmented with data that
can easily be collected by the agent while learning. We describe how to create
visual summaries of an agent's behavior in the form of short video-clips
highlighting key interaction moments, based on the proposed elements. We also
report on a user study where we evaluated the ability of humans to correctly
perceive the aptitude of agents with different characteristics, including their
capabilities and limitations, given visual summaries automatically generated by
our framework. The results show that the diversity of aspects captured by the
different interestingness elements is crucial to help humans correctly
understand an agent's strengths and limitations in performing a task, and
determine when it might need adjustments to improve its performance.

negative:
One of the long-term goals of artificial intelligence is to build an agent
that can communicate intelligently with human in natural language. Most
existing work on natural language learning relies heavily on training over a
pre-collected dataset with annotated labels, leading to an agent that
essentially captures the statistics of the fixed external training data. As the
training data is essentially a static snapshot representation of the knowledge
from the annotator, the agent trained this way is limited in adaptiveness and
generalization of its behavior. Moreover, this is very different from the
language learning process of humans, where language is acquired during
communication by taking speaking action and learning from the consequences of
speaking action in an interactive manner. This paper presents an interactive
setting for grounded natural language learning, where an agent learns natural
language by interacting with a teacher and learning from feedback, thus
learning and improving language skills while taking part in the conversation.
To achieve this goal, we propose a model which incorporates both imitation and
reinforcement by leveraging jointly sentence and reward feedbacks from the
teacher. Experiments are conducted to validate the effectiveness of the
proposed approach.

negative:
This paper assumes the hypothesis that human learning is perception based,
and consequently, the learning process and perceptions should not be
represented and investigated independently or modeled in different simulation
spaces. In order to keep the analogy between the artificial and human learning,
the former is assumed here as being based on the artificial perception. Hence,
instead of choosing to apply or develop a Computational Theory of (human)
Perceptions, we choose to mirror the human perceptions in a numeric
(computational) space as artificial perceptions and to analyze the
interdependence between artificial learning and artificial perception in the
same numeric space, using one of the simplest tools of Artificial Intelligence
and Soft Computing, namely the perceptrons. As practical applications, we
choose to work around two examples: Optical Character Recognition and Iris
Recognition. In both cases a simple Turing test shows that artificial
perceptions of the difference between two characters and between two irides are
fuzzy, whereas the corresponding human perceptions are, in fact, crisp.

negative:
We study the task of embodied visual active learning, where an agent is set
to explore a 3d environment with the goal to acquire visual scene understanding
by actively selecting views for which to request annotation. While accurate on
some benchmarks, today's deep visual recognition pipelines tend to not
generalize well in certain real-world scenarios, or for unusual viewpoints.
Robotic perception, in turn, requires the capability to refine the recognition
capabilities for the conditions where the mobile system operates, including
cluttered indoor environments or poor illumination. This motivates the proposed
task, where an agent is placed in a novel environment with the objective of
improving its visual recognition capability. To study embodied visual active
learning, we develop a battery of agents - both learnt and pre-specified - and
with different levels of knowledge of the environment. The agents are equipped
with a semantic segmentation network and seek to acquire informative views,
move and explore in order to propagate annotations in the neighbourhood of
those views, then refine the underlying segmentation network by online
retraining. The trainable method uses deep reinforcement learning with a reward
function that balances two competing objectives: task performance, represented
as visual recognition accuracy, which requires exploring the environment, and
the necessary amount of annotated data requested during active exploration. We
extensively evaluate the proposed models using the photorealistic Matterport3D
simulator and show that a fully learnt method outperforms comparable
pre-specified counterparts, even when requesting fewer annotations.

negative:
There has been an emerging paradigm shift from the era of "internet AI" to
"embodied AI", where AI algorithms and agents no longer learn from datasets of
images, videos or text curated primarily from the internet. Instead, they learn
through interactions with their environments from an egocentric perception
similar to humans. Consequently, there has been substantial growth in the
demand for embodied AI simulators to support various embodied AI research
tasks. This growing interest in embodied AI is beneficial to the greater
pursuit of Artificial General Intelligence (AGI), but there has not been a
contemporary and comprehensive survey of this field. This paper aims to provide
an encyclopedic survey for the field of embodied AI, from its simulators to its
research. By evaluating nine current embodied AI simulators with our proposed
seven features, this paper aims to understand the simulators in their provision
for use in embodied AI research and their limitations. Lastly, this paper
surveys the three main research tasks in embodied AI -- visual exploration,
visual navigation and embodied question answering (QA), covering the
state-of-the-art approaches, evaluation metrics and datasets. Finally, with the
new insights revealed through surveying the field, the paper will provide
suggestions for simulator-for-task selections and recommendations for the
future directions of the field.

negative:
Many tasks in AI require the collaboration of multiple agents. Typically, the
communication protocol between agents is manually specified and not altered
during training. In this paper we explore a simple neural model, called
CommNet, that uses continuous communication for fully cooperative tasks. The
model consists of multiple agents and the communication between them is learned
alongside their policy. We apply this model to a diverse set of tasks,
demonstrating the ability of the agents to learn to communicate amongst
themselves, yielding improved performance over non-communicative agents and
baselines. In some cases, it is possible to interpret the language devised by
the agents, revealing simple but effective strategies for solving the task at
hand.

negative:
Recent progress in artificial intelligence through reinforcement learning
(RL) has shown great success on increasingly complex single-agent environments
and two-player turn-based games. However, the real-world contains multiple
agents, each learning and acting independently to cooperate and compete with
other agents, and environments reflecting this degree of complexity remain an
open challenge. In this work, we demonstrate for the first time that an agent
can achieve human-level in a popular 3D multiplayer first-person video game,
Quake III Arena Capture the Flag, using only pixels and game points as input.
These results were achieved by a novel two-tier optimisation process in which a
population of independent RL agents are trained concurrently from thousands of
parallel matches with agents playing in teams together and against each other
on randomly generated environments. Each agent in the population learns its own
internal reward signal to complement the sparse delayed reward from winning,
and selects actions using a novel temporally hierarchical representation that
enables the agent to reason at multiple timescales. During game-play, these
agents display human-like behaviours such as navigating, following, and
defending based on a rich learned representation that is shown to encode
high-level game knowledge. In an extensive tournament-style evaluation the
trained agents exceeded the win-rate of strong human players both as teammates
and opponents, and proved far stronger than existing state-of-the-art agents.
These results demonstrate a significant jump in the capabilities of artificial
agents, bringing us closer to the goal of human-level intelligence.

negative:
In this work we create agents that can perform well beyond a single,
individual task, that exhibit much wider generalisation of behaviour to a
massive, rich space of challenges. We define a universe of tasks within an
environment domain and demonstrate the ability to train agents that are
generally capable across this vast space and beyond. The environment is
natively multi-agent, spanning the continuum of competitive, cooperative, and
independent games, which are situated within procedurally generated physical 3D
worlds. The resulting space is exceptionally diverse in terms of the challenges
posed to agents, and as such, even measuring the learning progress of an agent
is an open research problem. We propose an iterative notion of improvement
between successive generations of agents, rather than seeking to maximise a
singular objective, allowing us to quantify progress despite tasks being
incomparable in terms of achievable rewards. We show that through constructing
an open-ended learning process, which dynamically changes the training task
distributions and training objectives such that the agent never stops learning,
we achieve consistent learning of new behaviours. The resulting agent is able
to score reward in every one of our humanly solvable evaluation levels, with
behaviour generalising to many held-out points in the universe of tasks.
Examples of this zero-shot generalisation include good performance on Hide and
Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks
we characterise the behaviour of our agent, and find interesting emergent
heuristic behaviours such as trial-and-error experimentation, simple tool use,
option switching, and cooperation. Finally, we demonstrate that the general
capabilities of this agent could unlock larger scale transfer of behaviour
through cheap finetuning.

negative:
Humans sometimes choose actions that they themselves can identify as
sub-optimal, or wrong, even in the absence of additional information. How is
this possible? We present an algorithmic theory of metacognition based on a
well-understood trade-off in reinforcement learning (RL) between value-based RL
and policy-based RL. To the cognitive (neuro)science community, our theory
answers the outstanding question of why information can be used for error
detection but not for action selection. To the machine learning community, our
proposed theory creates a novel interaction between the Actor and Critic in
Actor-Critic agents and notes a novel connection between RL and Bayesian
Optimization. We call our proposed agent the Metacognitive Actor Critic (MAC).
We conclude with showing how to create metacognition in machines by
implementing a deep MAC and showing that it can detect (some of) its own
suboptimal actions without external information or delay.

negative:
Several approaches have been developed for answering users' specific
questions about AI behavior and for assessing their core functionality in terms
of primitive executable actions. However, the problem of summarizing an AI
agent's broad capabilities for a user is comparatively new. This paper presents
an algorithm for discovering from scratch the suite of high-level
"capabilities" that an AI system with arbitrary internal planning
algorithms/policies can perform. It computes conditions describing the
applicability and effects of these capabilities in user-interpretable terms.
Starting from a set of user-interpretable state properties, an AI agent, and a
simulator that the agent can interact with, our algorithm returns a set of
high-level capabilities with their parameterized descriptions. Empirical
evaluation on several game-based scenarios shows that this approach efficiently
learns descriptions of various types of AI agents in deterministic, fully
observable settings. User studies show that such descriptions are easier to
understand and reason with than the agent's primitive actions.

negative:
Recent advances in machine learning have made it possible to train
artificially intelligent agents that perform with super-human accuracy on a
great diversity of complex tasks. However, the process of training these
capabilities often necessitates millions of annotated examples -- far more than
humans typically need in order to achieve a passing level of mastery on similar
tasks. Thus, while contemporary methods in machine learning can produce agents
that exhibit super-human performance, their rate of learning per opportunity in
many domains is decidedly lower than human-learning. In this work we formalize
a theory of Decomposed Inductive Procedure Learning (DIPL) that outlines how
different forms of inductive symbolic learning can be used in combination to
build agents that learn educationally relevant tasks such as mathematical, and
scientific procedures, at a rate similar to human learners. We motivate the
construction of this theory along Marr's concepts of the computational,
algorithmic, and implementation levels of cognitive modeling, and outline at
the computational-level six learning capacities that must be achieved to
accurately model human learning. We demonstrate that agents built along the
DIPL theory are amenable to satisfying these capacities, and demonstrate, both
empirically and theoretically, that DIPL enables the creation of agents that
exhibit human-like learning performance.

negative:
This paper surveys the field of deep multiagent reinforcement learning. The
combination of deep neural networks with reinforcement learning has gained
increased traction in recent years and is slowly shifting the focus from
single-agent to multiagent environments. Dealing with multiple agents is
inherently more complex as (a) the future rewards depend on multiple players'
joint actions and (b) the computational complexity increases. We present the
most common multiagent problem representations and their main challenges, and
identify five research areas that address one or more of these challenges:
centralised training and decentralised execution, opponent modelling,
communication, efficient coordination, and reward shaping. We find that many
computational studies rely on unrealistic assumptions or are not generalisable
to other settings; they struggle to overcome the curse of dimensionality or
nonstationarity. Approaches from psychology and sociology capture promising
relevant behaviours, such as communication and coordination, to help agents
achieve better performance in multiagent settings. We suggest that, for
multiagent reinforcement learning to be successful, future research should
address these challenges with an interdisciplinary approach to open up new
possibilities in multiagent reinforcement learning.

negative:
Representations are internal models of the environment that can provide
guidance to a behaving agent, even in the absence of sensory information. It is
not clear how representations are developed and whether or not they are
necessary or even essential for intelligent behavior. We argue here that the
ability to represent relevant features of the environment is the expected
consequence of an adaptive process, give a formal definition of representation
based on information theory, and quantify it with a measure R. To measure how R
changes over time, we evolve two types of networks---an artificial neural
network and a network of hidden Markov gates---to solve a categorization task
using a genetic algorithm. We find that the capacity to represent increases
during evolutionary adaptation, and that agents form representations of their
environment during their lifetime. This ability allows the agents to act on
sensorial inputs in the context of their acquired representations and enables
complex and context-dependent behavior. We examine which concepts (features of
the environment) our networks are representing, how the representations are
logically encoded in the networks, and how they form as an agent behaves to
solve a task. We conclude that R should be able to quantify the representations
within any cognitive system, and should be predictive of an agent's long-term
adaptive success.

negative:
The integration of deep learning to reinforcement learning (RL) has enabled
RL to perform efficiently in high-dimensional environments. Deep RL methods
have been applied to solve many complex real-world problems in recent years.
However, development of a deep RL-based system is challenging because of
various issues such as the selection of a suitable deep RL algorithm, its
network configuration, training time, training methods, and so on. This paper
proposes a comprehensive software framework that not only plays a vital role in
designing a connect-the-dots deep RL architecture but also provides a guideline
to develop a realistic RL application in a short time span. We have designed
and developed a deep RL-based software framework that strictly ensures
flexibility, robustness, and scalability. By inheriting the proposed
architecture, software managers can foresee any challenges when designing a
deep RL-based system. As a result, they can expedite the design process and
actively control every stage of software development, which is especially
critical in agile development environments. To enforce generalization, the
proposed architecture does not depend on a specific RL algorithm, a network
configuration, the number of agents, or the type of agents. Using our
framework, software developers can develop and integrate new RL algorithms or
new types of agents, and can flexibly change network configuration or the
number of agents.

negative:
Numerous deep reinforcement learning agents have been proposed, and each of
them has its strengths and flaws. In this work, we present a Cooperative
Heterogeneous Deep Reinforcement Learning (CHDRL) framework that can learn a
policy by integrating the advantages of heterogeneous agents. Specifically, we
propose a cooperative learning framework that classifies heterogeneous agents
into two classes: global agents and local agents. Global agents are off-policy
agents that can utilize experiences from the other agents. Local agents are
either on-policy agents or population-based evolutionary algorithms (EAs)
agents that can explore the local area effectively. We employ global agents,
which are sample-efficient, to guide the learning of local agents so that local
agents can benefit from sample-efficient agents and simultaneously maintain
their advantages, e.g., stability. Global agents also benefit from effective
local searches. Experimental studies on a range of continuous control tasks
from the Mujoco benchmark show that CHDRL achieves better performance compared
with state-of-the-art baselines.

negative:
With the development of deep representation learning, the domain of
reinforcement learning (RL) has become a powerful learning framework now
capable of learning complex policies in high dimensional environments. This
review summarises deep reinforcement learning (DRL) algorithms and provides a
taxonomy of automated driving tasks where (D)RL methods have been employed,
while addressing key computational challenges in real world deployment of
autonomous driving agents. It also delineates adjacent domains such as behavior
cloning, imitation learning, inverse reinforcement learning that are related
but are not classical RL algorithms. The role of simulators in training agents,
methods to validate, test and robustify existing solutions in RL are discussed.

negative:
In recent years, there have been immense breakthroughs in Game AI research,
particularly with Reinforcement Learning (RL). Despite their success, the
underlying games are usually implemented with their own preset environments and
game mechanics, thus making it difficult for researchers to prototype different
game environments. However, testing the RL agents against a variety of game
environments is critical for recent effort to study generalization in RL and
avoid the problem of overfitting that may otherwise occur. In this paper, we
present Griddly as a new platform for Game AI research that provides a unique
combination of highly configurable games, different observer types and an
efficient C++ core engine. Additionally, we present a series of baseline
experiments to study the effect of different observation configurations and
generalization ability of RL agents.

negative:
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing
social intelligence in agents. In WAH, an AI agent needs to help a human-like
agent perform a complex household task efficiently. To succeed, the AI agent
needs to i) understand the underlying goal of the task by watching a single
demonstration of the human-like agent performing the same task (social
perception), and ii) coordinate with the human-like agent to solve the task in
an unseen environment as fast as possible (human-AI collaboration). For this
challenge, we build VirtualHome-Social, a multi-agent household environment,
and provide a benchmark including both planning and learning based baselines.
We evaluate the performance of AI agents with the human-like agent as well as
with real humans using objective metrics and subjective user ratings.
Experimental results demonstrate that the proposed challenge and virtual
environment enable a systematic evaluation on the important aspects of machine
social intelligence at scale.

negative:
In recent years deep reinforcement learning (RL) systems have attained
superhuman performance in a number of challenging task domains. However, a
major limitation of such applications is their demand for massive amounts of
training data. A critical present objective is thus to develop deep RL methods
that can adapt rapidly to new tasks. In the present work we introduce a novel
approach to this challenge, which we refer to as deep meta-reinforcement
learning. Previous work has shown that recurrent networks can support
meta-learning in a fully supervised context. We extend this approach to the RL
setting. What emerges is a system that is trained using one RL algorithm, but
whose recurrent dynamics implement a second, quite separate RL procedure. This
second, learned RL algorithm can differ from the original one in arbitrary
ways. Importantly, because it is learned, it is configured to exploit structure
in the training domain. We unpack these points in a series of seven
proof-of-concept experiments, each of which examines a key aspect of deep
meta-RL. We consider prospects for extending and scaling up the approach, and
also point out some potentially important implications for neuroscience.

negative:
Deep reinforcement learning is the combination of reinforcement learning (RL)
and deep learning. This field of research has been able to solve a wide range
of complex decision-making tasks that were previously out of reach for a
machine. Thus, deep RL opens up many new applications in domains such as
healthcare, robotics, smart grids, finance, and many more. This manuscript
provides an introduction to deep reinforcement learning models, algorithms and
techniques. Particular focus is on the aspects related to generalization and
how deep RL can be used for practical applications. We assume the reader is
familiar with basic machine learning concepts.

negative:
We present a new approach for improving the friendliness and warmth of a
virtual agent in an AR environment by generating appropriate movement
characteristics. Our algorithm is based on a novel data-driven friendliness
model that is computed using a user-study and psychological characteristics. We
use our model to control the movements corresponding to the gaits, gestures,
and gazing of friendly virtual agents (FVAs) as they interact with the user's
avatar and other agents in the environment. We have integrated FVA agents with
an AR environment using with a Microsoft HoloLens. Our algorithm can generate
plausible movements at interactive rates to increase the social presence. We
also investigate the perception of a user in an AR setting and observe that an
FVA has a statistically significant improvement in terms of the perceived
friendliness and social presence of a user compared to an agent without the
friendliness modeling. We observe an increment of 5.71% in the mean responses
to a friendliness measure and an improvement of 4.03% in the mean responses to
a social presence measure.

negative:
We present an effective technique for training deep learning agents capable
of negotiating on a set of clauses in a contract agreement using a simple
communication protocol. We use Multi Agent Reinforcement Learning to train both
agents simultaneously as they negotiate with each other in the training
environment. We also model selfish and prosocial behavior to varying degrees in
these agents. Empirical evidence is provided showing consistency in agent
behaviors. We further train a meta agent with a mixture of behaviors by
learning an ensemble of different models using reinforcement learning. Finally,
to ascertain the deployability of the negotiating agents, we conducted
experiments pitting the trained agents against human players. Results
demonstrate that the agents are able to hold their own against human players,
often emerging as winners in the negotiation. Our experiments demonstrate that
the meta agent is able to reasonably emulate human behavior.

negative:
While the interpretability of machine learning models is often equated with
their mere syntactic comprehensibility, we think that interpretability goes
beyond that, and that human interpretability should also be investigated from
the point of view of cognitive science. The goal of this paper is to discuss to
what extent cognitive biases may affect human understanding of interpretable
machine learning models, in particular of logical rules discovered from data.
Twenty cognitive biases are covered, as are possible debiasing techniques that
can be adopted by designers of machine learning algorithms and software. Our
review transfers results obtained in cognitive psychology to the domain of
machine learning, aiming to bridge the current gap between these two areas. It
needs to be followed by empirical studies specifically focused on the machine
learning domain.

negative:
Experts in Artificial Intelligence (AI) development predict that advances in
the development of intelligent systems and agents will reshape vital areas in
our society. Nevertheless, if such an advance is not made prudently and
critically, reflexively, it can result in negative outcomes for humanity. For
this reason, several researchers in the area have developed a robust,
beneficial, and safe concept of AI for the preservation of humanity and the
environment. Currently, several of the open problems in the field of AI
research arise from the difficulty of avoiding unwanted behaviors of
intelligent agents and systems, and at the same time specifying what we really
want such systems to do, especially when we look for the possibility of
intelligent agents acting in several domains over the long term. It is of
utmost importance that artificial intelligent agents have their values aligned
with human values, given the fact that we cannot expect an AI to develop human
moral values simply because of its intelligence, as discussed in the
Orthogonality Thesis. Perhaps this difficulty comes from the way we are
addressing the problem of expressing objectives, values, and ends, using
representational cognitive methods. A solution to this problem would be the
dynamic approach proposed by Dreyfus, whose phenomenological philosophy shows
that the human experience of being-in-the-world in several aspects is not well
represented by the symbolic or connectionist cognitive method, especially in
regards to the question of learning values. A possible approach to this problem
would be to use theoretical models such as SED (situated embodied dynamics) to
address the values learning problem in AI.

negative:
In large-scale multi-agent systems, the large number of agents and complex
game relationship cause great difficulty for policy learning. Therefore,
simplifying the learning process is an important research issue. In many
multi-agent systems, the interactions between agents often happen locally,
which means that agents neither need to coordinate with all other agents nor
need to coordinate with others all the time. Traditional methods attempt to use
pre-defined rules to capture the interaction relationship between agents.
However, the methods cannot be directly used in a large-scale environment due
to the difficulty of transforming the complex interactions between agents into
rules. In this paper, we model the relationship between agents by a complete
graph and propose a novel game abstraction mechanism based on two-stage
attention network (G2ANet), which can indicate whether there is an interaction
between two agents and the importance of the interaction. We integrate this
detection mechanism into graph neural network-based multi-agent reinforcement
learning for conducting game abstraction and propose two novel learning
algorithms GA-Comm and GA-AC. We conduct experiments in Traffic Junction and
Predator-Prey. The results indicate that the proposed methods can simplify the
learning process and meanwhile get better asymptotic performance compared with
state-of-the-art algorithms.

negative:
A common vision from science fiction is that robots will one day inhabit our
physical spaces, sense the world as we do, assist our physical labours, and
communicate with us through natural language. Here we study how to design
artificial agents that can interact naturally with humans using the
simplification of a virtual environment. We show that imitation learning of
human-human interactions in a simulated world, in conjunction with
self-supervised learning, is sufficient to produce a multimodal interactive
agent, which we call MIA, that successfully interacts with non-adversarial
humans 75% of the time. We further identify architectural and algorithmic
techniques that improve performance, such as hierarchical action selection.
Altogether, our results demonstrate that imitation of multi-modal, real-time
human behaviour may provide a straightforward and surprisingly effective means
of imbuing agents with a rich behavioural prior from which agents might then be
fine-tuned for specific purposes, thus laying a foundation for training capable
agents for interactive robots or digital assistants. A video of MIA's behaviour
may be found at https://youtu.be/ZFgRhviF7mY

negative:
Artificial Intelligence is a central topic in the computer science
curriculum. From the year 2011 a project-based learning methodology based on
computer games has been designed and implemented into the intelligence
artificial course at the University of the Bio-Bio. The project aims to develop
software-controlled agents (bots) which are programmed by using heuristic
algorithms seen during the course. This methodology allows us to obtain good
learning results, however several challenges have been founded during its
implementation.
  In this paper we show how linguistic descriptions of data can help to provide
students and teachers with technical and personalized feedback about the
learned algorithms. Algorithm behavior profile and a new Turing test for
computer games bots based on linguistic modelling of complex phenomena are also
proposed in order to deal with such challenges.
  In order to show and explore the possibilities of this new technology, a web
platform has been designed and implemented by one of authors and its
incorporation in the process of assessment allows us to improve the teaching
learning process.

negative:
Experts in Artificial Intelligence (AI) development predict that advances in
the development of intelligent systems and agents will reshape vital areas in
our society. Nevertheless, if such an advance isn't done with prudence, it can
result in negative outcomes for humanity. For this reason, several researchers
in the area are trying to develop a robust, beneficial, and safe concept of
artificial intelligence. Currently, several of the open problems in the field
of AI research arise from the difficulty of avoiding unwanted behaviors of
intelligent agents, and at the same time specifying what we want such systems
to do. It is of utmost importance that artificial intelligent agents have their
values aligned with human values, given the fact that we cannot expect an AI to
develop our moral preferences simply because of its intelligence, as discussed
in the Orthogonality Thesis. Perhaps this difficulty comes from the way we are
addressing the problem of expressing objectives, values, and ends, using
representational cognitive methods. A solution to this problem would be the
dynamic cognitive approach proposed by Dreyfus, whose phenomenological
philosophy defends that the human experience of being-in-the-world cannot be
represented by the symbolic or connectionist cognitive methods. A possible
approach to this problem would be to use theoretical models such as SED
(situated embodied dynamics) to address the values learning problem in AI.

negative:
Humans are well-versed in reasoning about the behaviors of physical objects
and choosing actions accordingly to accomplish tasks, while it remains a major
challenge for AI. To facilitate research addressing this problem, we propose a
new testbed that requires an agent to reason about physical scenarios and take
an action appropriately. Inspired by the physical knowledge acquired in infancy
and the capabilities required for robots to operate in real-world environments,
we identify 15 essential physical scenarios. We create a wide variety of
distinct task templates, and we ensure all the task templates within the same
scenario can be solved by using one specific strategic physical rule. By having
such a design, we evaluate two distinct levels of generalization, namely the
local generalization and the broad generalization. We conduct an extensive
evaluation with human players, learning agents with varying input types and
architectures, and heuristic agents with different strategies. Inspired by how
human IQ is calculated, we define the physical reasoning quotient (Phy-Q score)
that reflects the physical reasoning intelligence of an agent using the
physical scenarios we considered. Our evaluation shows that 1) all agents are
far below human performance, and 2) learning agents, even with good local
generalization ability, struggle to learn the underlying physical reasoning
rules and fail to generalize broadly. We encourage the development of
intelligent agents that can reach the human level Phy-Q score. Website:
https://github.com/phy-q/benchmark

negative:
We consider the design of distributed algorithms that govern the manner in
which agents contribute to a social sensing platform. Specifically, we are
interested in situations where fairness among the agents contributing to the
platform is needed. A notable example are platforms operated by public bodies,
where fairness is a legal requirement. The design of such distributed systems
is challenging due to the fact that we wish to simultaneously realise an
efficient social sensing platform, but also deliver a predefined quality of
service to the agents (for example, a fair opportunity to contribute to the
platform). In this paper, we introduce iterated function systems (IFS) as a
tool for the design and analysis of systems of this kind. We show how the IFS
framework can be used to realise systems that deliver a predictable quality of
service to agents, can be used to underpin contracts governing the interaction
of agents with the social sensing platform, and which are efficient.
  To illustrate our design via a use case, we consider a large, high-density
network of participating parked vehicles. When awoken by an administrative
centre, this network proceeds to search for moving missing entities of interest
using RFID-based techniques. We regulate which vehicles are actively searching
for the moving entity of interest at any point in time. In doing so, we seek to
equalise vehicular energy consumption across the network. This is illustrated
through simulations of a search for a missing Alzheimer's patient in Melbourne,
Australia. Experimental results are presented to illustrate the efficacy of our
system and the predictability of access of agents to the platform independent
of initial conditions.

negative:
A reinforcement learning environment with adversary agents is proposed in
this work for pursuit-evasion game in the presence of fog of war, which is of
both scientific significance and practical importance in aerospace
applications. One of the most popular learning environments, StarCraft, is
adopted here and the associated mini-games are analyzed to identify the current
limitation for training adversary agents. The key contribution includes the
analysis of the potential performance of an agent by incorporating control and
differential game theory into the specific reinforcement learning environment,
and the development of an adversary agents challenge (SAAC) environment by
extending the current StarCraft mini-games. The subsequent study showcases the
use of this learning environment and the effectiveness of an adversary agent
for evasion units. Overall, the proposed SAAC environment should benefit
pursuit-evasion studies with rapidly-emerging reinforcement learning
technologies. Last but not least, the corresponding tutorial code can be found
at GitHub.

negative:
Along with the development of modern computing technology and social
sciences, both theoretical research and practical applications of social
computing have been continuously extended. In particular with the boom of
artificial intelligence (AI), social computing is significantly influenced by
AI. However, the conventional technologies of AI have drawbacks in dealing with
more complicated and dynamic problems. Such deficiency can be rectified by
hybrid human-artificial intelligence (H-AI) which integrates both human
intelligence and AI into one unity, forming a new enhanced intelligence. H-AI
in dealing with social problems shows the advantages that AI can not surpass.
This paper firstly introduces the concept of H-AI. AI is the intelligence in
the transition stage of H-AI, so the latest research progresses of AI in social
computing are reviewed. Secondly, it summarizes typical challenges faced by AI
in social computing, and makes it possible to introduce H-AI to solve these
challenges. Finally, the paper proposes a holistic framework of social
computing combining with H-AI, which consists of four layers: object layer,
base layer, analysis layer, and application layer. It represents H-AI has
significant advantages over AI in solving social problems.

negative:
Multi-Agent reinforcement learning has received lot of attention in recent
years and have applications in many different areas. Existing methods involving
Centralized Training and Decentralized execution, attempts to train the agents
towards learning a pattern of coordinated actions to arrive at optimal joint
policy. However if some agents are stochastic to varying degrees of
stochasticity, the above methods often fail to converge and provides poor
coordination among agents. In this paper we show how this stochasticity of
agents, which could be a result of malfunction or aging of robots, can add to
the uncertainty in coordination and there contribute to unsatisfactory global
coordination. In this case, the deterministic agents have to understand the
behavior and limitations of the stochastic agents while arriving at optimal
joint policy. Our solution, DSDF which tunes the discounted factor for the
agents according to uncertainty and use the values to update the utility
networks of individual agents. DSDF also helps in imparting an extent of
reliability in coordination thereby granting stochastic agents tasks which are
immediate and of shorter trajectory with deterministic ones taking the tasks
which involve longer planning. Such an method enables joint co-ordinations of
agents some of which may be partially performing and thereby can reduce or
delay the investment of agent/robot replacement in many circumstances. Results
on benchmark environment for different scenarios shows the efficacy of the
proposed approach when compared with existing approaches.

negative:
It is safe to assume that, for the foreseeable future, machine learning,
especially deep learning will remain both data- and computation-hungry. In this
paper, we ask: Can we build a global exchange where everyone can contribute
computation and data to train the next generation of machine learning
applications?
  We present an early, but running prototype of DataBright, a system that turns
the creation of training examples and the sharing of computation into an
investment mechanism. Unlike most crowdsourcing platforms, where the
contributor gets paid when they submit their data, DataBright pays dividends
whenever a contributor's data or hardware is used by someone to train a machine
learning model. The contributor becomes a shareholder in the dataset they
created. To enable the measurement of usage, a computation platform that
contributors can trust is also necessary. DataBright thus merges both a data
market and a trusted computation market.
  We illustrate that trusted computation can enable the creation of an AI
market, where each data point has an exact value that should be paid to its
creator. DataBright allows data creators to retain ownership of their
contribution and attaches to it a measurable value. The value of the data is
given by its utility in subsequent distributed computation done on the
DataBright computation market. The computation market allocates tasks and
subsequent payments to pooled hardware. This leads to the creation of a
decentralized AI cloud. Our experiments show that trusted hardware such as
Intel SGX can be added to the usual ML pipeline with no additional costs. We
use this setting to orchestrate distributed computation that enables the
creation of a computation market. DataBright is available for download at
https://github.com/ds3lab/databright.

negative:
Designing an effective communication mechanism among agents in reinforcement
learning has been a challenging task, especially for real-world applications.
The number of agents can grow or an environment sometimes needs to interact
with a changing number of agents in real-world scenarios. To this end, a
multi-agent framework needs to handle various scenarios of agents, in terms of
both scales and dynamics, for being practical to real-world applications. We
formulate the multi-agent environment with a different number of agents as a
multi-tasking problem and propose a meta reinforcement learning (meta-RL)
framework to tackle this problem. The proposed framework employs a meta-learned
Communication Pattern Recognition (CPR) module to identify communication
behavior and extract information that facilitates the training process.
Experimental results are poised to demonstrate that the proposed framework (a)
generalizes to an unseen larger number of agents and (b) allows the number of
agents to change between episodes. The ablation study is also provided to
reason the proposed CPR design and show such design is effective.

negative:
We introduce Imagination-Augmented Agents (I2As), a novel architecture for
deep reinforcement learning combining model-free and model-based aspects. In
contrast to most existing model-based reinforcement learning and planning
methods, which prescribe how a model should be used to arrive at a policy, I2As
learn to interpret predictions from a learned environment model to construct
implicit plans in arbitrary ways, by using the predictions as additional
context in deep policy networks. I2As show improved data efficiency,
performance, and robustness to model misspecification compared to several
baselines.

negative:
Motivated by recent advancements in Deep Reinforcement Learning (RL), we have
developed an RL agent to manage the operation of storage devices in a household
and is designed to maximize demand-side cost savings. The proposed technique is
data-driven, and the RL agent learns from scratch how to efficiently use the
energy storage device given variable tariff structures. In most of the studies,
the RL agent is considered as a black box, and how the agent has learned is
often ignored. We explain the learning progression of the RL agent, and the
strategies it follows based on the capacity of the storage device.

negative:
Counterfactual thinking describes a psychological phenomenon that people
re-infer the possible results with different solutions about things that have
already happened. It helps people to gain more experience from mistakes and
thus to perform better in similar future tasks. This paper investigates the
counterfactual thinking for agents to find optimal decision-making strategies
in multi-agent reinforcement learning environments. In particular, we propose a
multi-agent deep reinforcement learning model with a structure which mimics the
human-psychological counterfactual thinking process to improve the competitive
abilities for agents. To this end, our model generates several possible actions
(intent actions) with a parallel policy structure and estimates the rewards and
regrets for these intent actions based on its current understanding of the
environment. Our model incorporates a scenario-based framework to link the
estimated regrets with its inner policies. During the iterations, our model
updates the parallel policies and the corresponding scenario-based regrets for
agents simultaneously. To verify the effectiveness of our proposed model, we
conduct extensive experiments on two different environments with real-world
applications. Experimental results show that counterfactual thinking can
actually benefit the agents to obtain more accumulative rewards from the
environments with fair information by comparing to their opponents while
keeping high performing efficiency.

negative:
AI technology has a long history which is actively and constantly changing
and growing. It focuses on intelligent agents, which contain devices that
perceive the environment and based on which takes actions in order to maximize
goal success chances. In this paper, we will explain the modern AI basics and
various representative applications of AI. In the context of the modern
digitalized world, AI is the property of machines, computer programs, and
systems to perform the intellectual and creative functions of a person,
independently find ways to solve problems, be able to draw conclusions and make
decisions. Most artificial intelligence systems have the ability to learn,
which allows people to improve their performance over time. The recent research
on AI tools, including machine learning, deep learning and predictive analysis
intended toward increasing the planning, learning, reasoning, thinking and
action taking ability. Based on which, the proposed research intends towards
exploring on how the human intelligence differs from the artificial
intelligence. Moreover, we critically analyze what AI of today is capable of
doing, why it still cannot reach human intelligence and what are the open
challenges existing in front of AI to reach and outperform human level of
intelligence. Furthermore, it will explore the future predictions for
artificial intelligence and based on which potential solution will be
recommended to solve it within next decades.

negative:
While AI has benefited humans, it may also harm humans if not appropriately
developed. The focus of HCI work is transiting from conventional human
interaction with non-AI computing systems to interaction with AI systems. We
conducted a high-level literature review and a holistic analysis of current
work in developing AI systems from an HCI perspective. Our review and analysis
highlight the new changes introduced by AI technology and the new challenges
that HCI professionals face when applying the human-centered AI (HCAI) approach
in the development of AI systems. We also identified seven main issues in human
interaction with AI systems, which HCI professionals did not encounter when
developing non-AI computing systems. To further enable the implementation of
the HCAI approach, we identified new HCI opportunities tied to specific
HCAI-driven design goals to guide HCI professionals in addressing these new
issues. Finally, our assessment of current HCI methods shows the limitations of
these methods in support of developing AI systems. We propose alternative
methods that can help overcome these limitations and effectively help HCI
professionals apply the HCAI approach to the development of AI systems. We also
offer strategic recommendations for HCI professionals to effectively influence
the development of AI systems with the HCAI approach, eventually developing
HCAI systems.

negative:
AI researchers employ not only the scientific method, but also methodology
from mathematics and engineering. However, the use of the scientific method -
specifically hypothesis testing - in AI is typically conducted in service of
engineering objectives. Growing interest in topics such as fairness and
algorithmic bias show that engineering-focused questions only comprise a subset
of the important questions about AI systems. This results in the AI Knowledge
Gap: the number of unique AI systems grows faster than the number of studies
that characterize these systems' behavior. To close this gap, we argue that the
study of AI could benefit from the greater inclusion of researchers who are
well positioned to formulate and test hypotheses about the behavior of AI
systems. We examine the barriers preventing social and behavioral scientists
from conducting such studies. Our diagnosis suggests that accelerating the
scientific study of AI systems requires new incentives for academia and
industry, mediated by new tools and institutions. To address these needs, we
propose a two-sided marketplace called TuringBox. On one side, AI contributors
upload existing and novel algorithms to be studied scientifically by others. On
the other side, AI examiners develop and post machine intelligence tasks
designed to evaluate and characterize algorithmic behavior. We discuss this
market's potential to democratize the scientific study of AI behavior, and thus
narrow the AI Knowledge Gap.

negative:
In this paper, we present our research on programming human-level artificial
intelligence (HLAI), including 1) a definition of HLAI, 2) an environment to
develop and test HLAI, and 3) a cognitive architecture for HLAI. The term AI is
used in a broad meaning, and HLAI is not clearly defined. I claim that the
essence of Human-Level Intelligence to be the capability to learn from others'
experiences via language. The key is that the event described by language has
the same effect as if the agent experiences it firsthand for the update of the
behavior policy. To develop and test models with such a capability, we are
developing a simulated environment called SEDRo. There is a 3D Home, and a
mother character takes care of the baby (the learning agent) and teaches
languages. The environment provides comparable experiences to that of a human
baby from birth to one year. Finally, I propose a cognitive architecture of
HLAI called Modulated Heterarchical Prediction Memory (mHPM). In mHPM, there
are three components: a universal module that learns to predict the next vector
given the sequence of vector signals, a heterarchical network of those modules,
and a reward-based modulation of learning. mHPM models the workings of the
neocortex but the innate auxiliary units such hippocampus, reward system,
instincts, and amygdala play critical roles, too.

negative:
Campbell-Goodhart's law relates to the causal inference error whereby
decision-making agents aim to influence variables which are correlated to their
goal objective but do not reliably cause it. This is a well known error in
Economics and Political Science but not widely labelled in Artificial
Intelligence research. Through a simple example, we show how off-the-shelf deep
Reinforcement Learning (RL) algorithms are not necessarily immune to this
cognitive error. The off-policy learning method is tricked, whilst the
on-policy method is not. The practical implication is that naive application of
RL to complex real life problems can result in the same types of policy errors
that humans make. Great care should be taken around understanding the causal
model that underpins a solution derived from Reinforcement Learning.

negative:
How to attribute responsibility for autonomous artificial intelligence (AI)
systems' actions has been widely debated across the humanities and social
science disciplines. This work presents two experiments ($N$=200 each) that
measure people's perceptions of eight different notions of moral responsibility
concerning AI and human agents in the context of bail decision-making. Using
real-life adapted vignettes, our experiments show that AI agents are held
causally responsible and blamed similarly to human agents for an identical
task. However, there was a meaningful difference in how people perceived these
agents' moral responsibility; human agents were ascribed to a higher degree of
present-looking and forward-looking notions of responsibility than AI agents.
We also found that people expect both AI and human decision-makers and advisors
to justify their decisions regardless of their nature. We discuss policy and
HCI implications of these findings, such as the need for explainable AI in
high-stakes scenarios.

negative:
We consider finite and infinite horizon dynamic programming problems, where
the control at each stage consists of several distinct decisions, each one made
by one of several agents. We introduce an approach, whereby at every stage,
each agent's decision is made by executing a local rollout algorithm that uses
a base policy, together with some coordinating information from the other
agents. The amount of local computation required at every stage by each agent
is independent of the number of agents, while the amount of total computation
(over all agents) grows linearly with the number of agents. By contrast, with
the standard rollout algorithm, the amount of total computation grows
exponentially with the number of agents. Despite the drastic reduction in
required computation, we show that our algorithm has the fundamental cost
improvement property of rollout: an improved performance relative to the base
policy. We also discuss possibilities to improve further the method's
computational efficiency through limited agent coordination and parallelization
of the agents' computations. Finally, we explore related approximate policy
iteration algorithms for infinite horizon problems, and we prove that the cost
improvement property steers the algorithm towards convergence to an
agent-by-agent optimal policy.

negative:
This paper describes an application of reinforcement learning to the mention
detection task. We define a novel action-based formulation for the mention
detection task, in which a model can flexibly revise past labeling decisions by
grouping together tokens and assigning partial mention labels. We devise a
method to create mention-level episodes and we train a model by rewarding
correctly labeled complete mentions, irrespective of the inner structure
created. The model yields results which are on par with a competitive
supervised counterpart while being more flexible in terms of achieving targeted
behavior through reward modeling and generating internal mention structure,
especially on longer mentions.

negative:
There is a growing focus on how to design safe artificial intelligent (AI)
agents. As systems become more complex, poorly specified goals or control
mechanisms may cause AI agents to engage in unwanted and harmful outcomes. Thus
it is necessary to design AI agents that follow initial programming intentions
as the program grows in complexity. How to specify these initial intentions has
also been an obstacle to designing safe AI agents. Finally, there is a need for
the AI agent to have redundant safety mechanisms to ensure that any programming
errors do not cascade into major problems. Humans are autonomous intelligent
agents that have avoided these problems and the present manuscript argues that
by understanding human self-regulation and goal setting, we may be better able
to design safe AI agents. Some general principles of human self-regulation are
outlined and specific guidance for AI design is given.

negative:
Convolutional Neural Networks achieve state-of-the-art accuracy in object
detection tasks. However, they have large computational and energy requirements
that challenge their deployment on resource-constrained edge devices. Object
detection takes an image as an input, and identifies the existing object
classes as well as their locations in the image. In this paper, we leverage the
prior knowledge about the probabilities that different object categories can
occur jointly to increase the efficiency of object detection models. In
particular, our technique clusters the object categories based on their spatial
co-occurrence probability. We use those clusters to design an adaptive network.
During runtime, a branch controller decides which part(s) of the network to
execute based on the spatial context of the input frame. Our experiments using
COCO dataset show that our adaptive object detection model achieves up to 45%
reduction in the energy consumption, and up to 27% reduction in the latency,
with a small loss in the average precision (AP) of object detection.

negative:
Learning by observation can be of key importance whenever agents sharing
similar features want to learn from each other. This paper presents an agent
architecture that enables software agents to learn by direct observation of the
actions executed by expert agents while they are performing a task. This is
possible because the proposed architecture displays information that is
essential for observation, making it possible for software agents to observe
each other. The agent architecture supports a learning process that covers all
aspects of learning by observation, such as discovering and observing experts,
learning from the observed data, applying the acquired knowledge and evaluating
the agents progress. The evaluation provides control over the decision to
obtain new knowledge or apply the acquired knowledge to new problems. We
combine two methods for learning from the observed information. The first one,
the recall method, uses the sequence on which the actions were observed to
solve new problems. The second one, the classification method, categorizes the
information in the observed data and determines to which set of categories the
new problems belong. Results show that agents are able to learn in conditions
where common supervised learning algorithms fail, such as when agents do not
know the results of their actions a priori or when not all the effects of the
actions are visible. The results also show that our approach provides better
results than other learning methods since it requires shorter learning periods.

negative:
Individuality is essential in human society, which induces the division of
labor and thus improves the efficiency and productivity. Similarly, it should
also be the key to multi-agent cooperation. Inspired by that individuality is
of being an individual separate from others, we propose a simple yet efficient
method for the emergence of individuality (EOI) in multi-agent reinforcement
learning (MARL). EOI learns a probabilistic classifier that predicts a
probability distribution over agents given their observation and gives each
agent an intrinsic reward of being correctly predicted by the classifier. The
intrinsic reward encourages the agents to visit their own familiar
observations, and learning the classifier by such observations makes the
intrinsic reward signals stronger and the agents more identifiable. To further
enhance the intrinsic reward and promote the emergence of individuality, two
regularizers are proposed to increase the discriminability of the classifier.
We implement EOI on top of popular MARL algorithms. Empirically, we show that
EOI significantly outperforms existing methods in a variety of multi-agent
cooperative scenarios.

negative:
Inattentional blindness is the psychological phenomenon that causes one to
miss things in plain sight. It is a consequence of the selective attention in
perception that lets us remain focused on important parts of our world without
distraction from irrelevant details. Motivated by selective attention, we study
the properties of artificial agents that perceive the world through the lens of
a self-attention bottleneck. By constraining access to only a small fraction of
the visual input, we show that their policies are directly interpretable in
pixel space. We find neuroevolution ideal for training self-attention
architectures for vision-based reinforcement learning (RL) tasks, allowing us
to incorporate modules that can include discrete, non-differentiable operations
which are useful for our agent. We argue that self-attention has similar
properties as indirect encoding, in the sense that large implicit weight
matrices are generated from a small number of key-query parameters, thus
enabling our agent to solve challenging vision based tasks with at least 1000x
fewer parameters than existing methods. Since our agent attends to only task
critical visual hints, they are able to generalize to environments where task
irrelevant elements are modified while conventional methods fail. Videos of our
results and source code available at https://attentionagent.github.io/

negative:
3D objects (artefacts) are made to fulfill functions. Designing an object
often starts with defining a list of functionalities that it should provide,
also known as functional requirements. Today, the design of 3D object models is
still a slow and largely artisanal activity, with few Computer-Aided Design
(CAD) tools existing to aid the exploration of the design solution space. To
accelerate the design process, we introduce an algorithm for generating object
shapes with desired functionalities. Following the concept of form follows
function, we assume that existing object shapes were rationally chosen to
provide desired functionalities. First, we use an artificial neural network to
learn a function-to-form mapping by analysing a dataset of objects labeled with
their functionalities. Then, we combine forms providing one or more desired
functions, generating an object shape that is expected to provide all of them.
Finally, we verify in simulation whether the generated object possesses the
desired functionalities, by defining and executing functionality tests on it.

negative:
This book-length article combines several peer reviewed papers and new
material to analyze the issues of ethical artificial intelligence (AI). The
behavior of future AI systems can be described by mathematical equations, which
are adapted to analyze possible unintended AI behaviors and ways that AI
designs can avoid them. This article makes the case for utility-maximizing
agents and for avoiding infinite sets in agent definitions. It shows how to
avoid agent self-delusion using model-based utility functions and how to avoid
agents that corrupt their reward generators (sometimes called "perverse
instantiation") using utility functions that evaluate outcomes at one point in
time from the perspective of humans at a different point in time. It argues
that agents can avoid unintended instrumental actions (sometimes called "basic
AI drives" or "instrumental goals") by accurately learning human values. This
article defines a self-modeling agent framework and shows how it can avoid
problems of resource limits, being predicted by other agents, and inconsistency
between the agent's utility function and its definition (one version of this
problem is sometimes called "motivated value selection"). This article also
discusses how future AI will differ from current AI, the politics of AI, and
the ultimate use of AI to help understand the nature of the universe and our
place in it.

negative:
Consider a platform that wants to learn a personalized policy for each user,
but the platform faces the risk of a user abandoning the platform if she is
dissatisfied with the actions of the platform. For example, a platform is
interested in personalizing the number of newsletters it sends, but faces the
risk that the user unsubscribes forever. We propose a general thresholded
learning model for scenarios like this, and discuss the structure of optimal
policies. We describe salient features of optimal personalization algorithms
and how feedback the platform receives impacts the results. Furthermore, we
investigate how the platform can efficiently learn the heterogeneity across
users by interacting with a population and provide performance guarantees.

negative:
Natural language understanding tasks such as open-domain question answering
often require retrieving and assimilating factual information from multiple
sources. We propose to address this problem by integrating a semi-parametric
representation of a large text corpus into a Transformer model as a source of
factual knowledge. Specifically, our method represents knowledge with `mention
memory', a table of dense vector representations of every entity mention in a
corpus. The proposed model - TOME - is a Transformer that accesses the
information through internal memory layers in which each entity mention in the
input passage attends to the mention memory. This approach enables synthesis of
and reasoning over many disparate sources of information within a single
Transformer model. In experiments using a memory of 150 million Wikipedia
mentions, TOME achieves strong performance on several open-domain
knowledge-intensive tasks, including the claim verification benchmarks HoVer
and FEVER and several entity-based QA benchmarks. We also show that the model
learns to attend to informative mentions without any direct supervision.
Finally we demonstrate that the model can generalize to new unseen entities by
updating the memory without retraining.

negative:
Tracking humans that are interacting with the other subjects or environment
remains unsolved in visual tracking, because the visibility of the human of
interests in videos is unknown and might vary over time. In particular, it is
still difficult for state-of-the-art human trackers to recover complete human
trajectories in crowded scenes with frequent human interactions. In this work,
we consider the visibility status of a subject as a fluent variable, whose
change is mostly attributed to the subject's interaction with the surrounding,
e.g., crossing behind another object, entering a building, or getting into a
vehicle, etc. We introduce a Causal And-Or Graph (C-AOG) to represent the
causal-effect relations between an object's visibility fluent and its
activities, and develop a probabilistic graph model to jointly reason the
visibility fluent change (e.g., from visible to invisible) and track humans in
videos. We formulate this joint task as an iterative search of a feasible
causal graph structure that enables fast search algorithm, e.g., dynamic
programming method. We apply the proposed method on challenging video sequences
to evaluate its capabilities of estimating visibility fluent changes of
subjects and tracking subjects of interests over time. Results with comparisons
demonstrate that our method outperforms the alternative trackers and can
recover complete trajectories of humans in complicated scenarios with frequent
human interactions.

negative:
Named entity linking is to map an ambiguous mention in documents to an entity
in a knowledge base. The named entity linking is challenging, given the fact
that there are multiple candidate entities for a mention in a document. It is
difficult to link a mention when it appears multiple times in a document, since
there are conflicts by the contexts around the appearances of the mention. In
addition, it is difficult since the given training dataset is small due to the
reason that it is done manually to link a mention to its mapping entity. In the
literature, there are many reported studies among which the recent embedding
methods learn vectors of entities from the training dataset at document level.
To address these issues, we focus on how to link entity for mentions at a
sentence level, which reduces the noises introduced by different appearances of
the same mention in a document at the expense of insufficient information to be
used. We propose a new unified embedding method by maximizing the relationships
learned from knowledge graphs. We confirm the effectiveness of our method in
our experimental studies.

negative:
This special issue is dedicated to get a better picture of the relationships
between computational linguistics and cognitive science. It specifically raises
two questions: "what is the potential contribution of computational language
modeling to cognitive science?" and conversely: "what is the influence of
cognitive science in contemporary computational linguistics?"

negative:
Occlusion is one of the most significant challenges encountered by object
detectors and trackers. While both object detection and tracking has received a
lot of attention in the past, most existing methods in this domain do not
target detecting or tracking objects when they are occluded. However, being
able to detect or track an object of interest through occlusion has been a long
standing challenge for different autonomous tasks. Traditional methods that
employ visual object trackers with explicit occlusion modeling experience drift
and make several fundamental assumptions about the data. We propose to address
this with a `tracking-by-detection` approach that builds upon the success of
region based video object detectors. Our video level object detector uses a
novel recurrent computational unit at its core that enables long term
propagation of object features even under occlusion. Finally, we compare our
approach with existing state-of-the-art video object detectors and show that
our approach achieves superior results on a dataset of furniture assembly
videos collected from the internet, where small objects like screws, nuts, and
bolts often get occluded from the camera viewpoint.