summaries: This paper focuses on deep reinforcement learning methods and discusses the presence of inductive biases in the existingRL algorithm. Specifically, they discuss biases that take the form of domain knowledge or hyper-parameter tuning. The authors state that such biases rise the tradeoff between generality and performance wherein strong biases can lead to efficient performance but deteriorate generalization across domains. Further, it motivates that most inductive biases has a cost associated to it and hence it is important to study and analyze the effect of such biases.

summaries: This paper contains various numerical experiments to see the effects of some heuristics in reinforcement learning. Those heuristics include reward clipping, discounting for effective learning, repeating actions, and different network structures. However, since the training algorithms also greatly affect the performance of RL agents, it seems hard to draw any quantitive conclusions from this paper.

authors:
Pierre-Luc Bacon
Jean Harb
Doina Precup

title: The option-critic architecture

venue: AAAI Conference on Artificial Intelligence,

authors:
Marc G Bellemare
Yavar Naddaf
Joel Veness
Michael Bowling

title: The Arcade Learning Environment: An evaluation platform for general agents

venue: JAIR,

authors:
Marc G. Bellemare
Georg Ostrovski
Arthur Guez
Philip S. Thomas
Rémi Munos

title: Increasing the action gap: New operators for reinforcement learning

title: Dynamic programming

title: Gradient-based optimization of hyperparameters

authors:
Peter Dayan
Geoffrey E Hinton

title: Feudal reinforcement learning

venue: In Advances in neural information processing systems,

authors:
Rachit Dubey
Pulkit Agrawal
Deepak Pathak
Thomas L. Griffiths
Alexei A. Efros

title: Investigating human priors for playing video

venue: games. CoRR,

authors:
Lasse Espeholt
Hubert Soyer
Remi Munos
Karen Simonyan
Volodymir Mnih
Tom Ward
Yotam Doron
Vlad Firoiu
Tim Harley
Iain Dunning
Shane Legg
Koray Kavukcuoglu

title: Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

title: Action-gap phenomenon in reinforcement learning

venue: In Advances in Neural Information Processing Systems,

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

venue: In International Conference on Machine Learning,

authors:
Matteo Hessel
Joseph Modayil
Hado van Hasselt
Tom Schaul
Georg Ostrovski
Will Dabney
Daniel Horgan
Bilal Piot
Mohammad Gheshlaghi Azar
David Silver

title: Rainbow: Combining improvements in deep reinforcement learning

authors:
Matteo Hessel
Hubert Soyer
Lasse Espeholt
Wojciech Czarnecki
Simon Schmitt
Hado van Hasselt

title: Multi-task deep reinforcement learning with popart

venue: arXiv preprint arXiv:1809.04474,

title: Long short-term memory

authors:
Dan Horgan
John Quan
David Budden
Gabriel Barth-Maron
Matteo Hessel
Hado van Hasselt
David Silver

title: Distributed prioritized experience replay

venue: In ICLR,

authors:
Max Jaderberg
Volodymyr Mnih
Wojciech Marian Czarnecki
Tom Schaul
Joel Z. Leibo
David Silver
Koray Kavukcuoglu

title: Reinforcement learning with unsupervised auxiliary tasks

authors:
Matthew Johnson
Katja Hofmann
Tim Hutton
David Bignell

title: The Malmo platform for artificial intelligence experimentation

venue: In IJCAI,

authors:
Michał Kempka
Marek Wydmuch
Grzegorz Runc
Jakub Toczek
Wojciech Jaśkowski

title: Vizdoom: A doom-based ai research platform for visual reinforcement learning

venue: In IEEE Conference on Computational Intelligence and Games (CIG),

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Aravind S Lakshminarayanan
Sahil Sharma
Balaraman Ravindran

title: Dynamic action repetition for deep reinforcement learning

venue: In AAAI,

authors:
Sergey Levine
Peter Pastor
Alex Krizhevsky
Deirdre Quillen

title: Learning hand-eye coordination for robotic grasping with large-scale data collection

venue: ISER,

authors:
M.C. Machado
M.G. Bellemare
E. Talvitie
J. Veness
M. Hausknecht
M. Bowling

title: Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

title: Innateness, alphazero, and artificial intelligence

authors:
V Mnih
A Puigdomènech Badia
M Mirza
A Graves
T Lillicrap
T Harley
D Silver
K Kavukcuoglu

title: Asynchronous methods for deep reinforcement learning

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A. Rusu
Joel Veness
Marc G. Bellemare
Alex Graves
Martin Riedmiller
Andreas K. Fidjeland
Georg Ostrovski
Stig Petersen
Charles Beattie
Amir Sadik
Ioannis Antonoglou
Helen King
Dharshan Kumaran
Daan Wierstra
Shane Legg
Demis Hassabis

title: Human-level control through deep reinforcement learning

venue: Nature,

authors:
Razvan Pascanu
Tomas Mikolov
Yoshua Bengio

title: Understanding the exploding gradient problem

authors:
David Silver
Aja Huang
Chris J. Maddison
Arthur Guez
Laurent Sifre
George van den Driessche
Julian Schrittwieser
Ioannis Antonoglou
Veda Panneershelvam
Marc Lanctot
Sander Dieleman
Dominik Grewe
John Nham
Nal Kalchbrenner
Ilya Sutskever
Timothy Lillicrap
Madeleine Leach
Koray Kavukcuoglu
Thore Graepel
Demis Hassabis

title: Mastering the game of Go with deep neural networks and tree search. Nature, 2016

authors:
David Silver
Thomas Hubert
Julian Schrittwieser
Ioannis Antonoglou
Matthew Lai
Arthur Guez
Marc Lanctot
Laurent Sifre
Dharshan Kumaran
Thore Graepel
Timothy P. Lillicrap
Karen Simonyan
Demis Hassabis

title: Mastering chess and shogi by self-play with a general reinforcement learning

authors:
R.S. Sutton
A.G. Barto

title: Reinforcement Learning: An Introduction

venue: MIT press,

title: Learning to predict by the methods of temporal differences

venue: Machine learning,

title: Adapting bias by gradient descent: An incremental version of delta-bar-delta

authors:
Richard S. Sutton
Doina Precup
Satinder P. Singh

title: Intra-option learning about temporally abstract actions

venue: In Proceedings of the Fifteenth International Conference on Machine Learning,

authors:
Richard S Sutton
David A McAllester
Satinder P Singh
Yishay Mansour

title: Policy gradient methods for reinforcement learning with function approximation

authors:
Tijmen Tieleman
Geoffrey Hinton

title: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine

authors:
Hado van Hasselt
Arthur Guez
Matteo Hessel
Volodymyr Mnih
David Silver

title: Learning values across many orders of magnitude

authors:
Hado van Hasselt
Arthur Guez
David Silver

title: Deep reinforcement learning with double Qlearning

authors:
Ziyu Wang
Tom Schaul
Matteo Hessel
Hado van Hasselt
Marc Lanctot
Nando de Freitas

title: Dueling network architectures for deep reinforcement learning

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

venue: Mach. Learning,

venue: PopArt

text:
Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent’s objective and its interface to the environment. These inductive biases can take many forms, including domain knowledge and pretuned hyperparameters. In general, there is a trade-off between generality and performance when we use such biases. Stronger biases can lead to faster learning, but weaker biases can potentially lead to more general algorithms that work on a wider class of problems. This trade-off is relevant because these inductive biases are not free; substantial effort may be required to obtain relevant domain knowledge or to tune hyper-parameters effectively. In this paper, we re-examine several domain-specific components that modify the agent’s objective and environmental interface. We investigated whether the performance deteriorates when all these fixed components are replaced with adaptive solutions from the literature. In our experiments, performance sometimes decreased with the adaptive components, as one might expect when comparing to components crafted for the domain, but sometimes the adaptive components performed better. We then investigated the main benefit of having fewer domain-specific components, by comparing the learning performance of the two systems on a different set of continuous control problems, without additional tuning of either system. As hypothesized, the system with adaptive components performed better on many of the tasks.
The deep reinforcement learning (RL) community has demonstrated that well-tuned deep RL algorithms can master a wide range of challenging tasks. Human-level performance has been approached or surpassed on board-games such as Go and Chess (Silver et al., 2017), video-games such as Atari (Mnih et al., 2015; Hessel et al., 2018a; Horgan et al., 2018), and custom 3D navigation tasks (Johnson et al., 2016; Kempka et al., 2016; Beattie et al., 2016; Mnih et al., 2016; Jaderberg et al., 2016; Espeholt et al., 2018). These results are a testament to the generality of the overall approach. At times, however, the excitement for the constant stream of new domains being mastered by suitable RL algorithms may have over-shadowed the dependency on inductive biases of these agents, and the amount of tuning that is often required for these to perform effectively in new domains. A clear example of the benefits of generality is the AlphaZero algorithm (Silver et al., 2017). AlphaZero differs from the earlier AlphaGo algorithm (Silver et al., 2016) by removing all dependencies on Go-specific inductive biases and human data. After removing these biases, AlphaZero could not just surpass human performance on Go but also learn to play Chess and Shogi.
In general, there is a trade off between generality and performance when we inject inductive biases into our algorithms. Inductive biases take many forms, including domain knowledge and pretuned learning parameters. If applied carefully, such biases can lead to faster and better learning. On the other hand, fewer biases can potentially lead to more general algorithms that work out of the box on a wider class of problems. Crucially, most inductive biases are not free: for instance, substantial effort can be required to attain the relevant domain knowledge or pretune parameters. This cost is often hidden—for instance, one might use hyperparameters established as good in prior work on the same domain, without knowing how much data or time was spent optimising these, nor how specific the settings are to the given domain. Systematic studies about the impact of different inductive biases are rare and the generality of these different biases is often unclear. Another consideration is that inductive biases may mask the generality of other parts of the system as a whole; if a learning algorithm tuned for a specific domain does not generalize out of the box to a new domain, it can
be unclear whether it is due to the having the wrong inductive biases or whether the underpinning learning algorithm is lacking something important.
In this paper, we consider several commonly used domain heuristics, and investigate if (and by how much) performance deteriorates when we replace these with more general adaptive components, and we assess the impact of such replacements on the generality of the agent. We consider two broad ways of injecting inductive biases in RL agents: 1) sculpting the agent’s objective (e.g., clipping and discounting rewards), 2) sculpting the agent-environment interface (e.g., repeating each selected action a hard-coded fixed number of times, or crafting of the learning agent’s observation). Our contributions show that all the carefully crafted heuristics commonly used in Atari, and often considered essential for good performance on this domain, can be safely replaced with adaptive components, while preserving competitive performance across the benchmark. Furthermore, we show that this results in increased generality for an actor critic agent; the resulting fully adaptive system can be applied with no additional tuning on a separate suite of continuous control tasks that have different environmental interfaces. Here the adaptive system achieved much higher performance than a comparable system using the Atari tuned heuristics, and also higher performance than an actor-critic agent that was tuned for this benchmark (Tassa et al., 2018).

heading: 1 BACKGROUND

text:
Reinforcement learning (Sutton & Barto, 1998) is a framework for learning and decision making under uncertainty, where an agent interacts with its environment, receiving an observationOt, selecting an action At, and then receiving the next reward Rt+1, for each discrete time t.
Problem setting: The behaviour of an agent is formally specified by a policy π(At|Ht): a probability distribution over actions conditional on all previous observations, i.e. on the agent’s history Ht = O1:t. The agent’s objective is to find a policy that collects as much reward as possible, in each episode of experience. Crucially, it must learn such a policy without direct supervision, by trial and error. The amount of reward collected from time t onwards - the return - is a random variable
Gt = Tend∑ k=0 γkRt+k+1, (1)
where Tend is the number of steps until episode termination and γ ∈ [0, 1] is a constant discount factor (e.g., modelling monetary inflation, or a fixed additional probability of termination). The agent seeks an optimal policy, that maximizes values v(Ht) = Eπ[Gt|Ht]. In fully observable environments the optimal policy depends on the last observation alone: π∗(At|Ht) = π∗(At|Ot). Otherwise, the history may be summarized in an agent state St = f(Ht). The agent’s objective is then to jointly learn the state representation f and policy π(At|St), so as to maximize values. The fully observable case is formalized as a Markov Decision Process (Bellman, 1957).
Actor-critic algorithms: In the literature, two families of RL algorithms have particular prominence. Value-based algorithms learn to estimate values vw(s) that approximate to the true values, vw(s) ≈ vπ(s) ≡ Eπ[Gt|St = s], under some policy π. Values can be learned efficiently by exploiting a recursive decomposition vπ(s) = E[Rt+1 + γvπ(St+1)|St = s], known as Bellman equation, used in in temporal difference learning (Sutton, 1988) by sampling and making incremental updates:
∆wt = (Rt+1 + γvw(St+1)− vw(St))∇wvw(St). (2) In policy-based reinforcement learning algorithms, the policy πθ(At|St) is parameterized directly, and a stochastic gradient estimate can be derived to update the parameters in the direction of steepest ascent of the value (Williams, 1992; Sutton et al., 2000): for instance, according to the update
∆θt = Gt∇ log πθ(At|St). (3) Value-based and policy-based methods are combined in actor-critic algorithms. If a state value estimate is available, the policy updates can be computed from incomplete episodes by using the truncated returns G(n)t = ∑n−1 k=0 γ kRt+k+1 +γ nvw(St) that bootstrap on the value estimate at state St+n according to vw. This can reduce the variance of the updates. The variance can be further reduced by using state values as a baseline in policy updates, as in the advantage actor-critic update:
∆θt = (G (n) t − vw(St))∇θ log πθ(At|St). (4)

heading: 2 INDUCTIVE BIASES AND LEARNED SOLUTIONS

text: We now describe a few commonly used heuristics within the Atari domain, together with the adaptive replacements that we investigated in our experiments.

heading: 2.1 SCULPTING THE AGENT’S OBJECTIVE

text:
Many current deep RL agents do not directly optimize the true objective that they are evaluated against. In particular, RL agents in the Atari environments are typically tasked with optimizing a different handcrafted objective that incorporates biases to make learning simpler. We consider two additional ways of sculpting the agent’s objective that are still commonly used: reward clipping, and the use of fixed discounting of future rewards by a factor different from the one used for evaluation.
In many deep RL algorithms, the magnitude of the updates scales linearly with the returns. This makes it difficult to train the same RL agent, with the same hyper-parameters, on multiple domains, because good settings for hyper-parameters such as the learning rate vary across tasks. One common solution is to clip the rewards to a fixed range (Mnih et al., 2015), for instance [−1, 1]. This clipping makes the magnitude of returns and updates more comparable across domains. However, this also radically changes the agent objective, e.g., if all non-zero rewards are larger than one, then this amounts to maximizing the frequency of positive rewards rather than their cumulative sums. This can simplify the learning problem, and, when it is a good proxy for the true objective, can result in good performance. In other tasks, however, clipping can result in sub-optimal policies because the objective that is optimized is ill-aligned with the true objective.
PopArt (van Hasselt et al., 2016; Hessel et al., 2018b) was introduced as a principled solution to learn effectively irrespective of the magnitude of returns. PopArt works by tracking the mean µ and standard deviation σ of bootstrapped returns G(n)t . Temporal difference errors on value estimates can then be computed in a normalized space, with nw(s) denoting the normalized value, while the unnormalized values (needed, for instance, for bootstrapping) are recovered by a linear transformation vw(s) = µ+ σ ∗ nw(s). Doing this naively increases the non-stationarity of learning since the unnormalized predictions for all states change every time we change the statistics. PopArt therefore combines the adaptive rescaling with an inverse transformation of the weights at the last layer of nw(s), thereby preserving outputs precisely under any change in statistics µ → µ′ and σ → σ′. This is done exactly by updating weights and biases as w′ = wσ/σ′ and b′ = (σb+ µ− µ′)/σ′. Discounting is part of the traditional MDP formulation of RL. As such, it is often considered a property of the problem rather than a tunable parameter on the agent side. Indeed, sometimes, the environment does define a natural discounting of future rewards (e.g., inflation in a financial setting). However, even in episodic settings where the agent should maximize the undiscounted return, a constant discount factor is often used to simplify the learning problem (by having the agent focus on a relatively short time horizon). Optimizing this proxy of the true return often results in the agent achieving superior performance even in terms of the undiscounted return (Machado et al., 2017). This benefit comes with the cost of adding a hyperparameter, and performance can be quite sensitive to it—i.e., learning might be fast if the discount is small, but the solution may be too myopic.
Instead of tuning the discount manually, we use meta-learning (cf. Sutton, 1992; Bengio, 2000; Finn et al., 2017; Xu et al., 2018) to adapt the discount factor. The meta-gradient algorithm by Xu et al. (2018) uses the insight that the updates in Equations (2) and (4) are differentiable functions of hyper-parameters such as the discount. On the next sample or rollout of experience, using updated parameters w + ∆w(γ), written here as an explicit function of the discount, the agent then applies a gradient based actor-critic update, not to parameters w, but to the parameter θ that defines the discount γ which is used in a standard learning update. Xu et al. (2018) demonstrated that this improved performance on Atari, while using a separate hand-tuned discount factor for the metaupdate. We instead use the undiscounted returns (γm = 1) to define the meta-gradient updates, to understand if this technique can fully replace the need to reason about timescales and discounts.
A related heuristic, quite specific to Atari, is to track the number of lives that the agent has available (in several Atari games the agent is allowed to die a fixed number of times before the game is over), and hard code an episode termination (γ = 0) when this happens. We ignore the number of lives channel exposed by the Arcade Learning Environment in all our experiments.

heading: 2.2 SCULPTING THE AGENT-ENVIRONMENT INTERFACE

text:
In reinforcement learning we commonly assume that time progresses in discrete steps with a fixed duration. While algorithms are typically defined in this native space, learning at the fastest timescale provided by the environment is sometimes not the most practical or efficient solution, at least with the current generation of learning algorithms. It is often convenient to have the agent operate at a slower timescale, for instance by repeating each selected action a fixed number of times. The use of such fixed action repetitions is a widely used heuristic (e.g., Mnih et al., 2015; van Hasselt et al., 2016; Wang et al., 2016; Mnih et al., 2016) with several advantages. First, operating at a slower timescale increases the action gap (Farahmand, 2011), which can lead to more stable learning (Bellemare et al., 2015) because it becomes easier to appropriately rank actions reliably when the value estimates are uncertain or noisy. Second, selecting an action every few steps can save a significant amount of computation. Finally, committing longer to each action may help exploration, because the diameter of the solution space has effectively been reduced, for instance removing some often-irrelevant sequences of actions that jitter back and forth at a fast time scale.
A more general solution approach is for the agent to learn the most appropriate time scale at which to operate. Solving this problem in full generality is one of the aims of hierarchical reinforcement learning (Dayan & Hinton, 1993; Wiering & Schmidhuber, 1997; Sutton et al., 1998; Bacon et al., 2017). This general problem remains largely unsolved. A simpler, though more limited, approach is to allow the agent to learn how long to commit to each selected action (Lakshminarayanan et al., 2017). At each step t, the agent may be allowed to select both an actionAt and a commitmentCt, by sampling from two separate policies, both trained with policy gradient. Committing to an action for multiple steps raises the issue of how to handle intermediate observations without missing out on the potential computational savings. Conventional deep RL agents for Atari max-pool multiple image frames into a single observation. In our setting, the agent gets one image frame as an observation after each new action selection. The agent needs to learn to trade-off the benefits of action repetition (e.g., lower variance, more directed behaviour) with its disadvantages (e.g., not being able to revise its choices during as often, and missing potentially useful intermediate observations).
Many state-of-the-art RL agents use non-linear function approximators to represent values, policies, and states. Being able to learn flexible state representations was essential to capitalize on the successes of deep learning, and to scale reinforcement learning algorithms to visually complex domains (Mnih et al., 2015). While the use of deep neural network to approximate value functions and policies is widespread, their input is often not the raw observations but the result of domain-specific heuristic transformations. In Atari, for instance, most agents rely on down-sampling the observations to an 84×84 grid (down from the original 210×160 resolution), grey scaling them, and finally concatenating them into a K-Markov representation, with K = 4. We replace this specific preprocessing pipeline with a state representation learned end-to-end. We feed the RGB pixel observations at the native resolution of the Arcade Learning Environment into a convolutional network with 32 and 64 channels (in the first and second layer, respectively), both using 5× 5 kernels with a stride of 5. The output is fed to a fully connected layer with 256 hidden units, and then to an LSTM recurrent network (Hochreiter & Schmidhuber, 1997) of the same size. The policy for selecting the action and its commitment is computed as logits coming from two separate linear outputs of the LSTM. The network must then integrate information over time to cope with any issues like the flickering of the screen that had motivated the standard heuristic pipeline used by deep RL agents on Atari.

heading: 3 EXPERIMENTS

text:
When designing an algorithm it is useful to keep in mind what properties we would like the algorithm to satisfy. If the aim is to design an algorithm that is general, in addition to standard metrics such as asymptotic performance and data efficiency, there are additional dimensions that are useful to consider. Consider the following questions. 1) Does the algorithm require careful reasoning to select an appropriate time horizon for decision making? Manually setting the farsightedness of agents is tricky without domain knowledge or tuning. 2) How robust is the algorithm to the scaling of rewards? Rewards can have arbitrary scales, that may change by orders of magnitudes during training, as the agent improves. 3) Can the agent use commitment (e.g. action repetitions, or options) to alleviate the difficulty of learning at the fastest time scale? 4) Does the algorithm scale to complex domains? 5) How much tuning does the algorithm require to work on a new domain? Tuning is
expensive, and combinatorially so when there are many different hyper-parameters because often the hyper-parameters interact in subtle ways. None of these dimensions is binary, and different algorithms may satisfy each of them to a different degree, but keeping them in mind can be helpful to drive research towards more general RL solutions. We first discuss the first three in isolation, in the context of simple toy environments, to increase the understanding about how adaptive solutions compare to the corresponding heuristics they are intended to replace. We then use the 57 Atari games in the Arcade Learning Environment (Bellemare et al., 2013) to evaluate the performance of the different methods at scale. Finally, we investigate how well the methods generalize to new domains, using 28 continuous control tasks in the DeepMind Control Suite (Tassa et al., 2018).

heading: 3.1 MOTIVATING EXAMPLES

text:
We use a simple tabular actor-critic agent (A2C) to investigate in a minimal setup how domain heuristics and adaptive solutions compare with respect to some of these dimensions. We report average reward per step, after 5000 environment steps, for each of 20 replicas of each agent.
First, we investigate the role of discounting for effective learning. Consider a small chain environment with T = 9 states and 2 actions. The agent starts every episode in the middle of the chain. Moving left provides a -1 penalty. Moving right provides a reward of 2d/T , where d is the distance from the left end. When either end of the chain is reached, the episode ends, with an additional reward T on the far right end. Figure 1a shows a parameter study over a range of values for the discount factor. We found the best performance was between 0.5 and 0.9, where learning is quite effective, but observed decreased performance for lower or higher discounts. This shows that it can be difficult to set a suitable discount factor, and that the naive solution of just optimizing the undiscounted return may also perform poorly. Compare this to the same agent, but equipped with the adaptive meta-gradient algorithm discussed in Section 2.1 (in orange in Figure 1.a). Even initializing the discount to the value of 0.95 (which performed poorly in the parameter study), the agent learned to reduce the discount and performed in par with the best tuned fixed discount.
Next, we investigate the impact of reward scaling. We used the same domain, but keep the discount fixed to a value of 0.8 (as it was previously found to work well). We examine instead the performance of the agent when all rewards are scaled by a constant factor. Note that in the plots we report the unscaled rewards to make the results interpretable. Figure 1.b shows that the performance of the vanilla A2C agent (in blue) degraded rapidly when the scale of the rewards was significantly smaller or larger than 1. Compare this to the same agent equipped with PopArt, we observe better performance across multiple orders of magnitude for the reward scales. In this tiny problem, learning could also be achieved by tuning the learning rate for each reward scale, but that does not suffice for larger problems. Adaptive optimization algorithm such as Adam (Kingma & Ba, 2014) or RMSProp
(Tieleman & Hinton, 2012) can also provide some degree of invariance, but, as we will see in Section 3.2, they are not as effective as PopArt normalization.
Finally, we investigate the role of action repeats. We consider states arranged into a simple cycle of 11 states. The agent starts in state 0, and only moves in one direction using one action, the other action does not move the agent. The reward is 0 everywhere, except if the agent selects the nonmoving action in the 11 − th state: in this case the agent receives a reward of 100 and the episode ends. We compare an A2C agent that learns to choose the number of action repeats (up to 10), to an agent that used a fixed number of repetitions C. Figure 1.c shows how the number of fixed action repeats used by the agent is a sensitive hyper-parameter in this domain. Compare this to the adaptive agent that learns how often to repeat actions via policy gradient (in orange in Figure 1.c). This agent quickly learned a suitable number of action repeats and thereby performed very well. This is a general problem, in many domains of interest it can be useful to combine fine-grained control in certain states, with more coarse and directed behaviour in other parts of the state space.

heading: 3.2 PERFORMANCE ON LARGE DOMAINS

text:
To evaluate the performance of the different methods on larger problems, we use A2C agents on many Atari games. However, differently from the previous experiments, the agent learns in parallel from multiple copies of the environment, similarly to many state-of-the-art algorithms for reinforcement learning. This configuration increases the throughput of acting and learning, and speeds up the experiments. In parallel learning training setups, the learning updates may be applied synchronously (Espeholt et al., 2018) or asynchronously (Mnih et al., 2016). Our learning updates are synchronous: the agent’s policy takes steps in parallel across 16 copies of the environment to create multi-step learning updates, batched together to compute a single update to the parameters. We train individual agents on each game. Per-game scores are averaged over 8 seeds, and we then track the median human normalized score across all games. All hyper-parameters for our A2C agents were selected for a generic A2C agent on Atari before the following experiments were performed, with details given in the appendix.
Our experiments measure the performance of a full adaptive A2C agent with learned action repeats, PopArt normalization, learned discount factors, and an LSTM-based state representation. We compare the performance of this agent to agents with exactly one adaptive component disabled and replaced with one of two fixed components. This fixed component is either falling back to the environment specified task (e.g. learning directly from undiscounted returns), or using the corresponding fixed heuristic from DQN. These comparisons enable us to investigate how important the original heuristic is for current RL algorithms, as well as how fully an adaptive solution can replace it.
In the top left quadrant of Figure 2, we investigate action repeats and their impact on learning. We compare the fully general agent to an agent that used exactly 4 action repetitions (as tuned for Atari (Mnih et al., 2015)), and to an agent that acted and learned at the native frame rate of the environment. The adaptively learned solution performed almost as well as the tuned domain heuristic of always repeating each action 4 times. Interestingly, in the first 100M frames, also acting at the fastest rate was competitive with the agents equipped with action repetition (whether fixed or learned), at least in terms of data efficiency. However, while the agents with action repeats were still improving performance until the very end of the training budget, the agent acting at the fastest timescale appeared to plateau earlier. This performance plateau is observed in multiple games (see appendix), and we speculate that the use of multiple action repetitions may be helping achieve better exploration. It is worth noting that in wall-clock time the gap in the performance of the agents with action repetitions was even larger due to the cost of the additional compute.
In the top right quadrant, we investigate discounting. The agent that used undiscounted returns directly in the updates to policy and values performed very poorly, demonstrating that in complex environments the naive solution of directly optimizing the real objective is problematic with modern deep RL agents. Interestingly, while performance was very poor overall, the agent did demonstrate good performance on a few specific games. For instance, in bowling it achieved a better score than state of the art agents such as Rainbow (Hessel et al., 2018a) and ApeX (Horgan et al., 2018). The agent with tuned discount and the agent with a discount factor learned through meta-gradient RL performed much better overall. The adaptive solution did slightly better than the heuristic.
In the bottom left quadrant, we investigate the effect of reward scales. We compare the fully adaptive agent to an agent where clipping was used in place of PopArt, and to a naive agent that used the environment reward directly. Again, the naive solution performed very poorly, compared to using either the domain heuristic or the learned solution. Note that the naive solution is using RMSProp as an optimizer, in combination with gradient clipping by norm (Pascanu et al., 2012); together these techniques should provide at least some robustness to scaling issues, but in our experiments PopArt provided an additional large increase in performance. In this case, the domain heuristic (reward clipping) retained a significant edge over the adaptive solution. This suggests that reward clipping might not be helping exclusively with reward scales; the inductive bias of optimizing for a weighted frequency of rewards is a very good heuristic in many Atari games, and the qualitative behaviour resulting from optimizing the proxy objective might result in a better learning dynamics. While clipping was better in aggregate, PopArt yielded significantly improved scores on several games (e.g., centipede) where the clipped agent was stuck in sub-optimal policies.
Finally, we compare the fully end to end pipeline with a recurrent network, to a feedforward neural network with the standard Atari pipeline. The recurrent end to end solution performed best, showing that a recurrent network is sufficiently flexible to learn on its own to integrate relevant information over time, despite the Atari-specific features of the observation stream (such as the flickering of the screen) that motivated the more common heuristic approach.

heading: 3.3 GENERALIZATION TO NEW DOMAINS

text:
Our previous analysis shows that learned solutions are mostly quite competitive with the domain heuristics on Atari, but do not uniformly provide additional benefits compared to the well tuned inductive biases that are commonly used in Atari. To investigate the generality of these different RL solutions, in this section we compare the fully general agent to an agent with all the usual inductive biases, but this time we evaluate them on a completely different benchmark: a collection of 28 continuous control tasks in the DeepMind Control Suite. The tasks represent a wide variety of physical control problems, and the dimension of the real-valued observation and action vectors
differs across the tasks. The environment state can be recovered from the observation in all but one task. The rewards are bounded between 0 and 1, and tasks are undiscounted.
Again we use a parallel A2C implementation, with 16 copies of the environment, and we aggregate results by first averaging scores across 8 seeds, and then taking the mean across all 28 tasks. Because all tasks in this benchmark are designed to have episode returns of comparable magnitude, there is no need to normalize the results to meaningfully aggregate them. For both agents we use the exact same solutions that were used in Atari, with no additional tuning. The agents naturally transfer to this new domain with two modifications: 1) we do not use convolutions since the observations do not have spatial structure. 2) the outputs of the policy head are interpreted as encoding the mean and variance of a Gaussian distribution instead of as the logits of a categorical one.
Figure 3 shows the fully general agent performed much better than the heuristic solution, which suggests that the set of inductive biases typically used by Deep RL agents on Atari do not generalize as well to this new domain as the set of adaptive solutions considered in this paper. This highlights the importance of being aware of the priors that we incorporate into our learning algorithms, and their impact on the generality of our agents. On the right side of Figure 3, we report the learning curves on the 10 tasks for which the absolute difference between the performance of the two agents was greatest (details on the full set of 28 tasks can be found in Appendix). The adaptive solutions performed equal or better than the heuristics on each of these 10 tasks, and the results in the appendix show performance was rarely worse. The reference horizontal black lines mark the performance of an A3C agent, tuned specifically for this suite of tasks, as reported by Tassa et al. (2018). The adaptive solution was also better, in aggregate, than this well tuned baseline; note however the tuned A3C agent achieved higher performance on a few games.

heading: 4 RELATED WORK AND DISCUSSION

text:
The present work was partially inspired by the work of Silver et al. (2017) in the context of Go. They demonstrated that specific domain specific heuristics (e.g. pretraining on human data, the use of handcrafted Go-specific features, and exploitation of certain symmetries in state space), while originally introduced to simplify learning (Silver et al., 2016), had actually outlived their usefulness: taking a purer approach, even stronger Go agents could be trained. Importantly, they showed removing these domain heuristics, the same algorithm could master other games, such as Shogi and Chess. In our paper, we adopted a similar philosophy but investigated the very different set of domain specific heuristics, that are used in more traditional deep reinforcement learning agents.
Our work relates to a broader debate (Marcus, 2018) about priors and innateness. There is evidence that we, as humans, posses specific types of biases, and that these have a role in enabling efficient learning (Spelke & Kinzler, 2007; Dubey et al., 2018); however, it is far from clear the extent to which these are essential for intelligent behaviour to arise, what form these priors take, and their impact on the generality of the resulting solutions. In this paper, we demonstrate that several heuristics we commonly use in our algorithms are already harming the generality of our methods. This does not mean that other different inductive biases could not be useful as we progress towards more flexible, intelligent agents; it is however a reminder that we must be careful with the domain knowledge and priors we bake into our solutions, and we must be ready to revise them over time.
We found that existing learned solutions are competitive with well tuned domain heuristics, even on the domain these heuristics were designed for, and they seem to generalize better to unseen domain. This makes a case for removing these biases in future research on Atari, since they are not essential for competitive performance, and they might hide issues in the core learning algorithm. The only case where we still found a significant gap in favour of the domain heuristic was in the case of clipping. While, to the best of our knowledge, PopArt does address scaling issues effectively, clipping still seems to help on several games. Changing the reward distribution has many subtle implications for the learning dynamics, beside affecting the magnitude of updates (e.g. exploration, risk-propensity, ...). We believe it is a fruitful direction for future research to investigate what other general solutions could be deployed in our agents to fully recover the advantages of clipping.
Several of the biases that we considered have knobs that could be tuned rather than learned (e.g. the discount, the number of repeats, etc); however, this is not a satisfying solution for several reasons. Tuning is expensive, and these heuristics interact subtly with each other, thus requiring to explore a combinatorial space to find suitable settings. Consider the use of fixed action repeats: when changing the number of repetitions you also need to change the discount factor, otherwise this will change the effective horizon of the agent, which in turn affects the magnitude of the returns and therefore the learning rate. Also, a fixed tuned value might still not give you the full benefits of an adaptive learned approach that can adapt to the various phases of the training process.
There are other two features of our algorithm that, despite not incorporating quite as much domain knowledge as the heuristics discussed in this paper, also constitute a potential impediment to its generality and scalability. 1) the use of parallel environments is not always feasible in practice, especially in real world applications (although recent work on robot farms Levine et al. (2016) shows that it might still be a valid approach when sufficient resources are available). 2) the use of back-propagation through time for training recurrent state representations constrains the length of the temporal relationship that we can learn, since the memory consumption is linear in the length of the rollouts. Further work in overcoming these limitations, successfully learning online from a single stream of experience, is a fruitful direction for future research.

heading: 5 TRAINING DETAILS

text:
We performed very limited tuning on Atari, both due to the cost of running so many comparison with 8 seeds at scale across 57 games, and because we were interested in generalization to a different domain. We used a learning rate of 1e − 3, an entropy cost of 0.01 and a baseline cost of 0.5. The learning rate was selected among 1e − 3, 1e − 4, 1e − 5 in an early version of the agent without any of the adaptive solutions, and verified to be reasonable as we were adding more components. Similarly the entropy cost was selected between 0.1 and 0.01. The baseline cost was not tuned, but we used the value of 0.5 that is common in the literature. We used the TensorFlow default settings for the additional parameters of the RMSProp optimizer.
The learning updates were batched across rollouts of 150 agent steps for 16 parallel copies of the environment. All experiments used gradient clipping by norm, with a maximum norm of 5., as in Wang et al. (2016). The adaptive agent could choose to repeat each action up to 6 times. PopArt used a step-size of 3e − 4, with bounds on the scale of 1e − 4 and 1e6 for numerical stability, as reported by Hessel et al. (2018b). We always used the undiscounted return to compute the metagradient updates, and when using the adaptive solutions these would update both the discount γ and the trace λ, as in the experiments by Xu et al. (2018). The meta-updates were computed on smaller rollouts of 15 agent steps, with a meta-learning rate of 1e− 3. No additional tuning was performed for any of the experiments on the Control Suite.

heading: 6 EXPERIMENT DETAILS

text:
In Figure 4 we report the detailed learning curves for all Atari games for three distinct agents: the fully adaptive agent (in red), the agent with fixed action repeats (in green), and the agent acting at the fastest timescale (in blue). It’s interesting to observe how on a large number of games (ice hockey, jamesbond, robotank, fishing, double dunk, etc) the agent with no action repeats seems to learn stably and effectively up to a point, but then plateaus quite abruptly. This suggests that exploration might be a major reason for the reduced performance of this agent that we observed in aggregate across the benchmark in the later stages of training.
In Figure 5 and 6 we show the discount factors and the eligibility traces λ as they were meta-learned over the course of training by the fully adaptive agent. We plot the soft time horizons T = (1−γ)−1 and T = (1−λ)−1. It’s interesting to observe how diverse these are for the various games. Consider for instance the discount factor γ: in games such as robotank, bank heist, and tutankham the horizon increases up to almost 1000, or even above, while in other games, such as surround and double dunk the discount reduces over time. Also for the eligibility trace parameter λ we observe very diverse values in different games. In Figure 7 we report the complete set of learning curves for all 28 tasks in the Control Suite (the 10 selected for the main text are those for which the difference in performance between the fully adaptive agent and the heuristic agent is greater).


summaries: This submission introduces a new method for vision+language navigation which tracks progress on the instruction using a progress monitor and a visual-textual co-grounding module. The method is shown to perform well on a standard benchmark. Ablation tests indicate the importance of each component of the model. Qualitative examples show that the proposed method attends to different parts of the instruction as the agent moves.

summaries: The paper considers the problem of following natural language route instructions in an unknown environment given only images. Integral to the proposed ("self-aware") approach is its ability to reason over which aspects of the instruction have been completed, which are to be followed next, which direction to go in next, as well as the agents current progress. This involves two primary components of the architecture. The first is a visual-textual module that grounds to the completed instruction, the next instruction, and the next direction based upon the visual input. The second is a "progress monitor" that takes the grounded instruction as input and captures the agent's progress towards completing the instruction.

abstractText: The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our selfmonitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at https://github.com/chihyaoma/selfmonitoring-agent.

authors:
Aishwarya Agrawal
Dhruv Batra
Devi Parikh
Aniruddha Kembhavi

title: Dont just assume; look and answer: Overcoming priors for visual question answering

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Muhannad Al-Omari
Paul Duckworth
David C Hogg
Anthony G Cohn

title: Natural language acquisition and grounding for embodied robotic systems

venue: In Association for the Advancement of Artificial Intelligence (AAAI),

authors:
Peter Anderson
Angel Chang
Devendra Singh Chaplot
Alexey Dosovitskiy
Saurabh Gupta
Vladlen Koltun
Jana Kosecka
Jitendra Malik
Roozbeh Mottaghi
Manolis Savva

title: On evaluation of embodied navigation agents

venue: arXiv preprint arXiv:1807.06757,

authors:
Peter Anderson
Qi Wu
Damien Teney
Jake Bruce
Mark Johnson
Niko Sünderhauf
Ian Reid
Stephen Gould
Anton van den Hengel

title: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Jacob Andreas
Dan Klein

title: Alignment-based compositional semantics for instruction following

venue: In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Stanislaw Antol
Aishwarya Agrawal
Jiasen Lu
Margaret Mitchell
Dhruv Batra
C Lawrence Zitnick
Devi Parikh

title: Vqa: Visual question answering

venue: Proceedings of the IEEE International Conference on Computer Vision (ICCV),

authors:
Jacob Arkin
Matthew R Walter
Adrian Boteanu
Michael E Napoli
Harel Biggie
Hadas KressGazit
Thomas M Howard

title: Contextual awareness: Understanding monologic natural language instructions for autonomous robots

venue: In Robot and Human Interactive Communication (RO-MAN),

authors:
Yoav Artzi
Luke Zettlemoyer

title: Weakly supervised learning of semantic parsers for mapping instructions to actions

venue: Transactions of the Association of Computational Linguistics (ACL),

authors:
Yael Benn
Thomas L Webb
Betty PI Chang
Yu-Hsuan Sun
Iain D Wilkinson
Tom FD Farrow

title: The neural basis of monitoring goal progress

venue: Frontiers in human neuroscience,

authors:
Elliot T Berkman
Matthew D Lieberman

title: The neuroscience of goal pursuit

venue: The psychology of goals,

authors:
Satchuthananthavale RK Branavan
Harr Chen
Luke S Zettlemoyer
Regina Barzilay

title: Reinforcement learning for mapping instructions to actions

venue: In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume

authors:
Angel Chang
Angela Dai
Thomas Funkhouser
Maciej Halber
Matthias Niessner
Manolis Savva
Shuran Song
Andy Zeng
Yinda Zhang

title: Matterport3D: Learning from RGB-D data in indoor environments

venue: International Conference on 3D Vision

authors:
Christopher H Chatham
Eric D Claus
Albert Kim
Tim Curran
Marie T Banich
Yuko Munakata

title: Cognitive control reflects context monitoring, not motoric stopping, in response inhibition

venue: PloS one,

authors:
Trevor Cohn
Cong Duy Vu Hoang
Ekaterina Vymolova
Kaisheng Yao
Chris Dyer
Gholamreza Haffari

title: Incorporating structural alignment biases into an attentional neural translation model

venue: In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL),

authors:
Abhishek Das
Satwik Kottur
Khushi Gupta
Avi Singh
Deshraj Yadav
José M.F. Moura
Devi Parikh
Dhruv Batra

title: Visual Dialog

authors:
Abhishek Das
Samyak Datta
Georgia Gkioxari
Stefan Lee
Devi Parikh
Dhruv Batra

title: Embodied question answering

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

authors:
Harm de Vries
Kurt Shuster
Dhruv Batra
Devi Parikh
Jason Weston
Douwe Kiela

title: Talk the walk: Navigating new york city through grounded dialogue

venue: arXiv preprint arXiv:1807.03367,

authors:
Krishna SR Dubba
Miguel R De Oliveira
Gi Hyun Lim
Hamidreza Kasaei
L Seabra Lopes
Ana Tomé
Anthony G Cohn

title: Grounding language in perception for scene conceptualization in autonomous robots. In Qualitative Representations for Robots: Papers from the AAAI Spring Symposium

venue: Technical report,

authors:
Felix Duvallet
Matthew R Walter
Thomas Howard
Sachithra Hemachandra
Jean Oh
Seth Teller
Nicholas Roy
Anthony Stentz

title: Inferring maps and behaviors from natural language instructions

venue: In Experimental Robotics,

authors:
Daniel Fried
Ronghang Hu
Volkan Cirik
Anna Rohrbach
Jacob Andreas
Louis-Philippe Morency
Taylor Berg-Kirkpatrick
Kate Saenko
Dan Klein
Trevor Darrell

title: Speaker-follower models for vision-and-language navigation

venue: In Advances in Neural Information Processing Systems (NIPS),

authors:
Kirill Gavrilyuk
Amir Ghodrati
Zhenyang Li
Cees GM Snoek

title: Actor and action video segmentation from a sentence

authors:
Daniel Gordon
Aniruddha Kembhavi
Mohammad Rastegari
Joseph Redmon
Dieter Fox
Ali Farhadi

title: Iqa: Visual question answering in interactive environments

authors:
Sachithra Hemachandra
Felix Duvallet
Thomas M. Howard
Nicholas Roy
Anthony Stentz
Matthew R. Walter

title: Learning models for following natural language directions in unknown environments

venue: In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),

authors:
Ronghang Hu
Huazhe Xu
Marcus Rohrbach
Jiashi Feng
Kate Saenko
Trevor Darrell

title: Natural language object retrieval

authors:
Michał Kempka
Marek Wydmuch
Grzegorz Runc
Jakub Toczek
Wojciech Jaśkowski

title: Vizdoom: A doom-based ai research platform for visual reinforcement learning

venue: Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG),

authors:
Thomas Kollar
Jayant Krishnamurthy
Grant P Strimel

title: Toward interactive grounded language acqusition

venue: In Robotics: Science and systems,

authors:
Cewu Lu
Ranjay Krishna
Michael Bernstein
Li Fei-Fei

title: Visual relationship detection with language priors

venue: In European Conference on Computer Vision (ECCV),

authors:
Jiasen Lu
Jianwei Yang
Dhruv Batra
Devi Parikh

title: Hierarchical question-image co-attention for visual question answering

venue: In Advances In Neural Information Processing Systems,

title: Neural baby talk

authors:
Chih-Yao Ma
Asim Kadav
Iain Melvin
Zsolt Kira
Ghassan AlRegib
Hans Peter Graf

title: Attend and interact: Higher-order object interactions for video understanding

authors:
Matt MacMahon
Brian Stankiewicz
Benjamin Kuipers

title: Walk the talk: Connecting language, knowledge, and action in route instructions

authors:
Cynthia Matuszek
Evan Herbst
Luke Zettlemoyer
Dieter Fox

title: Learning to parse natural language commands to a robot control system

authors:
Hongyuan Mei
Mohit Bansal
Matthew R Walter

title: Listen, attend, and walk: Neural mapping of navigational instructions to action sequences

authors:
Piotr Mirowski
Razvan Pascanu
Fabio Viola
Hubert Soyer
Andrew J Ballard
Andrea Banino
Misha Denil
Ross Goroshin
Laurent Sifre
Koray Kavukcuoglu

title: Learning to navigate in complex environments

venue: In Proceedings of the International Conference on Learning Representations (ICLR),

authors:
Piotr Mirowski
Matthew Koichi Grimes
Mateusz Malinowski
Karl Moritz Hermann
Keith Anderson
Denis Teplyashin
Karen Simonyan
Koray Kavukcuoglu
Andrew Zisserman
Raia Hadsell

title: Learning to navigate in cities without a map

venue: arXiv preprint arXiv:1804.00168,

authors:
Dipendra Misra
John Langford
Yoav Artzi

title: Mapping instructions and visual observations to actions with reinforcement learning

authors:
Dipendra K Misra
Jaeyong Sung
Kevin Lee
Ashutosh Saxena

title: Tell me dave: Context-sensitive grounding of natural language to manipulation instructions

venue: The International Journal of Robotics Research,

authors:
Arsalan Mousavian
Alexander Toshev
Marek Fiser
Jana Kosecka
James Davidson

title: Visual representations for semantic target driven navigation

venue: arXiv preprint arXiv:1805.06066,

authors:
Varun K Nagaraja
Vlad I Morariu
Larry S Davis

title: Modeling context between objects for referring expression understanding

authors:
Anna Rohrbach
Marcus Rohrbach
Ronghang Hu
Trevor Darrell
Bernt Schiele

title: Grounding of textual phrases in images by reconstruction

authors:
Michael Spranger
Luc Steels

title: Co-acquisition of syntax and semantics-an investigation in spatial language

authors:
Makarand Tapaswi
Yukun Zhu
Rainer Stiefelhagen
Antonio Torralba
Raquel Urtasun
Sanja Fidler

title: Movieqa: Understanding stories in movies through question-answering

venue: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Stefanie Tellex
Thomas Kollar
Steven Dickerson
Matthew R Walter
Ashis Gopal Banerjee
Seth J Teller
Nicholas Roy

title: Understanding natural language commands for robotic navigation and mobile manipulation

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Adam Vogel
Dan Jurafsky

title: Learning to follow navigational directions

venue: In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL),

authors:
Tsun-Hsuan Wang
Hung-Jui Huang
Juan-Ting Lin
Chan-Wei Hu
Kuo-Hao Zeng
Min Sun

title: Omnidirectional cnn for visual place recognition and navigation

venue: Proceedings of the IEEE Conference on Robotics and Automation (ICRA),

authors:
Xin Wang
Wenhan Xiong
Hongmin Wang
William Yang Wang

title: Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation

authors:
Greg Wayne
Chia-Chun Hung
David Amos
Mehdi Mirza
Arun Ahuja
Agnieszka GrabskaBarwinska
Jack Rae
Piotr Mirowski
Joel Z Leibo
Adam Santoro

title: Unsupervised predictive memory in a goal-directed agent

venue: arXiv preprint arXiv:1803.10760,

authors:
Haonan Yu
Xiaochen Lian
Haichao Zhang
Wei Xu

title: Guided feature transformation (gft): A neural language grounding module for embodied agents

venue: arXiv preprint arXiv:1805.08329,

title: Visual Semantic Planning using Deep Successor Representations

authors:
AR Zamir
F Xia
J He
A Sax
J Malik
S Savarese. Gibson

title: env: Real-world perception for embodied agents

authors:
Luowei Zhou
Chenliang Xu
Jason J Corso

title: Towards automatic learning of procedures from web instructional videos

authors:
Luowei Zhou
Yingbo Zhou
Jason J Corso
Richard Socher
Caiming Xiong

title: End-to-end dense video captioning with masked transformer

authors:
Yuke Zhu
Roozbeh Mottaghi
Eric Kolve
Joseph J Lim
Abhinav Gupta
Li Fei-Fei
Ali Farhadi

title: Target-driven visual navigation in indoor scenes using deep reinforcement learning

heading: 1 INTRODUCTION

text:
Recently, the Vision-and-Language (VLN) navigation task (Anderson et al., 2018b), which requires the agent to follow natural language instructions to navigate through a photo-realistic unknown environment, has received significant attention (Wang et al., 2018b; Fried et al., 2018). In the VLN task, an agent is placed in an unknown realistic environment and is required to follow natural language instructions to navigate from its starting location to a target location. In contrast to some existing navigation tasks (Kempka et al., 2016; Zhu et al., 2017; Mirowski et al., 2017; 2018), we address the class of tasks where the agent does not have an explicit representation of the target (e.g., location in a map or image representation of the goal) to know if the goal has been reached or not (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Arkin et al., 2017). Instead, the agent needs to be aware of its navigation status through the association between the sequence of observed visual inputs to instructions.
Consider an example as shown in Fig. 1, given the instruction ”Exit the bedroom and go towards the table. Go to the stairs on the left of the couch. Wait on the third step.”, the agent first needs to locate which instruction is needed for the next movement, which in turn requires the agent to be aware of (i.e., to explicitly represent or have an attentional focus on) which instructions were completed or ongoing in the previous steps. For instance, the action ”Go to the stairs” should be carried out once the agent has exited the room and moved towards the table. However, there exists inherent ambiguity for ”go towards the table”. Intuitively, the agent is expected to ”Go to the stairs” after completing ”go towards the table”. But, it is not clear what defines the completeness of ”Go towards the table”. The completeness of an ongoing action often depends on the availability of the next action. Since the transition between past and next part of the instructions is a soft
∗Work done while the authors were research interns at Salesforce Research.
boundary, in order to determine when to transit and to follow the instruction correctly the agent is required to keep track of both grounded instructions. On the other hand, assessing the progress made towards the goal has indeed been shown to be important for goal-directed tasks in humans decision-making (Benn et al., 2014; Chatham et al., 2012; Berkman & Lieberman, 2009). While a number of approaches have been proposed for VLN (Anderson et al., 2018b; Wang et al., 2018b; Fried et al., 2018), previous approaches generally are not aware of which instruction is next nor progress towards the goal; indeed, we qualitatively show that even the attentional mechanism of the baseline does not successfully track this information through time.
In this paper, we propose an agent endowed with the following abilities: (1) identify which direction to go by finding the part of the instruction that corresponds to the observed images—visual grounding, (2) identify which part of the instruction has been completed or ongoing and which part is potentially needed for the next action selection—textual grounding, and (3) ensure that the grounded instruction can correctly be used to estimate the progress made towards the goal, and apply regularization to ensure this —progress monitoring. Therefore, we introduce the self-monitoring agent consisting of two complementary modules: visual-textual co-grounding and progress monitor.
More specifically, we achieve both visual and textual grounding simultaneously by incorporating the full history of grounded instruction, observed images, and selected actions into the agent. We leverage the structural bias between the words in instructions used for action selection and progress made towards the goal and propose a new objective function for the agent to measure how well it can estimate the completeness of instruction-following. We then demonstrate that by conditioning on the positions and weights of grounded instruction as input, the agent can be self-monitoring of its progress and further ensure that the textual grounding accurately reflects the progress made.
Overall, we propose a novel self-monitoring agent for VLN and make the following contributions: (1) We introduce the visual-textual co-grounding module, which performs grounding interdependently across both visual and textual modalities. We show that it can outperform the baseline method by a large margin. (2) We propose to equip the self-monitoring agent with a progress monitor, and for navigation tasks involving instructions instantiate this by introducing a new objective function for training. We demonstrate that, unlike the baseline method, the position of grounded instruction can follow both past and future instructions, thereby tracking progress to the goal. (3) With the proposed self-monitoring agent, we set the new state-of-the-art performance on both seen and unseen environments on the standard benchmark. With 8% absolute improvement in success rate on the unseen test set, we are ranked #1 on the challenge leaderboard.

heading: 2 SELF-MONITORING NAVIGATION AGENT

heading: 2.1 NOTATION

text:
Given a natural language instruction with L words, its representation is denoted by X ={ x1,x2, . . . ,xL } , where xl is the feature vector for the l-th word encoded by an LSTM language encoder. Following Fried et al. (2018), we enable the agent with panoramic view. At each time step, the agent perceives a set of images at each viewpoint vt = { vt,1,vt,2, ...,vt,K } , where K
is the maximum number of navigable directions1, and vt,k represents the image feature of direction k. The co-grounding feature of instruction and image are denoted as x̂t and v̂t respectively. The selected action is denoted as at. The learnable weights are denoted with W , with appropriate sub/super-scripts as necessary. We omit the bias term b to avoid notational clutter in the exposition.

heading: 2.2 VISUAL AND TEXTUAL CO-GROUNDING

text:
First, we propose a visual and textual co-grounding model for the vision and language navigation task, as illustrated in Fig. 2. We model the agent with a sequence-to-sequence architecture with attention by using a recurrent neural network. More specifically, we use Long Short Term Memory (LSTM) to carry the flow of information effectively. At each step t, the decoder observes representations of the current attended panoramic image feature v̂t, previous selected action at−1 and current grounded instruction feature x̂t as input, and outputs an encoder context ht:
ht = LSTM([x̂t, v̂t,at−1]) (1)
where [, ] denotes concatenation. The previous encoder context ht−1 is used to obtain the textual grounding feature x̂t and visual grounding feature v̂t, whereas we use current encoder context ht to obtain next action at, all of which will be illustrated in the rest of the section.
Textual grounding. When the agent moves from one viewpoint to another, it is required to identify which direction to go by relying on a grounded instruction, i.e. which parts of the instruction should be used. This can either be the instruction matched with the past (ongoing action) or predicted for the future (next action). To capture the relative position between words within an instruction, we incorporate the positional encoding PE(·) (Vaswani et al., 2017) into the instruction features. We then perform soft-attention on the instruction features X , as shown on the left side of Fig. 2. The attention distribution over L words of the instructions is computed as:
ztextualt,l = (Wxht−1) >PE(xl), and αt = softmax(ztextualt ), (2)
where Wx are parameters to be learnt. ztextualt,l is a scalar value computed as the correlation between word l of the instruction and previous hidden state ht−1, andαt is the attention weight over features in X at time t. Based on the textual attention distribution, the grounded textual feature x̂t can be obtained by the weighted sum over the textual features x̂t = αTt X .
1Empirically, we found that using only the images on navigable directions to be slightly better than using all 36 surrounding images (12 headings × 3 elevations with 30 degree intervals).
Visual grounding. In order to locate the completed or ongoing instruction, the agent needs to keep track of the sequence of images observed along the navigation trajectory. We thus perform visual attention over the surrounding views based on its previous hidden vector ht−1. The visual attention weight βt can be obtained as:
zvisualt,k = (Wvht−1) >g(vt,k), and βt = softmax(zvisualt ), (3)
where g is a one-layer Multi-Layer Perceptron (MLP), Wv are parameters to be learnt. Similar to Eq. 2, the grounded visual feature v̂t can be obtained by the weighted sum over the visual features v̂t = β T t V .
Action selection. To make a decision on which direction to go, the agent finds the image features on navigable directions with the highest correlation with the grounded navigation instruction x̂t and the current hidden state ht. We use the inner-product to compute the correlation, and the probability of each navigable direction is then computed as:
ot,k = (Wa[ht, x̂t]) >g(vt,k) and pt = softmax(ot), (4)
where Wa are the learnt parameters, g(·) is the same MLP as in Eq. 3, and pt is the probability of each navigable direction at time t. We use categorical sampling during training to select the next action at. Unlike the previous method with the panoramic view (Fried et al., 2018), which attends to instructions only based on the history of observed images, we achieve both textual and visual grounding using the shared hidden state output containing grounded information from both textual and visual modalities. During action selection, we rely on both hidden state output and grounded instruction, instead of only relying on grounded instruction.

heading: 2.3 PROGRESS MONITOR

text:
It is imperative that the textual-grounding correctly reflects the progress towards the goal, since the agent can then implicitly know where it is now and what the next instruction to be completed will be. In the visual-textual co-grounding module, we ensure that the grounded instruction reasonably informs decision making when selecting a navigable direction. This is necessary but not sufficient for ensuring that the notion of progress to the goal is encoded. Thus, we propose to equip the agent with a progress monitor that serves as regularizer during training and prunes unfinished trajectories during inference.
Since the positions of localized instruction can be a strong indication of the navigation progress due to the structural alignment bias between navigation steps and instruction, the progress monitor can estimate how close the current viewpoint is to the final goal by conditioning on the positions and weights of grounded instruction. This can further enforce the result of textual-grounding to align with the progress made towards the goal and to ensure the correctness of the textual-grounding.
The progress monitor aims to estimate the navigation progress by conditioning on three inputs: the history of grounded images and instructions, the current observation of the surrounding images, and the positions of grounded instructions. We therefore represent these inputs by using (1) the previous hidden state ht−1 and the current cell state ct of the LSTM, (2) the grounded surrounding images v̂t, and (3) the distribution of attention weights of textual-grounding αt, as shown at the bottom of Fig. 2 represented by dotted lines.
Our proposed progress monitor first computes an additional hidden state output hpmt by using grounded image representations v̂t as input, similar to how a regular LSTM computes hidden states except we use concatenation over element-wise addition for empirical reasons2. The hidden state output is then concatenated with the attention weightsαt on textual-grounding to estimate how close the agent is to the goal3. The output of the progress monitor ppmt , which represents the completeness of instruction-following, is computed as:
hpmt = σ(Wh([ht−1, v̂t])⊗ tanh(ct)), p pm t = tanh(Wpm([αt,h pm t ])) (5)
where Wh and Wpm are the learnt parameters, ct is the cell state of the LSTM, ⊗ denotes the element-wise product, and σ is the sigmoid function.
2We found that using concatenation provides slightly better performance and stable training. 3We use zero-padding to handle instructions with various lengths.

heading: 2.4 TRAINING AND INFERENCE

text:
Training. We introduce a new objective function to train the proposed progress monitor. The training target ypmt is defined as the normalized distance in units of length from the current viewpoint to the goal, i.e., the target will be 0 at the beginning and closer to 1 as the agent approaches the goal4. Note that the target can also be lower than 0, if the agent’s current distance from the goal is farther than the starting point. Finally, our self-monitoring agent is optimized with a cross-entropy loss and a mean squared error loss, computed with respect to the outputs from both action selection and progress monitor.
Lloss = −λ T∑
t=1 ynvt log(pk,t)︸ ︷︷ ︸ action selection
−(1− λ) T∑
t=1
(ypmt − p pm t ) 2
︸ ︷︷ ︸ progress monitor
(6)
where pk,t is the action probability of each navigable direction, λ = 0.5 is the weight balancing the two losses, and ynvt is the ground-truth navigable direction at step t.
Inference. During inference, we follow Fried et al. (2018) by using beam search. we propose that, while the agent decides which trajectories in the beams to keep, it is equally important to evaluate the state of the beams on actions as well as on the agent’s confidence in completing the given instruction at each traversed viewpoint. We accomplish this idea by integrating the output of our progress monitor into the accumulated probability of beam search. At each step, when candidate trajectories compete based on accumulated probability, we integrate the estimated completeness of instruction-following ppmt (normalized between 0 to 1) with action probability pk,t to directly evaluate the partial and unfinished candidate routes: pbeamt = p pm t × pk,t.
Without beam search, we use greedy decoding for action selection with one condition. If the progress monitor output decreases (ppmt+1 < p pm t ), the agent is required to move back to the previous viewpoint and select the action with next highest probability. We repeat this process until the selected action leads to increasing progress monitor output. We denote this procedure as progress inference.

heading: 3 EXPERIMENTS

text:
R2R Dataset. We use the Room-to-Room (R2R) dataset (Anderson et al., 2018b) for evaluating our proposed approach. The R2R dataset is built upon the Matterport3D dataset (Chang et al., 2017) and has 7,189 paths sampled from its navigation graphs. Each path has three ground-truth navigation instructions written by humans. The whole dataset is divided into 4 sets: training, validation seen, validation unseen, and test sets unseen.
Evaluation metrics. We follow the same evaluation metrics used by previous work on the R2R task: (1) Navigation Error (NE), mean of the shortest path distance in meters between the agent’s final
4We set the target to 1 if the agent’s distance to the goal is less than 3.
position and the goal location. (2) Success Rate (SR), the percentage of final positions less than 3m away from the goal location. (3) Oracle Success Rate (OSR), the success rate if the agent can stop at the closest point to the goal along its trajectory. In addition, we also include the recently introduced Success rate weighted by (normalized inverse) Path Length (SPL) (Anderson et al., 2018a), which trades-off Success Rate against trajectory length.

heading: 3.1 COMPARISON WITH PRIOR ART

text:
We first compare the proposed self-monitoring agent with existing approaches. As shown in Table 1, our method achieves significant performance improvement compared to the state of the arts without data augmentation. We achieve 70% SR on the seen environment and 57% on the unseen environment while the existing best performing method achieved 63% and 50% SR respectively. When trained with synthetic data5, our approach achieves slightly better performance on the seen environments and significantly better performance on both the validation unseen environments and the test unseen environments when submitted to the test server. We achieve 3% and 8% improvement on SR on both validation and test unseen environments. Both results with or without data augmentation indicate that our proposed approach is more generalizable to unseen environments. At the time of writing, our self-monitoring agent is ranked #1 on the challenge leader-board among the state of the arts.
Note that both Speaker-Follower and our approach in Table 1 use beam search. For comparison without using beam search, please refer to the Appendix.
Textually grounded agent. Intuitively, an instruction-following agent is required to strongly demonstrate the ability to correctly focus and follow the corresponding part of the instruction as it navigates through an environment. We thus record the distribution of attention weights on instruction at each step as indications of which parts of the instruction being used for action selection. We average all runs across both validation seen and unseen dataset splits. Ideally, we expect to see the distribution of attention weights lies close to a diagonal, where at the beginning, the agent focuses on the beginning of the instruction and shifts its attention towards the end of instruction as it moves closer to the goal.
To demonstrate, we use the method with panoramic action space proposed in Fried et al. (2018) as a baseline for comparison. As shown in Figure 3, our self-monitoring agent with progress monitor demonstrates that the positions of grounded instruction over time form a line similar to a diagonal. This result may further indicate that the agent successfully utilizes the attention on instruction to complete the task sequentially. We can also see that both agents were able to focus on the first part of the instruction at the beginning of navigation consistently. However, as the agent moves further in unknown environments, our self-monitoring agent can still successfully identify the parts of instruction that are potentially useful for action selection, whereas the baseline approach becomes uncertain about which part of the instruction should be used for selecting an action.
5We use the exact same synthetic data generated from the Speaker as in Fried et al. (2018) for comparison.

heading: 3.2 ABLATION STUDY

text:
We now discuss the importance of each component proposed in this work. We begin with the same baseline as before (agent with panoramic action space in Fried et al. (2018))6.
Co-grounding. When comparing the baseline with row #1 in our proposed method, we can see that our co-grounding agent outperformed the baseline with a large margin. This is due to the fact that we use the LSTM to carry both the textually and visually grounded content, and the decision on each navigable direction is predicted with both textually grounded instruction and the hidden state output of the LSTM. On the other hand, the baseline agent relies on the LSTM to carry visually grounded content, and uses the hidden state output for predicting the textually grounded instruction. As a result, we observed that instead of predicting the instruction needed for selecting a navigable direction, the textually grounded instruction may match with the past sequence of observed images implicitly saved within the LSTM.
Progress monitor. Given the effective co-grounding, the proposed progress monitor further ensure that the grounded instruction correctly reflects the progress made toward the goal. This further improves the performance especially on the unseen environments as we can see from row #1 and #2.
When using the progress inference, the progress monitor serve as a progress indicator for the agent to decide when to move back to the last viewpoint. We can see from row #2 and #4 that the SR performance can be further improved around 2% on both seen and unseen environments.
Finally, we integrate the output of the progress monitor with the state-factored beam search (Fried et al., 2018), so that the candidate paths compete not only based on the probability of selecting a certain navigable direction but also on the estimated correspondence between the past trajectory and the instruction. As we can see by comparing row #2, #6, and #7, the progress monitor significantly improved the success rate on both seen and unseen environments and is the key for surpassing the state of the arts even without data augmentation. We can also see that when using beam search without progress monitor, the SR on unseen improved 7% (row #1 vs #6), while using beam search integrated with progress estimation improved 13% (row #2 vs #7).
Data augmentation. In the above, we have shown each row in our approach contributes to the performance. Each of them increases the success rate and reduces the navigation error incrementally. By further combining them with the data augmentation pre-trained from the speaker (Fried et al., 2018), the SR and OSR are further increased, and the NE is also drastically reduced. Interestingly, the performance improvement introduced by data augmentation is smaller than from Speaker-Follower on the validation sets (see Table 1 for comparison). This demonstrates that our proposed method is more data-efficient.
6Note that our results for this baseline are slightly higher on val-seen and slightly lower on val-unseen than those reported, due to differences in hyper-parameter choices.

heading: 3.3 QUALITATIVE RESULTS

text:
To further validate the proposed method, we qualitatively show how the agent navigates through unseen environments by following instructions as shown in Fig. 4. In each figure, the agent follows the grounded instruction (at the top of the figure) and decides to move towards a certain direction (green arrow). For the full figures and more examples of successful and failed agents in both unseen and seen environments, please see the supplementary material.
Consider the trajectory on the left side in Fig. 4, at step 3, the grounded instruction illustrated that the agent just completed ”turn right” and focuses mainly on ”walk straight to bedroom”. As the agent entered the bedroom, it then shifts the textual grounding to the next action ”Turn left and walk to bed lamp”. Finally, at step 6, the agent completed another ”turn left” and successfully stop at the rug (see the supplementary material for the importance of dealing with duplicate actions). Consider the example on the right side, the agent has already entered the hallway and now turns right to walk across to another room. However, it is ambiguous that which room the instructor is referring to. At step 5, our agent checked out the room on the left first and realized that it does not match with ”Stop in doorway in front of rug”. It then moves to the next room and successfully stops at the goal.
In both cases, we can see that the completeness estimated by progress monitor gradually increases as the agent steadily navigates toward the goal. We have also observed that the estimated completeness ends up much lower for failure cases (see the supplementary material for further details).

heading: 4 RELATED WORK

text:
Vision, Language, and Navigation.. There is a plethora work investigating the combination of vision and language for a multitude of applications (Zhou et al., 2018a;b; Antol et al., 2015; Tapaswi et al., 2016; Das et al., 2017), etc. While success has been achieved in these tasks to handle massive corpora of static visual input and text data, a resurgence of interest focuses on equipping an agent with the ability to interact with its surrounding environment for a particular goal such as object manipulation with instructions (Misra et al., 2016; Arkin et al., 2017), grounded language acquisition (Al-Omari et al., 2017; Kollar et al., 2013; Spranger & Steels, 2015; Dubba et al., 2014),
embodied question answering (Das et al., 2018; Gordon et al., 2018), and navigation (Matuszek et al., 2013; Hemachandra et al., 2015; Duvallet et al., 2016; Zhu et al., 2017; de Vries et al., 2018; Yuke Zhu, 2017; Mousavian et al., 2018; Wayne et al., 2018; Wang et al., 2018a; Mirowski et al., 2017; 2018; Zamir et al., 2018). In this work, we concentrate on the recently proposed the Visionand-Language Navigation task (Anderson et al., 2018b)—asking an agent to carry out sophisticated natural-language instructions in a 3D environment. This task has application to fields such as robotics; in contrast to traditional map-based navigation systems, navigation with instructions provides a flexible way to generalize across different environments.
A few approaches have been proposed for the VLN task. For example, Anderson et al. (2018b) address the task in the form of a sequence-to-sequence translation model. Yu et al. (2018) introduce a guided feature transformation for textual grounding. Wang et al. (2018b) present a planned-head module by combing model-free and model-based reinforcement learning approaches. Recently, Fried et al. (2018) propose to train a speaker to synthesize new instructions for data augmentation and further use it for pragmatic inference to rank the candidate routes. These approaches leverage attentional mechanisms to select related words from a given instruction when choosing an action, but those agents are deployed to explore the environment without knowing about what progress has been made and how far away the goal is. In this paper, we propose a self-monitoring agent that performs co-grounding on both visual and textual inputs and constantly monitors its own progress toward the goal as a way of regularizing the textual grounding.
Visual and textual grounding. Visual grounding learns to localize the most relevant object or region in an image given linguistic descriptions, and has been demonstrated as an essential component for a variety of vision tasks like image captioning (Hu et al., 2016; Rohrbach et al., 2016; Lu et al., 2018), visual question answering (Lu et al., 2016b; Agrawal et al., 2018), relationship detection (Lu et al., 2016a; Ma et al., 2018) and referral expression (Nagaraja et al., 2016; Gavrilyuk et al., 2018). In contrast to identifying regions or objects, we perform visual grounding to locate relevant images (views) in a panoramic photo constructed by stitching multiple images with the aim of choosing which direction to go. Extensive efforts have been made to ground language instructions into a sequence of actions (MacMahon et al., 2006; Branavan et al., 2009; Vogel & Jurafsky, 2010; Tellex et al., 2011; Artzi & Zettlemoyer, 2013; Andreas & Klein, 2015; Mei et al., 2016; Cohn et al., 2016; Misra et al., 2017). These early approaches mainly emphasize the incorporation of structural alignment biases between the linguistic structure and sequence of actions (Mei et al., 2016; Andreas & Klein, 2015), and assume the agents are in relatively easy environment where limited visual perception is required to fulfill the instructions.

heading: 5 CONCLUSION

text: We introduce a self-monitoring agent which consists of two complementary modules: visual-textual co-grounding module and progress monitor. The visual-textual co-grounding module locates the instruction completed in the past, the instruction needed in the next action, and the moving direction from surrounding images. The progress monitor regularizes and ensures the grounded instruction correctly reflects the progress towards the goal by explicitly estimating the completeness of instruction-following. This estimation is conditioned on the positions and weights of grounded instruction. Our approach sets a new state-of-the-art performance on the standard Room-to-Room dataset on both seen and unseen environments. While we present one instantiation of self-monitoring for a decision-making agent, we believe that this concept can be applied to other domains as well.

heading: ACKNOWLEDGMENTS

text: This research was partially supported by DARPAs Lifelong Learning Machines (L2M) program, under Cooperative Agreement HR0011-18-2-001. We thank the authors from Fried et al. (2018), Ronghang Hu and Daniel Fried, for communicating with us and providing details of the implementation and synthetic instructions for fair comparison.

heading: SUPPLEMENTARY MATERIALS

heading: COMPARISON WITH PRIOR ART WITHOUT BEAM SEARCH

text: We provide the comparison with state of the arts without using beam search. The results are shown in Table 3. We can see that our proposed method outperformed existing approaches with a large margin on both validation unseen and test sets. Our method with greedy decoding for action selection improved the SR by 9% and 8% on validation unseen and test set. When using progress inference for action selection, the performance on the test set significantly improved by 5% compared to using greedy decoding, yielding 13% improvement over the best existing approach.

heading: IMPLEMENTATION DETAILS

text:
Image feature. Similar to previous work, we use the pre-trained ResNet-152 on ImageNet to extract image features. Each image feature is thus a 2048-d vector. The embedded feature vector for each navigable direction is obtained by concatenating an appearance feature with a 4-d orientation feature [sinφ; cosφ; sinθ; cosθ], where φ and θ are the heading and elevation angles. Following the work in Fried et al. (2018), the 4-dim orientation features are tiled 32 times, resulting a embedding feature vector with 2176 dimension.
Network architecture. The embedding dimension for encoding the navigation instruction is 256. We use a dropout layer with ratio 0.5 after the embedding layer. We then encode the instruction using a regular LSTM, and the hidden state is 512 dimensional. The MLP g used for projecting the raw image feature is BN −→ FC −→ BN −→ Dropout −→ ReLU . The FC layer projects the 2176-d input vector to a 1024-d vector, and the dropout ratio is set to be 0.5. The hidden state of the LSTM used for carrying the textual and visual information through time in Eq. 1 is 512. We set the maximum length of instruction to be 80, thus the dimension of the attention weights of textual grounding αt is also 80. The dimension of the learnable matrices from Eq. 2 to 5 are: Wx ∈ R512×512,Wv ∈ R512×1024,Wa ∈ R1024×1024,Wh ∈ R1536×512, andWpm ∈ R592×1. Training. We use ADAM as the optimizer. The learning rate is 1e − 4 with batch size of 64 consistently through out all experiments. When using beam search, we set the beam size to be 15. We perform categorical sampling during training for action selection.

heading: SUBMISSION TO VISION AND LANGUAGE NAVIGATION CHALLENGE

text:
For evaluating our proposed approach on the unseen test set, we participate in the Vision and Language Navigation challenge and submitted our result with the full proposed approach to the test server. We achieved 61% success rate and ranked #1 on the test server at the time of writing.
We follow the submission guidelines, where picking the highest confidence trajectory from multiple trials for each instruction is not permissible. This means that using the beam search for competing and selecting a final trajectory is not allow directly. Similar to the submission from SpeakerFollower (Fried et al., 2018), we record all the viewpoints traversed during the beam search process. The final agent traverses through all recorded trajectories by first reaching the end of one trajectory and backtracking to the shared viewpoint with the next trajectory. This means that the agent could backtrack to the start point during this process. The trajectories are however logged according to the
closest previous trajectory, so that when a single agent traverses through all recorded trajectories, the overhead for switching from one trajectory to another can be reduced significantly. The final selected trajectory from beam search is then lastly logged to the trajectory. This therefore yields exactly the same success rate and navigation error, as the metrics are computed according to the last viewpoint from a trajectory.

heading: QUALITATIVE RESULTS

text: We provide and discuss additional qualitative results on the self-monitoring agent navigating on seen and unseen environments. We first discuss four successful examples in Fig. 5 and 6, and followed by two failure examples in Fig. 7.

heading: SUCCESSFUL EXAMPLES

text:
In Fig. 5 (a), at the beginning, the agent mostly focuses on ”walk up” for making the first movement. While the agent keeps its attention on ”walk up” as completed instruction or ongoing action, it shifts the attention on instruction to ”turn right” as it walks up the stairs. Once it reached the top of the stairs, it decides to turn right according to the grounded instruction. Once turned right, we can again see that the agent pays attention on both the past action ”turn right” and next action ”walk straight to bedroom”. The agent continues to do so until it decides to stop by grounding on the word ”stop”.
In Fig. 5 (b), the agent starts by focusing on both ”enter bedroom from balcony” and ”turn left” to navigate. It correctly shifts the attention on textual grounding on the following instruction. Interestingly, the given instruction ”walk straight across rug to room” at step 3 is ambiguous since there are two rooms across the rug. Our agent decided to sneak out of the first room on the left and noticed that it does not match with the description from instruction. It then moved to another room across the rug and decided to stop because there is a rug inside the room as described.
In Fig. 6 (a), the given instruction is ambiguous as it only asks the agent to take actions around the stairs. Since there are multiple duplicated actions described in the instruction, e.g. ”walk up” and ”turn left”, only an agent that is able to precisely follow the instruction step-by-step can successfully complete the task. Otherwise, the agent is likely to stop early before it reaches the goal. The agent also needs to demonstrate its ability to assess the completeness of instruction-following task in order to correctly stop at the right amount of repeated actions as described in the instruction.
In Fig. 6 (b), at the beginning (step 0), the agent only focuses on ’left’ for making the first movement (the agent is originally facing the painting). We can see that at each step, the agent correctly focuses on parts of the instruction for making every movements, and it finally believes that the instruction is completed (attention on the last sentence period) and stopped.

heading: FAILURE EXAMPLES

text:
In Fig. 7 (a) step 1, although the attention on instruction correctly focused on ”take a left” and ”go down”, the agent failed to follow the instruction and was not able to complete the task. We can however see that the progress monitor correctly reflected that the agent did not follow the given instruction successfully. The agent ended up stopping with progress monitor reporting that only 16% of the instruction was completed.
In Fig. 7 (b) step 2, the attention on instruction only focuses on ”go down” and thus failed to associate the ”go down steps” with the stairs previously mentioned in ”turn right to stairs”. The agent was however able to follow the rest of the instruction correctly by turning right and stopping near a mirror. Note that, different from Fig. 7 (a), the final estimated completeness of instruction-following from progress monitor is much higher (16%), which indicates that the agent failed to be aware that it was not correctly following the instruction.

title: IARY PROGRESS ESTIMATION