summaries: This paper presents a new model-based reinforcement learning method, termed hindsight modelling. The method works by training a value function which, in addition to depending on information available at the present time is conditioned on some learned embedding of a partial future trajectory. A model is then trained to predict the learned embedding based on information available at the current time-step. This predicted value is fed in place of the actual embedding to the same value model, to generate a value prediction for the current time-step. So instead of just learning a value function based on future returns, the method uses a two-step process of learning an embedding of value relevant information from the future and then learns to predict that embedding. The paper proposes a way to learn better representation for RL by employing a hindsight model-based approach. The reasoning is that during training, we can observe the future trajectory and use features from it to better predict past values/returns. However to make this practical, the proposed approach fits an approximator to predict these features of the future trajectory from the current state and then subsequently, use them to predict the value. The authors claim that this extra information can be used to learn a better representation in some problems and lead to faster learning of good policies (or optimal value functions) Value-Driven Hindsight Modelling proposes a method to improve value function learning. The paper introduces the hindsight value function which estimates the expected return at a state conditioned on the future trajectory of the agent. How use this hindsight value function is not obvious, since an agent does not have access to the future states needed in order to take actions (for Q-Learning) and the hindsight value function is a biased gradient estimator for training policy gradient methods. 
abstractText: Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn predictors for value from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future but have to compose with a potentially weak scalar signal (an estimate of the return). In this paper we develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end we determine which features of the future trajectory provide useful information to predict the associated return. This provides us with tractable prediction targets that are directly relevant for a task, and can thus accelerate learning of the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.
authors.name: HINDSIGHT MODELLING
references.authors: Marc G Bellemare Yavar Naddaf Joel Veness Michael Bowling
references.title: The arcade learning environment: An evaluation platform for general agents
references.venue: Journal of Artificial Intelligence Research,
references.authors: Lars Buesing Theophane Weber Yori Zwols Sebastien Racaniere Arthur Guez Jean-Baptiste Lespiau Nicolas Heess
references.title: Woulda, coulda, shoulda: Counterfactually-guided policy search
references.venue: arXiv preprint arXiv:1811.06272,
references.authors: Lasse Espeholt Hubert Soyer Remi Munos Karen Simonyan Volodymir Mnih Tom Ward Yotam Doron Vlad Firoiu Tim Harley Iain Dunning
references.title: Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures
references.venue: arXiv preprint arXiv:1802.01561,
references.authors: Carles Gelada Saurabh Kumar Jacob Buckman Ofir Nachum Marc G Bellemare
references.title: Deepmdp: Learning continuous latent space models for representation learning
references.authors: Arthur Guez Mehdi Mirza Karol Gregor Rishabh Kabra Sebastien Racaniere Theophane Weber David Raposo Adam Santoro Laurent Orseau Tom Eccles
references.title: An investigation of model-free planning
references.venue: In International Conference on Machine Learning,
references.authors: Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine
references.title: Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor
references.venue: arXiv preprint arXiv:1801.01290,
references.authors: Max Jaderberg Volodymyr Mnih Wojciech Marian Czarnecki Tom Schaul Joel Z Leibo David Silver Koray Kavukcuoglu
references.title: Reinforcement learning with unsupervised auxiliary tasks
references.venue: arXiv preprint arXiv:1611.05397,
references.authors: Steven Kapturowski Georg Ostrovski Will Dabney John Quan Remi Munos
references.title: Recurrent experience replay in distributed reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A Rusu Joel Veness Marc G Bellemare Alex Graves Martin Riedmiller Andreas K Fidjeland Georg Ostrovski
references.title: Human-level control through deep reinforcement learning
references.authors: Lerrel Pinto Marcin Andrychowicz Peter Welinder Wojciech Zaremba Pieter Abbeel
references.title: Asymmetric actor critic for image-based robot learning
references.venue: arXiv preprint arXiv:1710.06542,
references.authors: John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy optimization algorithms
references.venue: arXiv preprint arXiv:1707.06347,
references.authors: Richard S Sutton Andrew G Barto
references.title: Reinforcement learning: An introduction
references.authors: Richard S Sutton Joseph Modayil Michael Delp Thomas Degris Patrick M Pilarski Adam White Doina Precup
references.title: Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction
references.venue: In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume
references.authors: Erik Talvitie
references.title: Model regularization for stable sample rollouts
references.venue: In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence,
references.authors: Hado Van Hasselt Arthur Guez David Silver
references.title: Deep reinforcement learning with double qlearning
references.venue: In Thirtieth AAAI conference on artificial intelligence,
references.authors: Vladimir Vapnik Rauf Izmailov
references.title: Learning using privileged information: similarity control and knowledge transfer
references.venue: Journal of machine learning research,
references.authors: Yuke Zhu Ziyu Wang Josh Merel Andrei A. Rusu Tom Erez Serkan Cabi Saran Tunyasuvunakool János Kramár Raia Hadsell Nando de Freitas Nicolas Heess
references.title: Reinforcement and imitation learning for diverse visuomotor
references.venue: skills. CoRR,
sections.text: Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn predictors for value from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future but have to compose with a potentially weak scalar signal (an estimate of the return). In this paper we develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end we determine which features of the future trajectory provide useful information to predict the associated return. This provides us with tractable prediction targets that are directly relevant for a task, and can thus accelerate learning of the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.
sections.heading: 1 INTRODUCTION
sections.text: Consider a baseball player trying to perfect their pitch. The player performs an arm motion and releases the ball towards the batter, but suppose that instead of observing where the ball lands and the reaction of the batter, the player only gets told the result of the play in terms of points or, worse, only gets told the final result of the game. Improving their pitch from this experience appears hard and inefficient, yet this is essentially the paradigm we employ when optimizing policies in model-free reinforcement learning. The scalar feedback that estimates the return from a state (and action), encoding how well things went, drives the learning while the accompanying observations that may explain that result (e.g. flight path of the ball or the way the batter anticipated and struck the incoming baseball) are ignored. To intuitively understand how such information could help value prediction, consider a simple discrete Markov chain X → Y → Z, where Z is the scalar return and X is the observation from which we are trying to predict Z. If the space of possible values of Y is smaller than X , then it may be more efficient to estimate both P (Y |X) and P (Z|Y ) rather than directly estimating P (Z|X).1 In other words observing and then predicting Y can be advantageous to directly estimating the signal of interestZ. Model-based RL approaches would duly exploit the observed Y (by modeling the transition Y |X), but Y would, in general scenarios, contain information that is irrelevant to Z and hard to predict. Building a full high-dimensional predictive model to indiscriminately estimate all possible future observations, including potentially chaotic details of the ball trajectory and the spectators’ response, is a challenge that may not pay off if the task-relevant predictions (e.g., was the throw accepted, was the batter surprised) are error-ridden. Model-free RL methods directly consider the relation X to Z, and focus solely upon predicting and optimising this goal, rather than attempting to learn the full dynamics. These methods have recently dominated the literature, and have attained the best performance in a wide array of complex problems with high-dimensional observations (Mnih et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018; Guez et al., 2019).
  1In the discrete case, this follows from a counting argument from the size of the probability tables involved.
  In this paper, we propose to augment model-free methods with a lightweight model of future quantities of interest. The motivation is to model only those parts of the future observations (Y ) that are needed to obtain better value predictions. The major research challenge is to learn, from observational data, which aspects of the future are important to model (i.e. what Y should be). To this end, we propose to learn a special value function in hindsight that receives future observations as an additional input. This learning process reveals features of the future observations that would be most useful for value prediction (e.g. flight path of the ball or the reaction of the batter), if provided by an oracle. These important features are then predicted, in advance, using only information available at test time (at the time of releasing the baseball, we knew the identity of the batter, the type of throw and spin given to the ball). Learning these value-relevant features can help representation learning for an agent and provide an additional useful input to its value and policy. Experimentally, hindsight value functions surpassed the performance of model-free RL methods in a challenging association task (Portal Choice). When hindsight value functions were added to the prior state-of-the-art RL method for Atari games, they significantly increased median performance from 833% to 965%.
sections.heading: 2 BACKGROUND AND NOTATION
sections.text: We consider a reinforcement learning setting whereby an agent learns from interaction in a sequential decision-making environment (Sutton & Barto, 2011). An agent’s policy π, mapping states to an action distribution, is executed to obtain a sequence of rewards and observations as follows. At each step t, after observing state st, the policy outputs an action at, sampled from π(A|st), and obtains a scalar reward rt and the next-state st+1 from the environment. The sum of discounted rewards from state s is the return denoted by G = ∑∞ t=0 γ
  tRt, with γ < 1 denoting the discount factor. Its expectation, as a function of the starting state, is called the value function, vπ(s) = Eπ[G|S0 = s]. An important related quantity is the action-value, or Q-value, which corresponds to the same expectation with a particular action executed first: qπ(s, a) = Eπ[G|S0 = s,A0 = a]. The learning problem consists in adapting the policy π in order to achieve a higher value vπ . This usually entails learning an estimate of vπ for the current policy π, this is the problem we focus on in this paper.
  Note that in practice we are interested in partially-observed environments where the state of the world is not directly accessible. For this case, we can think of replacing the observed state s in the case of the fully-observed case by a learned function that depends on past observations.
sections.heading: 3 VALUE LEARNING
sections.heading: 3.1 DIRECT LEARNING
sections.text: A common approach to estimate v (or q) is to represent it as a parametric function vθ (or qθ) and directly update its parameters based on sample returns of the policy of interest. Value-based RL algorithms vary in how they construct a value target Y from a single trajectory. They may regress vθ towards the Monte-Carlo return (Yt = Gt), or exploit sequentiality in the reward process by relying on a form of temporal-difference learning to reduce variance (e.g. the TD(0) target Yt = Rt + γvθ(St+1)). For a given target definition Y , the value loss Lv to derive an update for θ is: Lv(θ) = 12Es[(vθ(s) − Y )
  2]. In constructing a target Yt based on a trajectory of observations and rewards from time t, the observations are either unused (for a Monte Carlo return) or only indirectly exploited (when bootstrapping to obtain a value estimate). In all cases, the trajectory is distilled into a scalar signal that estimates the return of a policy, and other relevant aspects of future observations are discarded. In particular in partially observed domains or domains with highdimensional observation spaces it can be difficult to discover correlations with this noisy signal.
sections.heading: 3.2 MODEL-BASED APPROACH
sections.text: An indirect way of estimating the value is to first learn a model of the dynamics. For example a 1-step observation model mθ learns to predict the conditional distribution st+1, rt|st, at. Then a value estimate v(s) for state s can be obtained by autoregressively rolling out the model (until the end of the episode or to a fixed depth with a parametric value bootstrap).
  The model is trained on potentially much richer data than the return signal since it exploits all information in the trajectory. Indeed, the observed transitions between states can reveal the structure behind a sparse reward signal. A drawback of classic model-based approaches is that they predict a high-dimensional signal, a task which may be costly and harder than directly predicting the scalar value . As a result, the approximation of the dynamics mθ may contain errors where it matters most for predicting the value (Talvitie, 2014). Although the observations carry all the data from the environment, most of it is not essential to the task (Gelada et al., 2019). The concern that modeling all observations is expensive also applies when the model is not used for actual rollouts but merely for representation learning. So while classic model-based methods fully use this highdimensional signal at some cost, model-free methods take the other extreme to focus only on the most relevant low-dimensional signal (the scalar return). Below we propose a method that strikes a balance between these paradigms.
sections.heading: 3.3 HINDSIGHT VALUE AND MODEL
sections.text: We introduce a new value function estimate that can only be computed at training time, the hindsight value function v+. This value still estimates the expected return from a state st but it is further conditioned on k additional observations τ+t = st+1, st+2, . . . st+k occurring after time t: 2
  v+(st, τ + t ) ≈ E[G|S0 = st, . . . , Sk = st+k]. (1)
  Furthermore, we require v+ to follow this particular parametric structure:
  v+(st, τ + t ; θ) = ψθ1(f(st),φθ2(τ + t )), (2)
  where θ = (θ1, θ2), which forces information about the future trajectory through some vector-valued function φ ∈ Rd. Intuitively, v+ is estimating the expected return from a past time point using privileged access to future observations. Note that if k is large enough, then v+ simply estimates the empirical return from time t given access to the state trajectory. However, if k is small and φ is lowdimensional, then φ becomes a bottleneck representation of the future trajectory τ+t . By learning in hindsight, we identify features that are maximally useful to predict the return on the trajectory from time t. The hindsight value function is not a useful quantity by itself, since – because of its use of privileged future observations – we cannot readily use it at test time. Furthermore, it cannot be used as a baseline either, as when computing the policy gradient it will yield a biased gradient estimator. . Instead, the idea is to learn a model φ̂ of φ, that can be used at test time. We conjecture that if privileged features φ are useful for estimating the value, then the model of those features will also be useful for estimating the value function. We propose to learn the approximate expectation model φ̂η2(s) conditioned on the current state s and parametrized by η2, minimizing the following squared loss:
  Lmodel(η2) = Es,τ+ [‖φθ2(s, τ+)− φ̂η2(s)‖22] (3)
  where the expectation is taken over the distribution of states and partial trajectories τ+ resulting from that state.
  The approximate model φ̂ can then be leveraged to obtain a better model-based value estimate vm(s; η) = ψη1(f(s), φ̂η2(s)). Although φ̂(s) cannot contain more information than included already in the state s, it can still benefit from having being trained using a richer signal before the value converges. Figure 3 summarizes the relation between the different quantities.
sections.heading: 3.4 ILLUSTRATIVE EXAMPLE
sections.text: We consider the following example to illustrate how the approaches to estimating the value function can differ. There are no actions in this example3 and each episode consists of a single transition from initial state s to terminal state s′, with a reward r(s, s′) on the way.
  Each instance of this example is parametrized by a square matrix W and a vector b sampled from a unit normal distribution, which determine the uncontrolled MDP. Initial states s are of dimension D and sampled from a multivariate unit normal distribution (si ∼ N(0, 1) for all state dimension i).
  2In general τ+t can be defined to include full future transitions, actions and rewards. 3This can be understood as a Markov Reward Process or a policy evaluation setting
  Given s = ( s1s2 ), where s1 and s2 are of dimension D1 and D2 (D = D1 + D2), the next state s′ = ( s′1 s′2 ) is determined according to the transition function: s′1 = MLP(s)+ and s ′ 2 = σ(Ws2+ b) where σ is the Heaviside function. s′1 acts as a distractor here, with additive noise ∼ N(0, 1). The reward obtained is r(s, s′) = ∑ i s (i) 1 ∑ i s ′(i) 2 / √ D. The true value in the start state is also v(s) = r(s, s′).
  The key aspect of this domain is that s′ reveals structure that helps predict the value function in the start state s. This is made visually obvious in the trajectories sampled in this domain shown in Figure 1.
  Let us consider how the different approaches to learning values presented above fare in this problem. For direct learning, the value from v(s′) is 0 since s′ is terminal, so any n-step return is identical to the Monte-Carlo return, that is, the information present in observation s′ is not leveraged. Results from learning v from s given the return is presented in Figure 2 (blue curve). A model-based approach first predicts s′ from s, then attempts to predict the value given s and the estimated next state. When increasing the input dimension, given a fixed capacity, the model does not focus its attention on the reward-relevant structure in s′ and makes error where it matters most. As a result, it can struggle to learn v faster than a model-free estimate (cf. red curve in Figure 2). When learning in hindsight, v+ can directly exploit the revealed structure in the observation of τ+, and as a result the hindsight value learns faster than the regular causal model-free estimate (cf. dotted yellow curve in Figure 2). This drives the learning of φ and its model φ̂, which directly gets trained to predict these useful features for the value. As a result, vm also benefits and learns faster than the regular v estimate on this problem (cf. green curve in Figure 2).
sections.heading: 3.5 WHEN IS IT ADVANTAGEOUS TO MODEL IN HINDSIGHT?
sections.text: To understand the circumstances in which hindsight modelling provides a better value estimate, we first consider an analysis that relies on the following assumptions. Suppose that vmθ is sharing the same function ψ as v+ (i.e., θ1 = η1), and let ψ be linear. If we write ψθ1(f,φ) = ( ω1 ω2 ) > ( f φ ) + b,
  where θ1 = (ω1, ω2), then we have for fixed values of the parameters:
  E[(vm(s; η)− v+(s, τ+; θ)2] = E[‖ω>2 (φ(τ+; θ2)− φ̂(s; η2))‖2] (4) ≤ E[‖ω2‖2‖φ(τ+)− φ̂(s)‖2] (5) = ‖ω2‖2Lmodel(η2), (6)
  using the Cauchy-Schwarz inequality. Let L define the value error for a particular value function v: L(v) = E[(v(s)−G)2] and L(v+) = E[(v+(s, τ+)−G)2]. Then we have:
  L(vm) = E[(vm(s)− v+(s, τ+) + v+(s, τ+)−G)2] (7) ≤ 2(‖ω2‖2Lmodel(η2) + L(v+)), (8)
  using the fact that E[(X +Y )2] ≤ 2(E[X2] +E[Y 2]) for random variables X and Y . If we assume L(v+) = CL(v) with C < 0.5 (i.e., estimating the value in hindsight with more information is an easier learning problem), then the following holds:
  Lmodel(η2) < (1− 2C)L(v)
  2‖ω2‖2 =⇒ L(vm) < L(v). (9)
  In other words, this relates how small the modeling error needs to be to guarantee that the value error for vm is smaller than the value error for the direct estimate v. The modeling error can be large for different reasons. If the environment or the policy is stochastic, then there is some irreducible modeling error for the deterministic model. Even in these cases, a small C can make hindsight modeling advantageous. The modeling error could also be high because predicting φ is hard. For example, it could be that φ essentially encodes the empirical return, which means predicting φ is at least as hard as predicting the value function (Lmodel(η2) ≥ L(v)). Or it could be that φ is highdimensional, this could cause both a hard prediction problem but also would cause the acceptable threshold for Lmodel to decrease (since ‖θ2‖2 will grow). We address some of these concerns with specific architectural choices like v+ having a limited view on future observations and having low dimensional φ (see next section). Note that the analysis above ignores any advantage that could be obtained from representation learning when training φ̂ (if the state encoding function f shares parameters with φ̂).
sections.heading: 4 ARCHITECTURE
sections.text: The architecture for Hindsight Modelling (HiMo) we found to work at scale and tested in the experimental section of the paper is described here. To deal with partial observability, we employ a recurrent neural network, the state-RNN, which replaces the state st with a learned internal state ht, a function of the current observation ot and past observations through ht−1: ht = f(ot, ht−1; η3), where we have extended the parameter description of vm as η = (η2, η1, η3). The model-based value function vm and the hindsight value function v+ share the same internal state representation h, but the learning of v+ assumes h is fixed (we do not backpropagate through the state-RNN in hindsight). In addition, we force φ̂ to only be learned through Lmodel, so that vm uses it as an additional input.
  To summarize:
  v+(ht, ht+k; θ) = ψθ1(ht,φθ2(ht+k)), (10)
  vm(ht; η) = ψη1(ht, φ̂η2(ht)), (11)
  with the bar notation denoting quantities treated as non-differentiable (i.e. where the gradient is stopped). The different losses in the HiMo architecture are combined in the following way:
  L(θ, η) = Lv(η) + αLv+(θ) + βLmodel(η). (12)
  A diagram of the architecture is presented in Figure 3, and further implementation details can be found in the appendix.
  This architecture can be straightforwardly generalized to cases where we also output a policy πη for an actor-critic setup, providing h and φ̂ as inputs to a policy network.4 For a Q-value based algorithm like Q-learning, we predict a vector of values qm and q+ instead of vm and v+. Computing v+ and training φ̂ can be done in an online fashion by simply delaying the updates by k steps (just like the computation of an n-step return).
sections.heading: 5 EXPERIMENTS
sections.text: The illustrative example in Section 3.4 demonstrated the positive effect of hindsight modeling in a simple policy evaluation setting. In this section, we now explore these benefits in the context of policy optimization in challenging domains, a custom navigation task called Portal Choice, and Atari 2600. To demonstrate the generality and scalability of our approach we test hindsight value functions in the context of two high-performance RL algorithms, IMPALA (Espeholt et al., 2018) and R2D2 (Kapturowski et al., 2019).
sections.heading: 5.1 PORTAL CHOICE TASK
sections.text: The Portal Choice (Fig. 4) is a two-phase navigation task where, in phase one an agent is presented with a contextual choice between two portals, whose positions vary between episodes. The position of the portal determines its destination in phase two, one of two different goal rooms (green and red rooms). Critically, the reward when terminating the episode in the goal room depends on both the color of the goal room in phase two and a visually indicated combinatorial context shown in the first phase. If the context matches the goal room color, then a reward of 2 is given, otherwise the reward is 0 when terminating the episode (see appendix for the detailed description).
  An easy suboptimal solution is to select the portal at random and finish the episode in the resulting goal room by reaching the goal pixel, which will result in a positive reward of 1 on average. A more
  4In this case, the total loss also contains an actor loss to update πη and a negative entropy loss.
  difficult strategy is to be selective about which portal to take depending on the context, in order to get the reward of 2 at each every episode. A model-free agent has to learn the joint mapping from contexts and portal positions to rewards. Even if the task is not visually complex, the context is combinatorial in nature (the agent needs to count randomly placed pixels) and the joint configuration space of context and portal is fairly large (around 250M). Since the mapping from portal position to rooms does not depend on context, learning the portal-room mapping independently is more efficient in this scenario.
  For this domain, we implemented the HiMo architecture within a distributed actor-critic agent, named IMPALA proposed by Espeholt et al. (2018). In this case, the target Yt to train vm (used as a critic in this context) and v+ is the V-trace target (Espeholt et al., 2018) to account for offpolicy corrections between the behavior policy and the learner policy. The actor shares the same network as the critic and receives h and φ̂ as inputs.
  We found that HiMo+IMPALA learned reliably faster to reach the optimal behavior, compared to the vanilla IMPALA baseline that shared the same network capacity (see Figure 5a). The hindsight value v+ rapidly learns to predict whether the portal-context association is rewarding based on seeing the goal room color in the future. Then φ learns to predict the new information from the future that helps that prediction: the identity of the room (see analysis Fig 5c). The prediction of φ becomes effectively a model of the mapping from portal to room identity (since the context does not correlate with the room identity). Having access to such mapping through φ̂ helped the value prediction (Fig 5b), which led to better action selection. Note that if the two rooms were visually indistinguishable, for example with no red/green rooms separation, HiMo would not be able to offer any advantage over its model-free counterpart.
sections.heading: 5.2 ATARI
sections.text: We tested our approach in Atari 2600 videogames using the Arcade Learning Environment (Bellemare et al., 2013). We added HiMo on top of Recurrent Replay Distributed DQN (R2D2), a DQN-
  based distributed architecture introduced by Kapturowski et al. (2019) which achieved state-of-theart scores in Atari games.
  In this value-based setting, HiMo trains qm(·, ·; η) and q+(·, ·; θ) based on n-step return targets:
  Yt = g ( n−1∑ m=0 γmRt+m + γ ng−1 ( qm(St+n, A ∗; η−) )) , (13)
  where g is an invertible function, η− are the periodically updated target network parameters (as in DQN by Mnih et al. (2015)), andA∗ = argmaxa q
  m(St+n, a; η) (the Double DQN update proposed by Van Hasselt et al. (2016)). The details of the architecture and hyperparameters are described in the appendix.
  We ran our approach on 57 Atari games for 200k gradient steps (around 1 day of training), with 3 seeds for each game. The evaluation averages the score between 200 episodes across seeds, each lasting a maximum of 30 minutes each and starting a random number (up to 30) of no-op actions. In order to compare scores between different games and aggregated results, we computed normalized scores for each game based on random and human performance so that 0% corresponds to random performance and 100% corresponds to human. We observed an increase of 132.5% in the me-
  dian human normalized score compared to the R2D2 baseline with the same network capacity, aggregate results are reported in Table 1. Figure 6 details the difference in normalized score between HiMo and our R2D2 baseline for all games individually. We note that the original R2D2 results reported by Kapturowski et al. (2019), which used a similar hardware configuration but a different network architecture, were around 750% median human normalized score after a day of training.
  In our experimental evaluation we observed that HiMo typically either offers improved data efficiency or has no overwhelming adverse effects in training performance. In Figure 6 we show training curves for a selection of representative Atari environments where at evaluation time HiMo both under-performed (left) and out-performed R2D2 (right); these seem to indicate that in the worst case scenario HiMo’s training performance reduces to R2D2’s.5
  Bowling is one of the Atari games where rewards are delayed with relevant information being communicated through intermediate observations (the ball hitting the pins), just like the baseball example
  5We will include a performance analysis over longer training regimes in a future version of the paper.
  we have used in the introduction. We found HiMo to perform better than the R2D2 baseline in this particular game. We also ran HiMo with the actor-critic setup (IMPALA) described in the previous section, finding similar performance gain with respect to the model-free baseline. Learning curves for these experiments are presented in Figure 7.
sections.heading: 6 RELATED WORK
sections.text: Recent work have used auxiliary predictions successfully in RL as a mean to obtain a richer signal for representation learning (Jaderberg et al., 2016; Sutton et al., 2011). However these additional prediction tasks are hard-coded and so they cannot adapt to the task demand when needed. We see them as a complementary approach to more efficient learning in RL.
  Buesing et al. (2018) have considered using observations in an episode trajectory in hindsight to infer variables in a structural causal model of the dynamics, allowing to reason more efficiently in a model-based way about counterfactual actions. However this approach requires learning an accurate generative model of the environment.
  In supervised learning, the learning using privileged information (LUPI) framework introduced by (Vapnik & Izmailov, 2015) considers ways of leveraging privileged information at train time. Although the techniques developed in that work do not apply directly in the RL setting, some of our approach can be understood in that setting as considering the future trajectory as the privileged information for a value prediction problem.
  Privileged information coming from full state observation has been leveraged in RL to learn better critic in asymmetric actor-critic architectures (Pinto et al., 2017; Zhu et al., 2018). However this does not use future information and only applies to settings where special side-information (full state) is available at train time.
sections.heading: 7 CONCLUSION
sections.text: High-dimensional observations in the intermediate future often contain task-relevant features that can facilitate the prediction of an RL agent’s final return. We introduced a reinforcement learning algorithm, HiMo, that leverages this insight by the following two-stage approach. First, by reasoning in hindsight, the algorithm learns to extract relevant features of future observations that would be been most helpful for estimating the final value. Then, a forward model is learned to predict these features, which in turn is used as input to an improved value function, yielding better policy evaluation and training at test time. We demonstrated that this approach can help tame complexity in environments with rich dynamics at scale, yielding increased data efficiency and improving the performance of state-of-the-art model-free architectures.
sections.heading: A APPENDIX
sections.text: A.1 GENERAL ARCHITECTURE DETAILS
  To compute v+ and train φ̂ in an online fashion, we process fixed-length unrolls of the state-RNN and compute the hindsight value and corresponding updates at time t if t+k is also within that same unroll. Also, we update v+ at a slower rate (i.e., α < β) to give enough time for the model φ̂ to adapt to the changing hindsight features φ. In our experiments we found that even a low-dimensional φ (in the order of d = 3) and a relatively short hindsight horizon k (in the order of 5) are sufficient to yield significant performance boosts, whilst keeping the extra model computational costs modest.
  A.2 PORTAL CHOICE
  Environment The observation is a 7× 23 RGB frame (see Figure 4). There are 3 possible spawning points for the agent in the center and 42 possible portal positions (half of which lead to the green room, the other half leading to the red room). At the start of an episode, two portals, each leading to a different room, are chosen are random. They are both displayed as cyan pixels. Included in the observation in the first phase is the context, a random permutation in a 5 × 5 grid of N pixels, where is uniformly sampled at the start of each episode: N ∼ U{1, 10}. A fixed map f : {1, . . . , 10} → {0, 1} determines which contexts are rewarding with the green room, the rest being rewarding with the red room. The reward when reaching the goal is determined according to:
  R = 2(f(N)G+ (1− f(N))(1−G)), (14)
  where G ∈ {0, 1} is whether the reached room is green.
  Network architecture The policy and value network takes in the observation and passes it to a ConvNet encoder (with filter channels [32, 32, 32], kernel shapes [4, 3, 3] applied with strides [2, 1, 1]) before being passed to a ConvLSTM network with 32 channels and 3x3 filters. The output of the ConvLSTM is the internal state h. The φ̂ network is a ConvNet with [32, 32, 32, 1] filter channels with kernels of size 3 except for a final 1x1 filter, whose output is flatten and passed to an MLP with 256 hidden units with ReLu activation, before a linear layer with dimension d = 3. The φ network is a similarly configured network with one less convolution layer and 128 hidden units in the MLP. The ψη network is an MLP with 256 hidden units followed by a linear layer that takes h and φ̂ as input and outputs the policy πm and the value vm. v+ is obtained similarly with a similar MLP that has a single scalar output. We used a future observation window of k = 5 steps in this domain and loss weights α = 0.25, β = 0.5. Unroll length was 20, and γ = 0.99. Optimization was done with the Adam optimizer (learning rate of 5e − 4), with batch size 32. The model-free baseline is obtained by using the same code and network architecture, and setting the modeling loss and hindsight value loss to 0 (α = β = 0).
  A.3 ATARI
  Hyper-parameters and infrastructure are the same as reported in Kapturowski et al. (2019), with deviations as listed in table 2. For our value target, we also average different n-step returns with exponential averaging as in Q(λ) (with the return being truncated at the end of unrolls). The Q network is composed of a convolution network (cf. Vision ConvNet in table) which is followed by an LSTM with 512 hidden units. What we refer to in the main text as the internal state h is the output of the LSTM. The φ and φ̂ networks are MLPs with a single hidden layer of 256 units and ReLu activation function, followed by a linear which outputs a vector of dimension d. The ψθ1 function concatenates h and φ as inputs to an MLP with 256 hidden units with ReLu activation function, followed by a linear which outputs q+ (a vector of dimension 18, the size of the Atari action set). qm is obtained by passing h and φ̂ to a dueling network as described by Kapturowski et al. (2019).
  Other HiMo parameters are described in table 3. The R2D2 baseline with the same capacity is obtained by running the same architecture with α = β = 0.