summaries: The paper presents a novel state representation technique that is based on Value Functions (VFs). The core idea of the paper is to use VFs to construct a high-level space representation in an hierarchical reinforcement learning (RL) scenario. The contributions of the paper are:  - Value Function Spaces (VFS): learned state representation - Practical algorithms with VFS both for model-free and model-based RL settings - Evaluation of VFS in 2 scenarios (MiniGrid and manipulation task) This paper proposes that given a set of skills or options, the vector of the value functions for the individual skills becomes the abstract state representation in the approach proposed by this paper. The authors then go on to develop a model-free RL algorithm akin to DQN and a model-based planning algorithm for planning in the state-space defined by the value functions of the component skills, and the action-space being the set of executable options.   The authors demonstrate that the value function space serves as a better representation than some baseline algorithms for representation learning for RL in a maze solving and robotic manipulation task.  The key assumptions made are as follows:  - The options are pre-defined and each option has an associated value function  - The new task is solvable with the defined set of options. The paper proposes a state abstraction in hierarchical reinforcement learning (HRL), called Value Function Spaces (VFS), that is constructed from the value functions of task-conditioned lower-level policies (or skills). The idea is that the value functions capture affordances of the lower-level skills while ignoring task-irrelevant information. The lower-level policies and the corresponding value functions are given a prior, and the state is constructed by concatenating the value estimate of each lower-level policy. The state abstraction is evaluated in a model-free Q-learning and model-based MPC type scenario. This paper presents Value Function Spaces (VFS), an abstract "skill-centric" representation for reinforcement learning and long-horizon planning. The core idea of VFSs is to leverage the learned value functions that are often trained alongside skills during skill learning to abstract knowledge about the world. Each element of this low-dimensional representation corresponds to the value estimated from one of the learned value function, so that the representation (by construction) tends to ignore distractor information unnecessary for accomplishing the skills. Using this representation on a set of pre-trained skills, the authors demonstrate improved performance on long-horizon planning tasks in grid-based maze and "locked door" environments, showing improved success probability on a number of such tasks compared to competitive baselines. They also show the ability of their abstraction to facilitate model-based goal-directed planning by learning a state-transition model (in VFS-space) that allows one to predict the outcome of executing a skill.
abstractText: Reinforcement learning can train policies that effectively perform complex tasks. However for long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and composing lower-level skills. Hierarchical reinforcement learning aims to enable this by providing a bank of lowlevel skills as action abstractions. Hierarchies can further improve on this by abstracting the space states as well. We posit that a suitable state abstraction should depend on the capabilities of the available lower-level policies. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill. These value functions capture the affordances of the scene, thus forming a representation that compactly abstracts task relevant information and robustly ignores distractors. Empirical evaluations for maze-solving and robotic manipulation tasks demonstrate that our approach improves long-horizon performance and enables better zero-shot generalization than alternative model-free and model-based methods.
authors.name: LONG-HORIZON REASONING
references.authors: Rishabh Agarwal Marlos C. Machado Pablo Samuel Castro Marc G Bellemare
references.title: Contrastive behavioral similarity embeddings for generalization in reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Marcin Andrychowicz Filip Wolski Alex Ray Jonas Schneider Rachel Fong Peter Welinder Bob McGrew Josh Tobin OpenAI Pieter Abbeel Wojciech Zaremba
references.title: Hindsight experience replay
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Pierre-Luc Bacon Jean Harb Doina Precup
references.title: The option-critic architecture
references.venue: In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
references.authors: Ershad Banijamali Rui Shu Mohammad Ghavamzadeh Hung Hai Bui Ali Ghodsi
references.title: Robust locally-linear controllable embedding
references.venue: In AISTATS,
references.authors: Andre Barreto Diana Borsa Shaobo Hou Gheorghe Comanici Eser Aygün Philippe Hamel Daniel Toyama Jonathan hunt Shibl Mourad David Silver Doina Precup
references.title: The option keyboard: Combining skills in reinforcement learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Marc G. Bellemare Will Dabney Rémi Munos
references.title: A distributional perspective on reinforcement learning
references.venue: In Proceedings of the 34th International Conference on Machine Learning,
references.authors: Pablo Samuel Castro
references.title: Scalable methods for computing state similarity in deterministic markov decision processes
references.venue: In AAAI,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton
references.title: A simple framework for contrastive learning of visual representations
references.authors: Maxime Chevalier-Boisvert Lucas Willems Suman Pal
references.title: Minimalistic gridworld environment for openai gym, 2018
references.venue: URL https://github.com/maximecb/gym-minigrid
references.authors: Petros Christodoulou
references.title: Soft actor-critic for discrete action settings. CoRR, 2019
references.venue: URL http: //arxiv.org/abs/1910.07207
references.authors: Kurtland Chua Roberto Calandra Rowan McAllister Sergey Levine
references.title: Deep reinforcement learning in a handful of trials using probabilistic dynamics models
references.venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,
references.authors: Dane Corneil Wulfram Gerstner Johanni Brea
references.title: Efficient model-based deep reinforcement learning with variational state tabulation
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Özgür Şimşek Andrew G. Barto
references.title: Using relative novelty to identify useful temporal abstractions in reinforcement learning
references.venue: In Proceedings of the Twenty-First International Conference on Machine Learning,
references.authors: Özgür Şimşek Alicia P. Wolfe Andrew G. Barto
references.title: Identifying useful subgoals in reinforcement learning by local graph partitioning
references.venue: In Proceedings of the 22nd International Conference on Machine Learning,
references.authors: Marco da Silva Frédo Durand Jovan Popović
references.title: Linear bellman combination for control of character animation
references.venue: ACM Trans. Graph.,
references.authors: Christian Daniel Gerhard Neumann Oliver Kroemer Jan Peters
references.title: Hierarchical relative entropy policy search
references.venue: Journal of Machine Learning Research,
references.authors: Christoph Dann Gerhard Neumann Jan Peters
references.title: Policy evaluation with temporal differences: A survey and comparison
references.venue: Journal of Machine Learning Research,
references.authors: Thomas G. Dietterich
references.title: Hierarchical reinforcement learning with the maxq value function decomposition
references.venue: J. Artif. Int. Res.,
references.authors: Debidatta Dwibedi Jonathan Tompson Corey Lynch Pierre Sermanet
references.title: Learning actionable representations from visual observations
references.venue: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
references.authors: Frederik Ebert Chelsea Finn Alex X. Lee Sergey Levine
references.title: Self-supervised visual planning with temporal skip connections
references.venue: In Annual Conference on Robot Learning, Proceedings of Machine Learning Research,
references.authors: Ben Eysenbach Russ R Salakhutdinov Sergey Levine
references.title: Search on the replay buffer: Bridging planning and reinforcement learning
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Chelsea Finn Sergey Levine
references.title: Deep visual foresight for planning robot motion
references.venue: In ICRA,
references.authors: Carlos Florensa Yan Duan Pieter Abbeel
references.title: Stochastic neural networks for hierarchical reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Roy Fox Sanjay Krishnan Ion Stoica Ken Goldberg
references.title: Multi-level discovery of deep options. CoRR, 2017
references.venue: URL http://arxiv.org/abs/1703.08294
references.authors: Dibya Ghosh Abhishek Gupta Sergey Levine
references.title: Learning actionable representations with goal conditioned policies
references.venue: In ICLR,
references.authors: James J. Gibson
references.title: The theory of affordances. In Perceiving, acting, and knowing: toward an ecological psychology
references.authors: Nakul Gopalan Marie desJardins Michael L. Littman James MacGlashan S. Squire Stefanie Tellex John Winder Lawson L.S. Wong
references.title: Planning with abstract markov decision processes
references.venue: In ICAPS,
references.authors: Anirudh Goyal Riashat Islam DJ Strouse Zafarali Ahmed Hugo Larochelle Matthew Botvinick Sergey Levine Yoshua Bengio
references.title: Transfer and exploration via the information bottleneck
references.venue: In International Conference on Learning Representations,
references.authors: Daniel Graves Johannes Günther Jun Luo
references.title: Affordance as general value function: A computational model
references.authors: Karol Gregor Danilo Jimenez Rezende Frederic Besse Yan Wu Hamza Merzic Aäron van den Oord
references.title: Shaping Belief States with Generative Environment Models for RL
references.authors: Abhishek Gupta Coline Devin Yuxuan Liu Pieter Abbeel Sergey Levine
references.title: Learning invariant feature spaces to transfer skills with reinforcement learning
references.venue: In 5th International Conference on Learning Representations,
references.authors: Tuomas Haarnoja Vitchyr Pong Aurick Zhou Murtaza Dalal Pieter Abbeel Sergey Levine
references.title: Composable deep reinforcement learning for robotic manipulation
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Danijar Hafner Timothy Lillicrap Ian Fischer Ruben Villegas David Ha Honglak Lee James Davidson
references.title: Learning latent dynamics for planning from pixels
references.venue: arXiv preprint arXiv:1811.04551,
references.authors: Danijar Hafner Timothy Lillicrap Jimmy Ba Mohammad Norouzi
references.title: Dream to control: Learning behaviors by latent imagination
references.venue: In International Conference on Learning Representations,
references.authors: Hado van Hasselt Arthur Guez David Silver
references.title: Deep reinforcement learning with double qlearning
references.venue: In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence,
references.authors: Matteo Hessel Joseph Modayil Hado van Hasselt Tom Schaul Georg Ostrovski Will Dabney Dan Horgan Bilal Piot Mohammad Azar David Silver
references.title: Rainbow: Combining improvements in deep reinforcement learning
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Irina Higgins Arka Pal Andrei A. Rusu Loı̈c Matthey Christopher P. Burgess Alexander Pritzel Matthew M. Botvinick Charles Blundell Alexander Lerchner. Darla
references.title: Improving zero-shot transfer in reinforcement
references.venue: learning. ArXiv,
references.authors: Manfred Huber Roderic A. Grupen
references.title: Learning robot control - using control policies as abstract actions
references.venue: Workshop: Abstraction and Hierarchy in Reinforcement Learning,
references.authors: Jonathan J Hunt Andre Barreto Timothy P Lillicrap Nicolas Heess
references.title: Composing entropic policies using divergence correction, 2019
references.venue: URL https://openreview.net/forum?id= SJ4Z72Rctm
references.authors: Brian Ichter Marco Pavone
references.title: Robot motion planning in learned latent spaces
references.venue: IEEE Robotics and Automation Letters,
references.authors: Brian Ichter Pierre Sermanet Corey Lynch
references.title: Broadly-exploring, local-policy trees for longhorizon task planning
references.venue: In Annual Conference on Robot Learning,
references.authors: Max Jaderberg Volodymyr Mnih Wojciech Marian Czarnecki Tom Schaul Joel Z. Leibo David Silver Koray Kavukcuoglu
references.title: Reinforcement learning with unsupervised auxiliary tasks
references.venue: In International Conference on Learning Representations. OpenReview.net,
references.authors: Eric Jang Alex Irpan Mohi Khansari Daniel Kappler Frederik Ebert Corey Lynch Sergey Levine Chelsea Finn
references.title: BC-z: Zero-shot task generalization with robotic imitation learning
references.venue: In 5th Annual Conference on Robot Learning,
references.authors: Dmitry Kalashnikov Jacob Varley Yevgen Chebotar Benjamin Swanson Rico Jonschkowski Chelsea Finn Sergey Levine Karol Hausman
references.title: Mt-opt: Continuous multi-task robotic reinforcement learning at scale
references.authors: Martin Klissarov Pierre-Luc Bacon Jean Harb Doina Precup
references.title: Learnings options end-to-end for continuous action
references.venue: tasks. CoRR,
references.authors: Levente Kocsis Csaba Szepesvári
references.title: Bandit based monte-carlo planning
references.venue: In Machine Learning: ECML 2006,
references.authors: George Konidaris Andrew Barto
references.title: Skill discovery in continuous reinforcement learning domains using skill chaining
references.venue: Advances in Neural Information Processing Systems,
references.authors: George Konidaris Leslie Kaelbling Tomas Lozano-Perez
references.title: Constructing symbolic representations for high-level planning
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Ramnandan Krishnamurthy Aravind S. Lakshminarayanan Peeyush Kumar Balaraman Ravindran
references.title: Hierarchical reinforcement learning using spatio-temporal abstractions and deep neural networks. 2016
references.venue: URL http://arxiv.org/abs/1605.05359
references.authors: Tejas D. Kulkarni Karthik R. Narasimhan Ardavan Saeedi Joshua B. Tenenbaum
references.title: Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: S. Lange Martin A. Riedmiller Arne Voigtländer
references.title: Autonomous reinforcement learning on raw visual input data in a real world application
references.venue: The 2012 International Joint Conference on Neural Networks (IJCNN),
references.authors: Michael Laskin Aravind Srinivas Pieter Abbeel
references.title: CURL: Contrastive unsupervised representations for reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Alex X. Lee Anusha Nagabandi Pieter Abbeel Sergey Levine
references.title: Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Manfred Otto Heess Tom Erez Yuval Tassa David Silver Daan Wierstra
references.title: Continuous control with deep reinforcement learning
references.authors: Kendall Lowrey Aravind Rajeswaran Sham Kakade Emanuel Todorov Igor Mordatch
references.title: Plan online, learn offline: Efficient learning and exploration via model-based control
references.venue: In International Conference on Learning Representations,
references.authors: Priyanka Mandikal Kristen Grauman
references.title: Learning dexterous grasping with object-centric visual affordances
references.authors: Amy McGovern Andrew G. Barto
references.title: Automatic discovery of subgoals in reinforcement learning using diverse density
references.venue: In Proceedings of the Eighteenth International Conference on Machine Learning,
references.authors: Ishai Menache Shie Mannor Nahum Shimkin
references.title: Q-cut - dynamic discovery of sub-goals in reinforcement learning
references.venue: In Machine Learning: ECML 2002, 13th European Conference on Machine Learning,
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A. Rusu Joel Veness Marc G. Bellemare Alex Graves Martin Riedmiller Andreas K. Fidjeland Georg Ostrovski Stig Petersen Charles Beattie Amir Sadik Ioannis Antonoglou Helen King Dharshan Kumaran Daan Wierstra Shane Legg Demis Hassabis
references.title: Human-level control through deep reinforcement learning
references.authors: Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.venue: In ICML,
references.authors: Ofir Nachum Haoran Tang Xingyu Lu Shixiang Gu Honglak Lee Sergey Levine
references.title: Why does hierarchy (sometimes) work so well in reinforcement
references.authors: Ashvin Nair Vitchyr Pong Murtaza Dalal Shikhar Bahl Steven Lin Sergey Levine
references.title: Visual reinforcement learning with imagined goals
references.venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,
references.authors: Junhyuk Oh Satinder Singh Honglak Lee
references.title: Value prediction network
references.venue: Advances in Neural Information Processing Systems,
references.authors: Doina Precup
references.title: Temporal abstraction in reinforcement learning
references.authors: Anil V. Rao
references.title: A survey of numerical methods for optimal control
references.authors: Matthew Riemer Miao Liu Gerald Tesauro
references.title: Learning abstract options
references.venue: In Advances in Neural Information Processing Systems, 2018a. URL https://proceedings.neurips.cc/
references.authors: Matthew Riemer Miao Liu Gerald Tesauro
references.title: Learning abstract options
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Mark Sandler Andrew Howard Menglong Zhu Andrey Zhmoginov Liang-Chieh Chen
references.title: Mobilenetv2: Inverted residuals and linear bottlenecks
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Nikolay Savinov Alexey Dosovitskiy Vladlen Koltun
references.title: Semi-parametric topological memory for navigation
references.venue: In International Conference on Learning Representations,
references.authors: Tom Schaul Daniel Horgan Karol Gregor David Silver
references.title: Universal value function approximators
references.venue: In Proceedings of the 32nd International Conference on Machine Learning,
references.authors: Jürgen Schmidhuber
references.title: Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments
references.venue: Technical report,
references.authors: Florian Schroff Dmitry Kalenichenko James Philbin
references.title: Facenet: A unified embedding for face recognition and clustering
references.venue: IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Pierre Sermanet Corey Lynch Jasmine Hsu Sergey Levine
references.title: Time-contrastive networks: Selfsupervised learning from multi-view observation
references.venue: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW),
references.authors: Archit Sharma Shixiang Gu Sergey Levine Vikash Kumar Karol Hausman
references.title: Dynamics-aware unsupervised discovery of skills
references.venue: In International Conference on Learning Representations,
references.authors: Evan Shelhamer Parsa Mahmoudieh Max Argus Trevor Darrell
references.title: Loss is its own reward: Self-supervision for reinforcement learning. CoRR, 2016
references.authors: Freek Stulp Stefan Schaal
references.title: Hierarchical reinforcement learning with movement primitives
references.venue: In 2011 11th IEEE-RAS International Conference on Humanoid Robots,
references.authors: Richard S. Sutton Andrew G. Barto
references.title: Reinforcement Learning: An Introduction. 2018
references.venue: URL http://incompleteideas.net/book/the-book-2nd.html
references.authors: Richard S. Sutton Doina Precup Satinder Singh
references.title: Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
references.venue: Artificial Intelligence,
references.authors: Chen Tessler Shahar Givony Tom Zahavy Daniel J. Mankowitz Shie Mannor
references.title: A deep hierarchical approach to lifelong learning in minecraft
references.venue: In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
references.authors: Philip S. Thomas Andrew G. Barto
references.title: Motor primitive discovery
references.venue: IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL),
references.authors: Saket Tiwari Philip S. Thomas
references.title: Natural option critic
references.venue: Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Emanuel Todorov
references.title: Compositionality of optimal control laws
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Emre Ugur Erol Şahin Ethan Oztop
references.title: Predicting future object states using learned affordances
references.authors: Aäron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation learning with contrastive predictive coding. CoRR, 2018
references.venue: URL http://arxiv.org/abs/1807.03748
references.authors: Laurens van der Maaten Geoffrey Hinton
references.title: Visualizing data using t-SNE
references.venue: Journal of Machine Learning Research,
references.authors: David Warde-Farley Tom Van de Wiele Tejas D. Kulkarni Catalin Ionescu Steven Hansen Volodymyr Mnih
references.title: Unsupervised control through non-parametric discriminative rewards
references.authors: Manuel Watter Jost Tobias Springenberg Joschka Boedecker Martin Riedmiller
references.title: Embed to control: A locally linear latent dynamics model for control from raw images
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Grady Williams Andrew Aldrich Evangelos A. Theodorou
references.title: Model predictive path integral control using covariance variable importance sampling
references.venue: URL http://arxiv. org/abs/1509.01149
references.authors: Danfei Xu Ajay Mandlekar Roberto Martı́n-Martı́n Yuke Zhu Silvio Savarese Li Fei-Fei
references.title: Deep affordance foresight: Planning through what can be done in the future, 2021
references.authors: Denis Yarats Amy Zhang Ilya Kostrikov Brandon Amos Joelle Pineau Rob Fergus
references.title: Improving sample efficiency in model-free reinforcement learning from images
references.venue: CoRR, abs/1910.01741,
references.authors: Philipp Zech Simon Haller Safoura Rezapour Lakani Barry Ridge Emre Ugur Justus Piater
references.title: Computational models of affordance in robotics: a taxonomy and systematic classification
references.venue: Adaptive Behavior,
sections.heading: 1 INTRODUCTION
sections.text: For an agent to perform complex tasks in realistic environments, it must be able to effectively reason over long horizons, and to parse high-dimensional observations to infer the contents of a scene and its affordances. It can do so by constructing a compact representation that is robust to distractors and suitable for planning and control. Consider, for instance, a robot rearranging objects on a desk. To succcessfully solve the task, the robot must learn to sequence a series of simple skills, such as picking and placing objects and opening drawers, and interpret its observations to determine which skills are most appropriate. This requires the ability to understand the capabilities of these simpler skills, as well as the ability to plan to execute them in the correct order.
  Hierarchical reinforcement learning (HRL) aims to enable this by leveraging abstraction, which simplifies the higher-level control or planning problem. Typically, this is taken to mean abstraction of actions in the form of primitive skills (e.g., options (Sutton et al., 1999)). However, significantly simplifying the problem for the higher level requires abstraction of both states and actions. This is particularly important with rich sensory observations, where standard options frameworks provide a greatly abstracted state space, but do not simplify the perception or estimation problem. The nature of the ideal state abstraction in HRL is closely tied to the action abstraction, as the most suitable abstraction of state should depend on the kinds of decisions that the higher-level policy needs to make, which in turn depends on the actions (skills) available to it. This presents a challenge in designing HRL methods, because it is difficult to devise a state abstraction that is both highly abstracted (and therefore removes many distractors) and still sufficient to make decisions for long-horizon tasks. This challenge differs markedly from representation learning problems in other domains, like computer vision and unsupervised learning (Schroff et al., 2015; van den Oord et al., 2018; Chen et al., 2020), since it is intimately tied to the capabilities exposed to the agent via its skills.
  We therefore posit that a suitable representation for a higher-level policy in HRL should depend on the capabilities of the skills available to it. If this representation is sufficient to determine the abilities and outcomes of these skills, then the high-level policy can select the skill most likely to perform the desired task. This concept is closely tied to the notion of affordances, which has long been studied in cognitive science and psychology as an action-centric representation of state (Gibson, 1977), and has inspired techniques in robotics and RL (Zech et al., 2017; Xu et al., 2021; Mandikal & Grauman,
  Open Drawer Pick Blue Place Blue in Drawer Close Drawer
  = = = = =
  t0 t3t2t1 t4
  (a) A trajectory through the skill value function space for the task “Place blue block in drawer”. VFS, visualized on top of corresponding scene, captures positional information about the contents of the scene, preconditions for interactions, and the effects of executing a feasible skill, making it suitable for high-level planning.
  Open Drawer Pick Blue Place Blue in Drawer Close Drawer
  = = = = =
  (b) VFS learns a skill-centric representation of the scene and ignores factors like arm pose, task-centric object positions, color of the desk, or additional objects, which do not affect the values of the low-level skills. All configurations shown above are functionally equivalent and hence, map to the same VFS representation.
  Figure 1: Visualizing VFS embeddings in an example desk rearrangement task. VFS can capture the affordances of the low-level skills while ignoring exogenous distractors.
  2021). Given a set of skills that span the possible interactions in an environment, we propose that the value functions corresponding to these policies can provide a representation that suitably captures the capabilities of the skills in the current state and thus can be used to form a compact embedding space for high-level planning. We call this embedding space Value Function Spaces (VFS). Note that we intend to use these skills as high-level actions without modifications – the important problems of skill discovery and test-time adaptation are beyond the scope of our work.
  Figure 1a illustrates the state abstraction constructed by VFS for the desk rearrangement example discussed above: VFS captures the affordances of the skills and represents the state of the environment, along with preconditions for the low-level skills, forming a functional representation to plan over. Since VFS constructs a skill-centric representation of states using the value functions of the low-level skills, it captures functional equivalence of states in terms of their affordances: in Figure 1b, states with varying object or arm positions (i-iii), different background textures (iv), and distractor objects (v) are functionally equivalent for planning, and map to the same embedding in our representation. This significantly simplifies the high-level planning problem, making it easier for the higher level policy to generalize to novel environments to the limit of the skills themselves.
  Statement of Contributions. The primary contribution of this work is VFS, a novel state representation derived from the value functions of low-level skills available to the agent. We show that VFS leverages the properties of value functions, notably their ability to model possible skills and completed skills, to form an effective representation for high-level planning, and is compatible with both model-free and model-based high-level policies. Empirical evaluations in maze-solving and robotic manipulation demonstrate that the skill-centric representation constructed by VFS outperforms representations learned using contrastive and information-theoretic objectives in long-horizon tasks. We also show that VFS can generalize to novel environments in a zero-shot manner.
sections.heading: 2 RELATED WORK
sections.text: Hierarchical RL has been studied extensively in the literature, commonly interpreted as a temporal abstraction of the original MDP. Early works have interpreted the hierarchy introduced in this setting as an abstraction of state and action spaces (Sutton et al., 1999; Dietterich, 2000; Thomas & Barto, 2012). The popular options framework Sutton et al. (1999); Precup (2000) provides a natural way of incorporating temporally extended actions into RL systems. An agent that possesses the transition model and reward model for such a SMDP (known as an option model) is capable of sample-based planning in discrete (Kocsis & Szepesvári, 2006) and continuous state spaces (Konidaris et al., 2014; Gopalan et al., 2017). However, doing so in environments with high-dimensional observations (such
  as images) is challenging. In this work, we explore the efficacy of learned skills operating on highdimensional observations for long-horizon control in realistic environments.
  To improve the quality of lower-level policies, recent work in HRL has studied various facets of the problem, including discovery of skills (Konidaris & Barto, 2009; Zhang et al., 2021b; Florensa et al., 2017; Warde-Farley et al., 2018), end-to-end training of both levels (Kulkarni et al., 2016; Tessler et al., 2017) and integrating goal-conditioned behaviors (Ghosh et al., 2019; Nachum et al., 2019). In this work, we assume that the low-level skills are given, and do not focus on discovering them. Instead, we focus on how skills can simplify the higher-level control problem by providing a representation that is suitable for either model-based or model-free control. The high-level policy does not rely on any hand-crafted rewards and only receives a sparse outcome reward.
  The problem of learning meaningful long-horizon behavior by combining a set of lower-level skills, or options, has been studied extensively in prior work. Popular approaches to composing skills involve combining them in the space of value functions (Todorov, 2009; Hunt et al., 2019; da Silva et al., 2009; Ziebart, 2010; Haarnoja et al., 2018), and more recently, using pseudo-rewards (Barreto et al., 2019). While these methods have been very effective at composing low-level skills, they do not address the question of learning a meaningful state abstraction using these skills. We take inspiration from these works and use the value function space to construct a skill-centric state abstraction, along with a high-level policy that composes these skills for temporally extended tasks.
  Our method can hence be interpreted as a representation learning approach. Representation learning techniques have been employed extensively in model-free RL by augmenting auxiliary tasks based on reconstruction losses (Lange et al., 2012; Higgins et al., 2017; Yarats et al., 2019) and predicting the future conditioned on past observations (Schmidhuber, 1990; Jaderberg et al., 2017; van den Oord et al., 2018; Shelhamer et al., 2016). Contrastive learning has also been used in recent works to discover a meaningful latent space and extract reward signals for RL (Sermanet et al., 2017; Warde-Farley et al., 2018; Dwibedi et al., 2018; Laskin et al., 2020). Another popular approach to learning useful representation is to optimize for effective transfer of the lower-level skills and the learned state abstraction (Gupta et al., 2017; Konidaris & Barto; Goyal et al., 2019). Unlike these prior works, our aim is specifically to learn a representation that is grounded in the capabilities of the low-level skills, which gives us a skill-centric abstraction of high-dimensional observations. While we do not explicitly optimize for transfer like Gupta et al. (2017), our experiments show that VFS can compactly represent affordances of the scene and generalize to novel environments while being robust to functionally-irrelevant distractors.
  Alongside developments in model-free RL, prior work has also sought to learn predictive models of the environment for sampling and planning, in a model-based RL framework. This has been demonstrated by learning dynamics using future predictions (Watter et al., 2015; Oh et al., 2017; Ebert et al., 2017; Banijamali et al., 2018; Ha & Schmidhuber; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang et al., 2019), learning belief representations (Gregor et al., 2019; Lee et al., 2019) and representing state similarity using the bisimulation metric (Castro, 2020; Zhang et al., 2021a; Agarwal et al., 2021). The combination of learned model-free policies with structures like graphs (Savinov et al., 2018; Eysenbach et al., 2019) and trees (Ichter et al., 2021) to plan over extended horizons has also been demonstrated to improve generalization and exploration. Although our method can utilize a model-based high-level controller to plan over temporally extended skills, our objective is not to develop better model-based RL algorithms. Rather, we focus on developing suitable state abstractions in a hierarchical RL framework, and then evaluate these representations by employing them in both model-based and model-free higher-level controllers.
sections.heading: 3 PRELIMINARIES
sections.text: We assume that an agent has access to a finite set of temporally extended options O, or skills, which it can sequence to solve long-horizon tasks. These skills can be trained for a wide range of tasks by using manual reward specification (Huber & Grupen, 1998; Stulp & Schaal, 2011), using relabeling techniques (Andrychowicz et al., 2017), or via unsupervised skill discovery (Daniel et al., 2016; Fox et al., 2017; Warde-Farley et al., 2018; Sharma et al., 2020). In this work, the skills are learned via RL with sparse, hand-specified reward functions, but in general could come from any source. Since many prior works have focused on skill discovery, we do not explicitly address how these skills are produced, but assume that they are given.
  We assume that each skill has a maximum rollout length of τ time steps, after which it is terminated. We also assume that each skill oi ∈ O is accompanied by a critic, or value function, Voi denoting the expected cumulative skill reward executing skill oi from current state st. This is generally available for policies trained with RL (Huber & Grupen, 1998; Andrychowicz et al., 2017; Kalashnikov et al., 2021) or learned via options discovery (Bacon et al., 2017; Klissarov et al., 2017; Tiwari & Thomas, 2019; Riemer et al., 2018b). While this may not be true for skills discovered by other methods, such as segmentation, graph partioning or bottleneck discovery (McGovern & Barto, 2001; Menache et al., 2002; Krishnamurthy et al., 2016; Şimşek & Barto, 2004; Şimşek et al., 2005; Riemer et al., 2018a), we can use policy evaluation to automatically obtain a value function – this can be done via regression to empirical returns or temporal difference learning (Dann et al., 2014).
  Our setting is closely related to the options framework of Sutton et al. (1999). Options are skills that consist of three components: a policy π : S × A → [0, 1], a termination condition β : S+ → [0, 1], and an initiation set I ⊆ S , where S,A are the low-level state and action spaces in a fully observable decision process. An option 〈I, π, β〉 is available at state st if and only if st ∈ I. We do not assume we have an initiation set, and show that the value functions provide this information. We assume that the policies come with a termination condition, or alternatively, have a fixed horizon τ . Next, we describe a framework to formulate a decision problem that uses these options for planning.
  We assume that the low-level observation and action space of the agent can be described as a fully observable semi-Markov decision process (SMDP)M described by a tuple (S,O,R, P, τ, γ), where S ⊆ Rn is the n-dimensional continuous state space; O is a finite set of temporally extended skills, or options, with a maximum horizon of τ time steps; R(s′|s, oi) is the task reward received after executing the skill oi ∈ O at state s ∈ S; P (s′|s, oi) is a PDF describing the probability of arriving in state s′ ∈ S after executing skill oi ∈ O at state s ∈ S; γ ∈ (0, 1] is the discount factor. Given a finite set of options, our objective is to obtain a high-level decision-policy that selects among them, resulting in a sequence of options that reach the desired goal.
  Note that generally, many image-based problems are partially observed so that the entire history of observation-action pairs may be required to describe the state. Explicitly addressing partial observability is outside the scope of our work, so we follow the convention in prior work by assuming full observability and refer to images as states (Lillicrap et al., 2016; Nair et al., 2018).
sections.heading: 4 SKILL VALUE FUNCTIONS AS STATE SPACES
sections.text: The notion of value in RL is closely related to affordances, in that the value function predicts the capabilities of the skill being learned. The supervised learning problem of affordance prediction (Ugur et al., 2009) can in fact be cast as a special case of value prediction (Sutton & Barto, 2018; Graves et al., 2020). In this section, we construct a skill-centric representation of state derived from skill value functions that captures the affordances of the low-level skills, and empirically show that this representation is effective for high-level planning.
  Given an SMDP M(S,O,R, P, τ, γ) with a finite set of k skills oi ∈ O trained with sparse outcome rewards and their corresponding value functions Voi , we construct an embedding space Z by stacking these skill value functions. This gives us an abstract representation that maps a state st to a k-dimensional representation Z(st) := [Vo1(st), Vo2(st), ..., Vok(st)], which we call the Value Function Spaces, or VFS for short. This representation captures functional information about the exhaustive set of interactions that the agent can have with the environment by means of executing the skills, and is thus a suitable state abstraction for downstream tasks.
  Figure 1(a) illustrates the proposed state abstraction for a conceptual desk rearrangement task rollout with eight low-level skills. A raw observation, such as an image of the robotic arm and desk, is abstracted by a 8-dimensional tuple of value functions of the available skills. This representation captures positional information about the contents of the scene (e.g., both blocks are on the counter and drawer is closed since the corresponding values are 1 at t0), preconditions for interactions (e.g., both blocks can be lifted since the “pick” values are high at t0), and the effects of executing a feasible skill (e.g., the value corresponding to the “Open Drawer” skill increases on executing it at t1), making it suitable for high-level planning.
  We hypothesize that since VFS learns a skill-centric representation of the scene, it is robust to exogenous factors of variation, such as background distractors and appearance of task-irrelevant
  components of the scene. This also enables VFS to generalize to novel environments with the same set of low-level skill, which we demonstrate empirically. Figure 1(b) illustrates this for the desk rearrangement task. All configurations shown are functionally equivalent – an open drawer with the red cube in it, blue cylinder on the countertop, empty robot arm – and can be interacted with identically. Intuitively, VFS would map these configurations to the same abstract state by ignoring factors like arm pose, task-centric position of objects, color of the desk, or additional objects, which do not affect the low-level skills.
sections.heading: 5 MODEL-FREE RL WITH VALUE FUNCTION SPACES
sections.text: In this section, we instantiate a hierarchical model-free RL algorithm that uses VFS as the state abstraction and the skills as the low-level actions. We compare the long-horizon performance of VFS to alternate representations for HRL trained with constrastive and information theoretic objectives for the task of maze-solving and find that VFS outperforms the next best baseline by up to 54% on the most challenging tasks. Lastly, we compare the zero-shot generalization performance of these representations and empirically demonstrate that the skill-centric representation constructed by our method can successfully generalize to novel environments with the same set of low-level skills.
sections.heading: 5.1 AN ALGORITHM FOR HIERARCHICAL RL
sections.text: We instantiate a hierarchical RL algorithm that learns a Q-function Q(Z, o) at the high-level using VFS as the “state” Z and the skills oi ∈ O as the temporally extended actions. Given such a Qfunction, a greedy high-level policy can be obtained by πQ(Z) = argmaxoi∈O Q(Z, oi). We use DQN (Mnih et al., 2015), which uses mini-batches sampled from a replay buffer of transition tuples (Zt, ot, rt, Zt+1) to train the learned Q-function to satisfy the Bellman equation. This is done using gradient descent on the loss L = E (Q(Zt, ot)− yt)2, where yt = rt + γmaxot′∈O Q(Zt+1, ot′). In order to make the optimization problem more stable, the targets yt are computed using a separate target network which is update at a slower pace than the main network. Q-learning also suffers from an overestimation bias, due to the maximization step above, and harms learning. Hasselt et al. (2016) address this overestimation by decoupling the selection of action from its evaluation. We use this variant, called DDQN, for subsequent evaluations in this work. Note that we are using the skills as given, without updating or finetuning them.
  While there have been several improvements to DDQN (Hessel et al., 2018) and alternative algorithms for model-free RL in discrete action spaces (Schaul et al., 2015; Bellemare et al., 2017; Christodoulou, 2019), and these improvements will certainly improve the overall performance of our algorithm, our goal is to evaluate the efficacy of VFS as a state representation and we study it in the context of a simple DDQN pipeline.
sections.heading: 5.2 EVALUATING LONG-HORIZON PERFORMANCE IN MAZE-SOLVING
sections.text: To evaluate the long-horizon performance of VFS against commonly used representation learning methods, we use the versatile MiniGrid environment (Chevalier-Boisvert et al., 2018) in a fully observable setting, where the agent receives a top-down view of the environment rather than an egocentric local image We consider two tasks: (i) MultiRoom, where the agent is tasked with reaching the goal by solving a variable-sized maze spanning up to 10 rooms. The agent must cross each room and open the door to access the following rooms; (ii) KeyCorridor, where the agent is tasked with reaching a goal up to 7 rooms away and may face locked doors. The agent must find the key corresponding to a color-coded door and open it to access subsequent rooms. Both these tasks have a sparse reward that is only provided for successfully reaching the goal. This presents a challenging domain for long-horizon sequential reasoning, where tasks may require over 200 time steps to succeed, making a great testbed for evaluating the ability of the state abstractions to capture relevant information for sequencing multiple skills. The agents have access to the following temporally extended skills – GoToObject, PickupObject, DropObject and UnlockDoor – where the first three skills are text-conditioned and Objectmay refer to a door, key, box or circle of any color. Since MiniGrid is easily reconfigurable, we also generate a set of holdout MultiRoom mazes with different grid layouts to evaluate the zero-shot generalization performance of the policies trained with these representations. Note that the grid layouts, as well as object positions, for these tasks
  are randomly generated for every experiment and are not static. Example grid layouts are shown in Figure 2. We provide further implementation details in Appendix A.1.
  Baselines: We compare VFS extensively against a variety of competitive baselines for representation learning in RL using contrastive and information-theoretic objectives. We consider representations learned both offline and online (in loop with RL); note that VFS is constructed entirely from the values of the available skills and is not learned. Further, all baselines have access to these skills.
  1. Raw Observations: We train a high-level policy operating on raw input observations.
  2. Autoencoder (AE): We use an autoencoder to extract a compact latent space using a reconstruction loss on an offline dataset of trajectories, similar to Lange et al. (2012).
  3. Contrastive Predicting Coding (CPC): We learn a representation by optimizing the InfoNCE loss over an offline dataset of trajectories (van den Oord et al., 2018).
  4. Online Variational Autoencoder (VAE): We learn a VAE representation jointly (online) with the high-level policy operating on this representation (Yarats et al., 2019).
  5. Contrastive Unsupervised Representations for RL (CURL): We learn a representation by optimizing a contrastive loss jointly (online) with the RL objective (Laskin et al., 2020).
  Evaluation: We run experiments for the two tasks with varying number of rooms, to study the performance of the algorithms with increasing time horizon, and report the success rates in Table 1. HRL from raw observations demonstrates a success rate of 64% in the two-room environment, which can be attributed to the powerful set of skills available to the high-level policy, but this quickly drops to 29% in the largest environment. The offline baselines (AE and CPC) construct a compact state abstraction, which makes the learning problem easier for DDQN, and show significant improvements in smaller environments (MultiRoom-2,4 and KeyCorridor-3). However, they are unable to improve the performance in larger environments. We hypothesize that this is due to the inability of the representations to capture information necessary for the high-level policy, since they are learned independent of the controller. The performance of representations learned online (VAE and CURL), which are implemented analogous to their offline counterparts (AE and CPC, respectively) support this hypothesis and improves the performance across all tasks by learning representations jointly with the controller, and scoring up to 54% in the most challenging environment. We hypothesize that the limited performance of these methods is due to the lack of direct influence of the downstream task on the representation. Despite being constructed offline, VFS explicitly captures the capabilities of the low-level skills and provides an action-centric representation for the high-level policy, greatly simplifying the control problem. This is reflected in the performance of VFS across all tasks – it outperforms all baselines, scoring 98% on the simplest task and up to 68% on the most challenging task, beating the next best method by over 25%. Figure 2 shows sample rollouts of the high-level policy using VFS as the state representation in the two tasks discussed.
sections.heading: 5.3 ZERO-SHOT GENERALIZATION
sections.text: We evaluate the generalization abilities of these representations by training them for the KeyCorridor task (as above) and evaluating on MultiRoom. Since the skills available in MultiRoom are a subset of those in KeyCorridor, we ignore any invalid actions executed by the agent. Note that the high-level agent has not been trained in MultiRoom and the policies are not updated in the target environments.
  Table 2 shows the success rates of the representations in the MultiRoom tasks with 4 (MR4) and 10 rooms (MR10). Unsurprisingly, HRL with raw observations fails to generalize, because the maze layout differs significantly (e.g. see Figure 3). AE and CPC learn a compact representation from high-dimensional observations and allow the high-level policy to generalize to simpler tasks like MR4 and achieve up to 47% success rate. Interestingly, their online counterparts (VAE and CURL) also fail to generalize and perform poorly, likely because representations learned jointly with the RL policy can overfit to the source environment. VFS learns a skill-centric representation that can generalize zero-shot to tasks that use the same skills, and thus outperforms the next best baseline by up to 180%, closely matching the performance of an HRL policy trained from scratch in the target environment with online interaction (HRL-Target).
sections.heading: 6 MODEL-BASED PLANNING WITH VALUE FUNCTION SPACES
sections.text: In this section, we introduce a simple model-based RL algorithm that uses VFS as the “state” for planning, which we term VFS-MB. We study the performance of VFS in the context of a robotic manipulation task, and compare it to alternate representations for model-based RL. We find that the performance of VFS as an abstract representation for raw image observations outperforms all baselines and closely matches that of a pipeline with access to oracular state of the simulator, showing the efficacy of our method in long-horizon planning.
sections.heading: 6.1 A SIMPLE ALGORITHM FOR MODEL-BASED RL
sections.text: We use a simple model-based planner that learns a one-step predictive model, using VFS as the “state.” Specifically, this model learns the transition dynamics Zt+1 = f̂(Zt, ot)∀ ot ∈ O via supervised learning using a dataset of prior interactions in the environment. Note that the predictive model (and the subscript of Z) is over high-level policy steps, which may be up to τ steps of the lowlevel skills. This dataset can be collected simply by executing the available skills in the environment for τ steps, where τ is the maximum horizon of the SMDPM, or until termination.
  In order to use the learned model f̂(Zt, ot), a goal latent state Zg , and a scoring function (e.g. mean squared error) for the high-level task, we need to solve the following optimization problem for the optimal sequence of skills (ot, . . . , ot+H−1) to reach the goal:
  (ot, . . . , ot+H−1) = argmin ot,...,ot+H−1 (Ẑt+H , Zg) : Ẑt = Zt, Ẑt′+1 = f̂(Ẑt′ , ot′) (1)
  We use a sampling-based method to find solutions to the above equation. We use random shooting (Rao, 2009) to randomly generate K candidate option sequences, predict the corresponding Z sequences using the learned model f̂ , compute the rewards for all sequences, and pick the candidate action sequence leading to a latent state closest to the goal, according to Equation 1. We execute the policy using model-predictive control: the policy executes only the first skill ot, receives the updated Zt+1 from the environment, and recalculates the optimal action sequence iteratively. Figure 4 shows an overview of the algorithm. We provide further implementation details in Appendix A.2
  Note that our goal is to study the behavior of VFS as a state representation in existing RL pipelines, rather than developing a better model-based algorithm. Using sophisticated methods that adjust the sampling distribution, as in the cross-entropy method (Finn & Levine, 2017; Hafner et al., 2018; Chua et al., 2018) or path integral optimal control (Williams et al., 2015; Lowrey et al., 2019), as well as more sophisticated models and planning or control methods would likely improve overall performance further, but is outside the scope of this work.
  𝑓 (𝑍t, 𝜋 ~ 𝛱) = 𝑍t+1
  𝑓 ( , 𝜋Close Drawer) =
  𝑓 ( , 𝜋Pick Blue) =
  …
  𝑍t (ot ) = {Vπi(o )}
  𝑓 (𝑍t+1, 𝜋 ~ 𝛱) = 𝑍t+2
  𝑓 ( , 𝜋Pick Blue) =
  𝑓 ( , 𝜋Place Blue in Drawer) =
  𝑍0t+H…
  …
  …
  𝑍1t+H
  𝑍Nt+Hot
  “Place both in drawer” 𝑍g
  argmini ∈ N {||𝑍 i t+T , 𝑍g ||}
  𝑍t (st ) = {Vo(st)}o ∈ O
  𝜋(𝑍t) ~
  [oPick Blue
  oPick Red … oOpen Drawer oClose Drawer]
  oPick Blue(at | st)
  𝑍t (ot ) = {Vπi(ot)}Current: ot 𝜋HL(𝑍t, 𝑍goal) ~ [𝜋Pick Blue 𝜋Pick Red … 𝜋Open Drawer 𝜋Close Drawer] 𝜋Pick Blue(at | ot)
  Goal: 𝑍goal “Place both in drawer”
  Current: st
sections.heading: 6.2 APPLICATION: ROBOTIC MANIPULATION
sections.text: We evaluate the performance of VFS-MB in a complex image-based task using a simulated manipulation environment with an 8-DoF robotic arm, similar to the setup used by Jang et al. (2021). The robot only has access to high-dimensional egocentric visual observations. We consider the task of semantic rearrangement, which requires rearranging objects into semantically meaningful positions – e.g., grouped into (multiple) clusters, or moved to a corner (Figure 5). Efficiently solving this task requires the robot to plan over a series of manipulation maneuvers, interacting with up to 10 distinct objects. This requires long-horizon planning, and presents a major challenge for RL methods. Each of the methods we compare have access to a library of MoveANearB skills, trained as a multi-task policy with MT-Opt (Kalashnikov et al., 2021). There are 10 objects, and 9 possible destinations near which they can be moved, resulting in a total of 90 skills, which then comprise the action space for planning. The skills control the robot at a frequency of 0.5 Hz, taking an average of 14 time steps to execute with a success rate of 94%. We provide further details about these skills in Appendix A.2.
  We consider two versions of the task, which arrange either 5 or 10 objects to semantic positions. The O5 environment uses a random subset of the 10 objects in every experiment; lesser objects allow a smaller planning horizon, making planning problem simpler than in the O10 environment, which has all 10 objects. We randomize the object positions on the table and command the algorithms to reach the same goal with a planning horizon H = 7 steps for the smaller environment and H = 15 steps for the larger environment, reporting the task success rate averaged over 20 experimental runs.
  Representation O5 O10 Raw Image 0.25 0.1 VAE 0.6 0.3 CPC 0.65 0.4 VFS (Ours) 1 0.8 Oracle State 1 0.85
  Table 3: Success rates for robotic manipulation. VFS outperforms all baselines and closely matches oracular performance.
  Baselines: To evaluate the efficacy of the proposed representation for high-level model-based planning, we compare it against four alternative representations used in conjunction with the algorithm described in Section 6.1. All methods have access to the skills and use them as the low-level action space. Raw Image learns a policy on raw visual observations from the robot’s onboard camera. VAE uses a variational autoencoder to project the observations to a 100−dimensional learned embedding space, similar to Corneil et al. (2018). We train this VAE offline from trajectories collected by rolling out the models. CPC uses contrastive predictive coding (van den Oord et al., 2018) to learn a similar representation from offline trajectories. We also compare against an Oracle State baseline that has access to privileged information – the simulator state – and learns the model on this . This gives us an upper bound for the performance of the baselines.
  Evaluation: Table 3 shows the success rates on the two versions of the task for each prior method. The model trained using image observations fares poorly and fails at all but the simplest starting configurations. It succeeds in 2/20 experiments in O10, largely due to poor model predictions. The VAE and CPC representations, which learn a more compact embedding space to plan over, succeed in up to 65% of the tasks in the simpler O5 environment. However, their performance falls sharply in the O10 environment, where the longer horizons read to greater error accumulation. VFS constructs an effective representation that captures the state of the scene as well as the affordances of the skills, and succeeds in all tasks in the O5 environment, matching the oracle performance. It also succeeds in 80% of the tasks in O10, closely matching the oracle’s performance of 85%.
  To understand the factors captured by VFS, we sample encoded observations from a large number of independent trajectories and visualize their 2D t-SNE embeddings (van der Maaten & Hinton, 2008). Figure 6 shows that VFS can successfully capture information about objects in the scene and affordances (e.g. which object is in the robot’s arm and can be manipulated), while ignoring distractors like the poses of the objects on the table and the arm.
sections.heading: 7 DISCUSSION
sections.text: We proposed Value Function Spaces as state abstractions: a novel skill-centric representation that captures the affordances of the low-level skills. States are encoded into representations that are invariant to exogenous factors that do not affect values of these skills. We show that this representation is compatible with both model-free and model-based policies for hierarchical control, and demonstrate significantly improved performance both in terms of successfully performing longhorizon tasks and in terms of zero-shot generalization to novel environments, which leverages the invariances that are baked into our representation.
  The focus of our work is entirely on utilizing a pre-specified set of skills, and we do not address how such skills are learned. A promising way to use VFS in real-world settings is to learn, or discover, these skills in a cheap simulated domain and transfer them to the real-world, where interactions are expensive, enabling real-world learning of long-horizon behavior with limited samples used to learn the high-level policy. Improving the low-level skills jointly with the high-level policy is another direction that could lead to even better performance on particularly complex tasks, and would be an exciting direction for future work. More broadly, since our method connects skills directly to state representations, it could be used to turn unsupervised skill discovery methods directly into unsupervised representation learning methods, which could be an exciting path toward more general approaches that retain the invariances and generalization benefits of our method.
  REPRODUCIBILITY STATEMENT
  The primary contribution of our work, VFS, is a skill-centric representation designed to work with existing model-based and model-free pipelines. The implementation simply involves concatenating the value functions corresponding to the available skills into an embedding vector that can be used for HRL or planning. We provide all necessary information for setting up VFS to work with a reader’s RL algorithm of choice in Sections 5.1, 6.1 and Appendix A. Our model-free experiments are conducted on an open-source gym environment and we provide configuration details for setting up our quantitative experiments in Appendix A.1. We hope that this encourages the community to utilize and build upon the ideas presented in the paper. We plan to release the code along with information about the proprietary environments used in a public release of this article upon publication.
sections.heading: B EXAMPLE ROLLOUTS
sections.text: In this section, we analyze some example rollouts in the robotic manipulation environment showing the evolution of the proposed VFS representation along the rollout. We consider a simplified version of the experiments presented in the manuscript where the robot has access to PickObject/PlaceObject skills corresponding to all objects in the scene. For each of the following figures, the x-axis is timestep, the y-axis is the value function, and the red skill is the selected skill being executed.
  1. In Figure 7, we notice that the only object on the table is a coffee cup and the corresponding value is high (capturing precondition). On picking up the cup, the value goes to 1, capturing the success outcome.
  2. In Figure 8, there are three objects on the table (coffee cup, bottle and bowl) and the corresponding values are high (capturing preconditions). When the cup is picked up, the value corresponding to it shoots to 1 (capturing the success).
  3. Figure 9 shows a slightly more complicated rollout with multiple objects but if we focus on the bowl and apple that are present in the scene, we notice that the apple is inside the bowl. As the robot proceeds to pick up the bowl, the values corresponding to pick skills for both the bowl and action shoot up until the last time step – the apple falls off the bowl and its value drops sharply. This is a complex affordance captured by the VFS state that would be otherwise very hard to represent using a learned state estimator.
  While this was only a small number of examples that break down the rollouts with VFS, we believe that such behavior is expected of VFS as it, intuitively, represents the “functional information” of the observations and hence, qualitatively, captures the preconditions and outcomes.