summaries: The paper addresses how to learn policies for tasks in which constraints are specified in natural language. Towards this, the paper proposes a model that encodes the different types of natural language constraints into intermediate representations that model both spatial and temporal information between states. Then, they use this as input along with the observation to produce an action at each time step for a safe trajectory.  The paper proposed an algorithm to learn a policy when provided with natural language constraints. The paper defined a navigation task called Hazard World, in which an agent navigation on the map to collect items. The authors defined three types of constraints to restrict agents to visit certain states: 1. budgetary constraints, 2. relational constraints and 3. sequential constraints. The three constraints are described in natural language. The authors proposed a two-step solution. In step one, the algorithm learns a mapping between a natural language constraint to an intermediate representation. In step two the algorithm takes the intermediate representation to learn a policy that satisfy the constraints. This paper presents a new test environment, Hazard World, for learning the safe reinforcement learning agents with given natural language constraints. In this problem, the goal of the agent is to find an optimal policy that maximizes the cumulative rewards while satisfying the constraints given in natural language. The authors introduce the model that contains the following two separate components; constraint interpreter for encoding the language constraints and policy network for learning the RL agent. Finally, they report the results of their proposed algorithm and compare it with the baselines.
abstractText: While safe reinforcement learning (RL) holds great promise for many practical applications like robotics or autonomous cars, current approaches require specifying constraints in mathematical form. Such specifications demand domain expertise, limiting the adoption of safe RL. In this paper, we propose learning to interpret natural language constraints for safe RL. To this end, we first introduce HAZARDWORLD, a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We then develop an agent with a modular architecture that can interpret and adhere to such textual constraints while learning new tasks. Our model consists of (1) a constraint interpreter that encodes textual constraints into spatial and temporal representations of forbidden states, and (2) a policy network that uses these representations to produce a policy achieving minimal constraint violations during training. Across different domains in HAZARDWORLD, we show that our method achieves higher rewards (up to 11x) and fewer constraint violations (by 1.8x) compared to existing approaches. However, in terms of absolute performance, HAZARDWORLD still poses significant challenges for agents to learn efficiently, motivating the need for future work.1
authors.name: Tsung-Yen Yang
authors.name: Michael Hu
authors.name: Yinlam Chow
authors.name: Karthik Narasimhan
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A. Rusu Joel Veness Marc G. Bellemare Alex Graves Martin Riedmiller Andreas K. Fidjeland Georg Ostrovski Stig Petersen Charles Beattie Amir Sadik Ioannis Antonoglou Helen King Dharshan Kumaran Daan Wierstra Shane Legg Demis Hassabis
references.title: Human-level control through deep reinforcement learning
references.venue: Nature, 518(7540):529–533,
references.authors: David Silver Richard S. Sutton Martin Müller
references.title: Reinforcement learning of local shape in the game of go
references.venue: In Proceedings of International Joint Conferences on Artificial Intelligence,
references.authors: S.R.K. Branavan David Silver Regina Barzilay
references.title: Learning to win by reading manuals in a Monte-Carlo framework
references.venue: In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,
references.authors: Peter Anderson Qi Wu Damien Teney Jake Bruce Mark Johnson Niko Sünderhauf Ian D. Reid Stephen Gould Anton van den Hengel
references.title: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
references.venue: IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Dipendra Misra Andrew Bennett Valts Blukis Eyvind Niklasson Max Shatkhin Yoav Artzi
references.title: Mapping instructions to actions in 3D environments with visual goal prediction
references.venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,
references.authors: Gabriel Dulac-Arnold Daniel Mankowitz Todd Hester
references.title: Challenges of real-world reinforcement learning
references.venue: arXiv preprint arXiv:1904.12901,
references.authors: Joshua Achiam David Held Aviv Tamar Pieter Abbeel
references.title: Constrained policy optimization
references.venue: In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning,
references.authors: Yinlam Chow Ofir Nachum Aleksandra Faust Mohammad Ghavamzadeh Edgar Duenez- Guzman
references.title: Lyapunov-based safe policy optimization for continuous control
references.authors: Tsung-Yen Yang Justinian Rosca Karthik Narasimhan Peter J. Ramadge
references.title: Projection-based constrained policy optimization
references.venue: In 8th International Conference on Learning Representations,
references.authors: Tsung-Yen Yang Justinian Rosca Karthik Narasimhan Peter J. Ramadge
references.title: Accelerating safe reinforcement learning with constraint-mismatched policies
references.venue: arXiv preprint arXiv:2006.11645,
references.authors: Joshua Achiam Dario Amodei
references.title: Benchmarking safe exploration in deep reinforcement learning
references.authors: Yinlam Chow Ofir Nachum Edgar Duenez-Guzman Mohammad Ghavamzadeh
references.title: A lyapunov-based approach to safe reinforcement learning
references.venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,
references.authors: Felix Berkenkamp Matteo Turchetta Angela Schoellig Andreas Krause
references.title: Safe model-based reinforcement learning with stability guarantees
references.venue: Advances in Neural Information Processing Systems,
references.authors: Mahmoud El Chamie Yue Yu Behçet Açıkmeşe
references.title: Convex synthesis of randomized policies for controlled markov chains with density safety upper bound constraints
references.venue: In 2016 American Control Conference (ACC),
references.authors: Matteo Turchetta Andrey Kolobov Shital Shah Andreas Krause Alekh Agarwal
references.title: Safe reinforcement learning via curriculum induction
references.venue: arXiv preprint arXiv:2006.12136,
references.authors: Brijen Thananjeyan Ashwin Balakrishna Suraj Nair Michael Luo Krishnan Srinivasan Minho Hwang Joseph E. Gonzalez Julian Ibarz Chelsea Finn Ken Goldberg
references.title: Recovery rl: Safe reinforcement learning with learned recovery zones
references.venue: IEEE Robotics and Automation Letters,
references.authors: Maxime Chevalier-Boisvert Dzmitry Bahdanau Salem Lahlou Lucas Willems Chitwan Saharia Thien Huu Nguyen Yoshua Bengio
references.title: Babyai: A platform to study the sample efficiency of grounded language learning
references.venue: In 7th International Conference on Learning Representations,
references.authors: Alex Ray Joshua Achiam Dario Amodei
references.title: Benchmarking Safe Exploration in Deep Reinforcement Learning
references.venue: arXiv preprint arXiv:1910.01708,
references.authors: Matt MacMahon Brian Stankiewicz Benjamin Kuipers
references.title: Walk the talk: Connecting language, knowledge, and action in route instructions
references.venue: In Proceedings of the Conference on Artificial Intelligence,
references.authors: David L. Chen Raymond J. Mooney
references.title: Learning to interpret natural language navigation instructions from observations
references.venue: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence,
references.authors: Yoav Artzi Luke Zettlemoyer
references.title: Weakly supervised learning of semantic parsers for mapping instructions to actions
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Dipendra Misra John Langford Yoav Artzi
references.title: Mapping instructions and visual observations to actions with reinforcement learning
references.venue: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
references.authors: Karl Moritz Hermann Mateusz Malinowski Piotr Mirowski Andras Banki-Horvath Keith Anderson Raia Hadsell
references.title: Learning to follow directions in street view
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Weituo Hao Chunyuan Li Xiujun Li Lawrence Carin Jianfeng Gao
references.title: Towards learning a generic agent for vision-and-language navigation via pre-training
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Javier Garcia Fernando Fernandez
references.title: A comprehensive survey on safe reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Thiago D Simão Nils Jansen Matthijs TJ Spaan
references.title: Alwayssafe: Reinforcement learning without safety constraint violations during training
references.venue: In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems,
references.authors: Chen Tessler Daniel J. Mankowitz Shie Mannor
references.title: Reward constrained policy optimization
references.venue: In 7th International Conference on Learning Representations,
references.authors: Adam Stooke Joshua Achiam Pieter Abbeel
references.title: Responsive safety in reinforcement learning by PID lagrangian methods
references.venue: In Proceedings of the 37th International Conference on Machine Learning,
references.authors: Stéphane Ross Geoffrey Gordon Drew Bagnell
references.title: A reduction of imitation learning and structured prediction to no-regret online learning
references.venue: In Proceedings of International Conference on Artificial Intelligence and Statistics,
references.authors: Aravind Rajeswaran Vikash Kumar Abhishek Gupta Giulia Vezzani John Schulman Emanuel Todorov Sergey Levine
references.title: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations
references.venue: In Proceedings of Robotics: Science and Systems,
references.authors: Yang Gao Ji Lin Fisher Yu Sergey Levine Trevor Darrell
references.title: Reinforcement learning from imperfect demonstrations
references.venue: In Proceedings of the 35th International Conference on Machine Learning,
references.authors: Adam Vogel Daniel Jurafsky
references.title: Learning to follow navigational directions
references.venue: In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics,
references.authors: Stefanie Tellex Thomas Kollar Steven Dickerson Matthew Walter Ashis Banerjee Seth Teller Nicholas Roy
references.title: Understanding natural language commands for robotic navigation and mobile manipulation
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Joohyun Kim Raymond Mooney
references.title: Adapting discriminative reranking to grounded language learning. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 218–227
references.venue: Sofia, Bulgaria,
references.authors: Jacob Andreas Dan Klein
references.title: Alignment-based compositional semantics for instruction following
references.venue: In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing,
references.authors: Jesse Thomason Michael Murray Maya Cakmak Luke Zettlemoyer
references.title: Vision-and-dialog navigation
references.venue: In Proceedings of Conference on Robot Learning,
references.authors: Jelena Luketina Nantas Nardelli Gregory Farquhar Jakob N. Foerster Jacob Andreas Edward Grefenstette Shimon Whiteson Tim Rocktäschel
references.title: A survey of reinforcement learning informed by natural language
references.venue: In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence,
references.authors: Stefanie Tellex Nakul Gopalan Hadas Kress-Gazit Cynthia Matuszek
references.title: Robots that use language
references.venue: Annual Review of Control, Robotics, and Autonomous Systems,
references.authors: Christopher Wang Candace Ross Yen-Ling Kuo Boris Katz Andrei Barbu
references.title: Learning a natural-language to ltl executable semantic parser for grounded robotics
references.venue: arXiv preprint arXiv:2008.03277,
references.authors: Yonatan Bisk Kevin J. Shih Yejin Choi Daniel Marcu
references.title: Learning interpretable spatial operations in a rich 3d blocks world
references.venue: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,
references.authors: Howard Chen Alane Suhr Dipendra Misra Noah Snavely Yoav Artzi
references.title: TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments
references.venue: In IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Harm de Vries Kurt Shuster Dhruv Batra Devi Parikh Jason Weston Douwe Kiela
references.title: Talk the walk: Navigating new york city through grounded dialogue
references.venue: arXiv preprint arXiv:1807.03367,
references.authors: Michael Janner Karthik Narasimhan Regina Barzilay
references.title: Representation learning for grounded spatial reasoning
references.venue: Transactions of the Association for Computational Linguistics,
references.authors: Valts Blukis Dipendra Misra Ross A. Knepper Yoav Artzi
references.title: Mapping navigation instructions to continuous control actions with position-visitation prediction
references.venue: In Proceedings of the Conference on Robot Learning,
references.authors: Daniel Fried Ronghang Hu Volkan Cirik Anna Rohrbach Jacob Andreas Louis-Philippe Morency Taylor Berg-Kirkpatrick Kate Saenko Dan Klein Trevor Darrell
references.title: Speakerfollower models for vision-and-language navigation
references.venue: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems
references.authors: Ninghao Liu Mengnan Du Xia Hu
references.title: Representation interpretation with spatial encoding and multimodal analytics
references.venue: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining,
references.authors: Vihan Jain Gabriel Magalhaes Alexander Ku Ashish Vaswani Eugene Ie Jason Baldridge
references.title: Stay on the path: Instruction fidelity in vision-and-language navigation
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: David Gaddy Dan Klein
references.title: Pre-learning environment representations for data-efficient neural instruction following
references.venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,
references.authors: Yordan Hristov Daniel Angelov Michael Burke Alex Lascarides Subramanian Ramamoorthy
references.title: Disentangled relational representations for explaining and learning from demonstration
references.venue: In Proceedings of the Conference on Robot Learning,
references.authors: Justin Fu Anoop Korattikara Sergey Levine Sergio Guadarrama
references.title: From language to goals: Inverse reinforcement learning for vision-based instruction following
references.venue: In 7th International Conference on Learning Representations,
references.authors: Sagar Gubbi Venkatesh Anirban Biswas Raviteja Upadrashta Vikram Srinivasan Partha Talukdar Bharadwaj Amrutur
references.title: Spatial reasoning from natural language instructions for robot manipulation
references.venue: arXiv preprint arXiv:2012.13693,
references.authors: Philip S. Thomas Bruno Castro da Silva Andrew G. Barto Stephen Giguere Yuriy Brun Emma Brunskill
references.title: Preventing undesirable behavior of intelligent
references.venue: machines. Science,
references.authors: Thomas M Howard Stefanie Tellex Nicholas Roy
references.title: A natural language planner interface for mobile manipulators
references.venue: In 2014 IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Bharat Prakash Nicholas Waytowich Ashwinkumar Ganesan Tim Oates Tinoosh Mohsenin
references.title: Guiding safe reinforcement learning policies using structured language constraints
references.venue: UMBC Student Collection,
references.authors: Eitan Altman
references.title: Constrained Markov decision processes, volume 7
references.venue: CRC Press,
references.authors: Maxime Chevalier-Boisvert Lucas Willems Suman Pal
references.title: Minimalistic gridworld environment for openai gym
references.venue: https://github.com/maximecb/gym-minigrid,
references.authors: Michael Buhrmester Tracy Kwang Samuel D. Gosling
references.title: Amazon’s mechanical turk: A new source of inexpensive, yet high-quality data? 2016
references.authors: Aaron Walsman Yonatan Bisk Saadia Gabriel Dipendra Misra Yoav Artzi Yejin Choi Dieter Fox
references.title: Early fusion for goal directed robotic vision
references.venue: In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems,
references.authors: John Schulman Sergey Levine Pieter Abbeel Michael I. Jordan Philipp Moritz
references.title: Trust region policy optimization
references.venue: Proceedings of the 32nd International Conference on Machine Learning, ICML 2015,
references.authors: Eric Jang Shixiang Gu Ben Poole
references.title: Categorical reparameterization with gumbel-softmax. volume
references.authors: John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel
references.title: Highdimensional continuous control using generalized advantage estimation
references.venue: In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations,
references.authors: Yan Duan Xi Chen Rein Houthooft John Schulman Pieter Abbeel
references.title: Benchmarking deep reinforcement learning for continuous control
references.venue: Proceedings of the 33nd International Conference on Machine Learning,
sections.heading: 1 Introduction
sections.text: Although reinforcement learning (RL) has shown promise in several simulated domains such as games [1, 2, 3] and autonomous navigation [4, 5], deploying RL in real-world scenarios remains challenging [6]. In particular, real-world RL requires ensuring the safety of the agent and its surroundings, which means accounting for constraints during training that are orthogonal to maximizing rewards. For example, a cleaning robot must be careful to not knock the television over, even if the television lies on the optimal path to cleaning the house.
  Safe RL tackles these challenges with algorithms that maximize rewards while simultaneously minimizing constraint violations during exploration [7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. However, these algorithms have two key limitations that prevent their widespread use. First, they require us
  ∗ Equal contribution. 1Code and data are available at https://github.com/princeton-nlp/SRL-NLC
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  to provide constraints in mathematical or logical forms, which calls for specific domain expertise. Second, a policy trained with a specific set of constraints cannot be transferred easily to learn new tasks with the same set of constraints, since current approaches do not maintain an explicit notion of constraints separate from reward-maximizing policies. This means one would have to retrain the policy (with constraints) from scratch.
  We consider the use of natural language to specify constraints (which are orthogonal to rewards) on learning. Human languages provide an intuitive and easily-accessible medium for describing constraints–not just for machine learning experts or system developers, but also for potential end users interacting with agents such as household robots. Consider the environment in Fig. 1 for example. Instead of expressing a constraint as ∑T t=0 1st∈lava · 16∃st′∈water, t′∈[0,1,...,t−1] = 0, one could simply say “Do not visit the lava before visiting the water”. The challenge of course, lies in training the RL agent to accurately interpret and adhere to the textual constraints as it learns a policy for the task.
  To study this problem, we first create HAZARDWORLD, a collection of grid-world and robotics environments for safe RL with textual constraints (Fig. 1). HAZARDWORLD consists of separate ‘safety training’ and ‘safety evaluation’ sets, with disjoint sets of reward functions and textual constraints between training and evaluation. To do well on HAZARDWORLD, an agent has to learn to interpret textual con-
  straints during safety training and safely adhere to any provided constraints while picking up new tasks during the safety evaluation phase. Built on existing RL software frameworks [17, 18], HAZARDWORLD consists of navigation and object collection tasks with diverse, crowdsourced, free-form text specifying three kinds of constraints: (1) budgetary constraints that limit the frequency of being in unsafe states, (2) relational constraints that specify unsafe states in relation to surrounding entities, and (3) sequential constraints that activate certain states to be unsafe based on past events (e.g., “Make sure you don’t walk on water after walking on grass”). Our setup differs from instruction following [19, 20, 21, 22, 23, 24] in two ways. First, instructions specify what to do, while textual constraints only inform the agent on what not to do, independent of maximizing rewards. Second, learning textual constraints is a means for ensuring safe exploration while adapting to a new reward function.
  In order to demonstrate learning under this setting, we develop Policy Optimization with Language COnstraints (POLCO), where we disentangle the representation learning for textual constraints from policy learning. Our model first uses a constraint interpreter to encode language constraints into representations of forbidden states. Next, a policy network operates on these representations and state observations to produce actions. Factorizing the model in this manner allows the agent to retain its constraint comprehension capabilities while modifying its policy network to learn new tasks.
  Experiments demonstrate that our approach achieves higher rewards (up to 11x) while maintaining lower constraint violations (up to 1.8x) compared to several baselines on two different domains within HAZARDWORLD. Nevertheless, HAZARDWORLD remains far from being solved, especially in tasks with high-dimensional observations, complex textual constraints and those requiring high-level planning or memory-based systems.
sections.heading: 2 Related Work
sections.text: Safe RL. Safe RL deals with learning constraint-satisfying policies [25], or learning to maximize rewards while minimizing constraint violations.2 This is a constrained optimization problem, and thus different from simply assigning negative reward values to unsafe states. Furthermore, large negative reward values for constraint violations can destabilize training and lead to degenerate behavior, such as the agent refusing to move. In prior work, the agent typically learns policies either by (1) exploring the environment to identify forbidden behaviors [7, 27, 8, 9, 28], or (2) using expert demonstration data to recognize safe trajectories [29, 30, 31, 10]. All these works require a human to specify the cost constraints in mathematical or logical form, and the learned constraints cannot be easily reused for new learning tasks. In this work, we design a modular architecture to learn to interpret textual constraints, and demonstrate transfer to new learning tasks.
  Instruction following. Our work closely relates to the paradigm of instruction following in RL, which has previously been explored in several environments [19, 32, 20, 33, 21, 34, 35, 36, 37, 38, 39]. Prior work has also focused on creating realistic vision-language navigation datasets [40, 41, 4, 42] and proposed computational models to learn multi-modal representations that fuse images with goal instructions [43, 44, 45, 46, 47, 48, 49, 50, 51]. Our work differs from the traditional instruction following setup in two ways: (1) Instruction following seeks to (roughly) ‘translate’ an instruction directly into an action policy. This does not apply to our setting since the textual constraints only tell an agent what not to do. To actually obtain rewards, the agent has to explore and figure out optimal policies on its own. (2) Since constraints are decoupled from rewards and policies, agents trained to understand certain constraints can transfer their understanding to respect these constraints in new tasks, even when the new optimal policy is drastically different. Therefore, we view this work as orthogonal to traditional instruction following–one could of course combine both instructions and textual constraints to simultaneously advise an agent on what to do and what not to do.
  Connection to Seldonian algorithms. POLCO can also be interpreted as a Seldonian algorithm [52]. Seldonian algorithms ensure ML safety through three steps: (1) defining a goal, (2) defining an interface for users to provide constraints, and (3) creating an algorithm that satisfies the goal and constraints. Here, we use natural language as the interface for end users and map natural language into optimization constants and vector representations. Thus, POLCO is also a potential step towards widely developing and deploying Seldonian algorithms.
  Constraints in natural language. Our notion of ‘constraints’ in this paper differs from prior work that uses instructions to induce planning constraints [33, 53, 39]–these works again provide instructions for the agent on how to perform the task. Perhaps closest to this paper is the work of Misra et al. [5], which proposes datasets to study spatial and temporal reasoning, containing a subset focusing on trajectory constraints (e.g., “go past the house by the right side of the apple”). However, they do not disentangle the rewards from the constraints, which may be orthogonal to each other. Prakash et al. [54] train a constraint checker to identify whether a constraint (specified in text) has been violated in a trajectory. While their motivation is similar, they ultimately convert constraints to negative rewards, whereas we use a modular approach that allows disentangling reward maximization from minimizing constraint violations and is compatible with modern algorithms for safe RL.
sections.heading: 3 Preliminaries
sections.text: Problem formulation. Our learning problem can be viewed as a partially observable constrained Markov decision process [55], which is defined by the tuple < S,O,A, T, Z,X , R, C >. Here S is the set of states, O is the set of observations, A is the set of actions, T is the conditional probability T (s′|s, a) of the next state s′ given the current state s and the action a, and Z is the conditional probability Z(o|s) of the observation o given the state s. In addition, X is the set of textual constraint specifications, R : S ×A → R is the reward function, which encodes the immediate reward provided when the agent takes an action a in state s, and C : S ×A×X → R is the true underlying constraint function described by x ∈ X , which specifies positive penalties for constraint violations due to an action a in a state s. Finally, we assume each x ∈ X corresponds to a specific cost function C.
  2In this paper, we consider minimizing constraint violations in expectation [9] and leave stricter notions such as enforcing zero violations [26] to future work.
  RL with constraints. The goal of the learning agent is to acquire a good control policy that maximizes rewards, while adhering to the specified constraints as much as possible during the learning process. Thus, the agent learns a policy π : O ×X → P(A), which is a mapping from the observation space O and constraint specification X to the distributions over actions A. Let γ ∈ (0, 1) denote a discount factor, µ(S) denote the initial state distribution, and τ denote a trajectory sequence of observations and actions induced by a policy π, i.e., τ = (o0, a0, o1, · · · ). For any given x, we seek a policy π that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost JC below a specified cost constraint threshold hC(x):
  max π JR(π) . = E τ∼π [ ∞∑ t=0 γtR(st, at) ] s.t. JC(π) . = E τ∼π [ ∞∑ t=0 γtC(st, at, x) ] ≤ hC(x),
  where τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : s0 ∼ µ, ot ∼ Z(·|st), at ∼ π(·|ot, x), st+1 ∼ T (·|st, at). We use C(st, at, x) and hC(x) here to emphasize that both functions depend on the particular constraint specification x.
  Task setup. Our goal is to show that constraints specified in natural language allow for generalization to new tasks that require similar constraints during learning. With this in mind, we consider the following safety training and safety evaluation setup:
  (1) Safety training: During training, we generate random environment layouts and starting states s0 while keeping the reward function R fixed. For each episode, we randomly generate a constraint function C and limit hC . We then sample a constraint text x that describes C and hC from the training set of texts. The constraint text x is an input to the agent’s policy. Whenever the agent violates a constraint (at any step), it is provided with a scalar cost penalty learned by the model from C(s, a, x). The agent, therefore, sees a variety of different task layouts and constraints, and learns a policy with respect to the constraints for this task as well as how to interpret textual constraints.
  (2) Safety evaluation: During evaluation, we place the agent in new environments with randomly generated layouts, with a different reward function R′. The set of possible constraints C is the same as seen in training, but the corresponding constraint texts are from an unseen test set. During this phase, the agent is not provided any cost penalties from the task. This setup allows us to measure two things: (1) how well an agent can learn new tasks while following previously learned textual constraints, and (2) the applicability of our method when using textual constraints unseen in training.
sections.heading: 4 HAZARDWORLD
sections.text: To our knowledge, there do not currently exist datasets for evaluating RL agents that obey textual constraints.3 Thus, we design a new benchmark called HAZARDWORLD in which the agent starts
  3Even though there are several instruction following tasks, our task setup is different, as mentioned previously.
  each episode at a random location within a procedurally generated environment and receives a textual constraint x, sampled from a pool of available constraints. The agent’s goal is to collect all the reward-providing entities while adhering to the specified constraint. Other than the constraint specified, the agent has complete freedom and is not told about how to reach reward-providing states.
  HAZARDWORLD contains three types of constraints–(1) budgetary constraints, which impose a limit on the number of times a set of states can be visited, (2) relational constraints, which define a minimal distance that must be maintained between the agent and a set of entities, and (3) sequential constraints, which are constraints that activate unsafe states when a specific condition has been met. In total, we collect 984 textual constraints for HAZARDWORLD-grid (GridWorld environment) and 2,381 textual constraints for HAZARDWORLD-robot (robotic tasks). Table 1 provides examples.
  HAZARDWORLD-grid. We implement HAZARDWORLD-grid (Table 1(a)) atop the 2D GridWorld layout of BabyAI [17, 56]. We randomly place three reward entities on the map: ‘ball,’ ‘box,’ and ‘key,’ with rewards of 1, 2, and 3, respectively. We also randomly place several cost entities on the map: ‘lava,’ ‘water,’ and ‘grass’. We give a cost penalty of 1 when agents step onto any cost entities, which are specified using a textual constraint x. The entire state st is a grid of size 13×13, including the walls, and the agent’s observation ot is a 7×7 grid of its local surroundings. There are 4 actions–for moving up, down, left and right. We use the deterministic transition here.
  Train-test split. We generate two disjoint training and evaluation datasets Dtrain and Deval. Dtrain consists of 10,000 randomly generated maps paired with 80% of the textual constraints (787 constraints overall), i.e., on average each constraint is paired with 12.70 different maps. Deval consists of 5,000 randomly generated maps paired with the remaining 20% of the textual constraints (197 constraints), i.e., on average one constraint is paired with 25.38 maps. In Deval we change the rewards for ball, box, and key to 1, 2, and -3, respectively. Therefore, in Deval, the agent has to avoid collecting the key to maximize reward.
  HAZARDWORLD-robot. We build HAZARDWORLD-robot (Table. 1(b)) atop the SAFETY GYM environment [18] to show the applicability of our model to tasks involving high-dimensional continuous observations. In this environment, there are five constraint entities paired with textual constraints: hazards (dark blue puddles), vases (stationary but movable teal cubes), pillars (immovable cylinders), buttons (touchable orange spheres), and gremlins (moving purple cubes). This task is more challenging than the 2D case since some obstacles are constantly moving. The agent receives a reward of 4 for reaching a goal position and a cost penalty of 1 for bumping into any constraint entities. The observation ot is a vector of size 109, including coordinate location, velocity of the agent, and observations from lidar rays that detect the distance to entities. The agent has two actions–control signals applied to the actuators to make it move forward or rotate. The transitions are all deterministic.
  Train-test split. We follow the same process for obtaining a train-test split as in HAZARDWORLD-grid. Dtrain consists of 10,000 randomly generated maps paired with 80% of textual constrains (1,905 constraints), i.e., on average one constraint is paired with 5.25 maps. Deval consists of 1,000 randomly generated maps paired with the remaining 20% of textual constrains (476 constraints), i.e., on average one constraint is paired with 2.10 maps. In Deval we add four additional goal locations to each map (i.e., the maximum reward is 20). The agent has to learn to navigate to these new locations.
  Data collection. For the textual constraints in both environments, we collected free-form text in English using Amazon Mechanical Turk (AMT) [57]. To generate a constraint for HAZARDWORLD, we provided workers with a description and picture of the environment, the cost entity to be avoided, and one of the following: (a) the cost budget (budgetary), (b) the minimum safe distance (relational), or (c) the other cost entity impacted by past events (sequential). We then cleaned the collected text by writing a keyword matching script followed by manual verification to ensure the constraints are valid.
sections.heading: 5 Learning to Interpret Textual Constraints
sections.text: We seek to train agents that can adhere to textual constraints even when learning policies for new tasks with different reward structures. We now describe our model and training and evaluation procedures.
sections.heading: 5.1 Model
sections.text: We design the RL agent as a deep neural network that consists of two parts (Fig. 2)–(1) a constraint interpreter which processes the text into structured safety criteria (a constraint mask and threshold)
  and (2) a policy network which uses the output of the interpreter along with observations to produce an action. For simplicity, in the following descriptions, we assume state s and observation o to be 2D matrices, although the model can easily be extended to other input representations.
  (1) Constraint interpreter (Fig. 3). We concatenate an observation embedding of size n× n×m from observations o of size n× n with the embedding of the textual constraints x of size l from a long-short-term-memory (LSTM), followed by using a convolutional neural network (CNN) to get an embedding vector. We use this vector to produce a constraint mask M̂C , a binary matrix with the same dimension as o–each cell of the matrix is 0/1 depending on whether the model believes the absence or presence of a constraint-related entity (e.g., ‘lava’) in the corresponding cell of the observation o. In addition, we feed the textual constraints into an LSTM to produce ĥC , a real-valued scalar which predicts the constraint threshold, i.e., the number of times an unsafe state is allowed.
  For the case of sequential constraints with long-term dependency of the past events, M̂C will depend on the past states visited by the agent. For example, in Fig. 3(b), after the agent visits ‘water’, M̂C starts to locate the cost entity (‘grass’). Thus, for sequential constraints, we modify the interpreter by adding an LSTM layer before computing M̂C to take the state history into account. Using M̂C and ĥC allows us to embed textual constraints in the policy network.
  (2) Policy network. The policy network produces an action using the state observation ot and the safety criteria produced by the constraint interpreter. The environment embedding is concatenated with the constraint mask M̂C (predicted by the constraint interpreter) and a cost budget mask, denoted by M̂B . The cost budget mask is derived from ĥC (also predicted by the constraint interpreter) and keeps track of the number of constraint violations that the agent has made in the past over the threshold. M̂B is an n × n matrix where each element takes the value of ∑t′ t=0 Ĉ(st, at;x) − ĥC (i.e., the value of constraint violations past the budget until t′th step) if there is a cost entity in ot(i, j), or zero otherwise. During the safety evaluation phase, we estimate the cumulative cost ∑t′ t=0 Ĉ(st, at;x) using the predicted M̂C and the agent’s current location at time t. After concatenating both the constraint mask M̂C and cost budget mask M̂B to the observation embedding, we then feed the resulting tensor into CNN to obtain a vector (grey in Fig. 2). This vector is concatenated with a vectorized int(ĥC) (i.e., ĥC rounded down) and fed into an MLP to produce an action.
  POLCO in HAZARDWORLD-robot. To apply POLCO in this environment, the constraint interpreter predicts the cost entity given the textual constraints. We then map the cost entity to the pre-defined embedding vector (i.e., one-hot encoding). We then concatenate the embedding vector, the embeddings of the predicted ĥC , and the value of cost budget (rounded down) to the observation vector. Finally, the policy network takes in this concatenated observation and produces a safe action.
  Advantages of the design. The design of POLCO tightly incorporates textual constraints into the policy network. Our model factorization–into (1) a constraint interpreter and (2) a policy network–
  allows us to design specific constraint interpreters for different types of constraints.4 Furthermore, our approach scales gracefully to handling multiple constraints. While existing safe RL algorithms require retraining the policy for each unique combination of constraints, we can simply add together the M̂C of each constraint to handle multiple constraints imposed simultaneously.
sections.heading: 5.2 Safety training
sections.text: We first train the constraint interpreter using a random policy to collect trajectories, and then we use the trained interpreter to predict constraints while training the policy network.
  Stage 1: Interpreter learning. We use a random policy to explore the environment, and obtain trajectories consisting of observations ot, along with the corresponding textual constraint x. Using the constraint violations encountered in the trajectory and the cost specification C, we obtain a target MC for training the constraint interpreter. In addition, we also obtain the ground-truth value of hC for learning the constraint threshold module.
  We train the constraint mask module of the constraint interpreter by minimizing the following binary cross-entropy loss over these trajectories: L(Θ1) = −E(ot,x)∼Dtrain [ 1 |MC | ∑n i,j=1 y log ŷ + (1 −
  y) log(1 − ŷ) ] , where y is the target MC(i, j; ot, x), which denotes the target (binary) mask label
  in ith row and jth column of the n × n observation ot, ŷ is the predicted M̂C(i, j; ot, x), i.e., the probability prediction of constraint mask, and Θ1 are the parameters of the constraint mask module.
  For the constraint threshold module, we minimize the following loss: L(Θ2) = E(ot,x)∼Dtrain [ (hC(x)− ĥC(x))2 ] , where Θ2 are the parameters of the constraint threshold module.
  This approach ensures cost satisfaction during both policy learning and safety evaluation, an important feature of safe RL. If we train both the policy and the interpreter simultaneously, then we risk optimizing according to inaccurate M̂C and ĥC values, as observed in our experiments.
  Stage 2: Policy learning. We use a safe RL algorithm called projection-based constrained policy optimization (PCPO) [9] to train the policy network. During training, the agent interacts with the environment to obtain rewards and penalty costs (M̂C) are provided from the trained constraint interpreter for computing JR(π) and JC(π) (ground-truth C is not used). PCPO is an iterative method that performs two key steps in each iteration5–optimize the policy according to reward and project the policy to a set of policies that satisfy the constraint. During safety evaluation, we evaluate our model in the new task with the new reward function and the textual constraints from Deval.
  4M̂B equates to a scaled up version of M̂C since we assume only one constraint specification per episode, but this is not necessary in general since we may have multiple constraints over different cost entities. In that case, M̂B may have different cost budgets for different cells (entities).
  5One can use other safe RL algorithms such as Constrained Policy Optimization (CPO) [7]
sections.heading: 5.3 Safety evaluation
sections.text: (1) Transfer to new tasks: We take the policy trained in Dtrain and fine-tune it on tasks having new reward functions with textual constraints from Deval. We do not retrain the constraint interpreter on Deval. The policy is fine-tuned to complete the new tasks without the penalty signals from the cost function C. In HAZARDWORLD-robot, we optimize the policy using CPO [7].
  (2) Handling multiple textual constraints: We also test the ability of our model to handle multiple constraints imposed simultaneously (from Deval), by adding the cost constraint masks M̂C of each constraint together when given multiple constraints. During safety training, the policy is still trained with a single constraint. No fine-tuning is performed and the reward function is maintained the same across training and evaluation in this case.
sections.heading: 6 Experiments
sections.text: Our experiments aim to study the following questions: (1) Does the policy network, using representations from the constraint interpreter, achieve fewer constraint violations in new tasks with different reward functions? (2) How does each component in POLCO affect its performance?
sections.heading: 6.1 Setup
sections.text: Baselines. We consider the following baselines: (1) Constraint-Fusion (CF) with PCPO: This model [58] takes a concatenation of the observations and text representations as inputs (without MC , MB and hC ) and produces an action, trained with an end-to-end approach using PCPO. This model jointly processes the observations and the constraints. (2) CF with TRPO: We train CF using trust region policy optimization (TRPO) [59], which ignores all constraints and only optimizes the reward. This is to demonstrate that the agent will have substantial
  constraint violations when ignoring constraints. (3) Random Walk (RW): We also include a random walk (RW) baseline, where the agent samples actions uniformly at random.
  Evaluation metrics. To evaluate models, we use (1) the average value of the reward JR(π), and (2) the average constraint violations ∆C := max(0, JC(π)−hC). Good models should have a small ∆C (i.e., close to zero) while maximizing JR(π). More details on the implementation, hyper-parameters, and computational resources are included in the Appendix A,B, and C.
sections.heading: 6.2 Results
sections.text: HAZARDWORLD-grid. Fig. 4(1) shows results for all models in the first evaluation setting of transfer to new tasks. POLCO has lower constraint violations in excess of hC while still achieving better reward performance in all cases. In comparison, the high cost values (∆C ) obtained by RW and CF with TRPO indicate the challenges of task. This supports our idea of using the learned constraint interpreter to learn a new task with similar textual constraints while ensuring constraint satisfaction. CF with PCPO has higher constraint violations, and in the most cases, does not optimize the reward, which suggests that it cannot transfer the constraint understanding learned in Dtrain to Deval.
  Fig. 4(2) shows our evaluation with multiple textual constraints. We see that POLCO achieves superior reward and cost performance compared to the baselines, while CF with PCPO has worse reward and cost performance. This shows that our approach is flexible enough to impose multiple constraints than that of existing safe RL methods which requires retraining the policy for each unique combination of constraints.
  HAZARDWORLD-robot. Fig. 5 shows transfer to new tasks in HAZARDWORLD-robot. The JR(π) and ∆C of RW is relatively small since the agent does not move much because of random force applied to each actuator. For the budgetary constraints, although CF with TRPO achieves the best reward when hC = 0, it has very large constraint violations. POLCO performs better than the baselines–it induces policies with higher reward under fewer constraint violations in most cases. In contrast, CF with CPO has lower reward performance.
  Having demonstrated the overall effectiveness of POLCO, our remaining experiments analyze (1) the learned models’ performance evaluated on the same reward function as in Dtrain, and (2) the importance of each component–MB ,MC and hC embedding in POLCO. For compactness, we restrict our consideration in HAZARDWORLD-grid.
  Evaluation with reward function from Dtrain. To provide another point of comparison in addition to our main results, we evaluate all models using the same reward function as in Dtrain, but with unseen textual constraints from Deval. ( Fig. 6) We observe POLCO achieves the lowest violations across different choices of hC compared to the baselines. This implies that merely combining the observations and the text is not sufficient to learn an effective representation for parsing the constraints. In addition, POLCO achieves the best reward performance under cost satisfaction for the more complex relational and sequential constraints. For the relational case, although the CF agent trained with PCPO satisfies the constraints, it has a relatively low reward.
  Ablation studies. We also examine the importance of each part in POLCO (Fig. 7). To eliminate prediction errors from the constraint interpreter, we use the true MC and hC here. Our full model achieves the best performance in all cases, averaging 5.12% more reward and 2.22% fewer constraint violations. Without MC , the agent cannot recognize cost entities effectively, which causes the agent to incur 66.67% higher ∆C compared with the full model (which has a ∆C close to zero). This shows that hC embedding and the MB mask are useful in enabling constraint satisfaction given textual constraints.
sections.heading: 7 Conclusion
sections.text: Our work provides a view towards machines that can interoperate with humans. As autonomous agents proliferate into our world, they should be able to understand safety constraints set by human agents around them. Accordingly, we proposed the problem of safe RL with natural language constraints, created a new benchmark called HAZARDWORLD to test agents and developed a new algorithm for the task (POLCO) that learns to interpret constraints. Our paper defines and trains machine agents that understand what not to do in natural language, much like instruction following tasks enable agents in understanding what to do.
  The thesis of our POLCO approach is that modularity enables reuse. By bootstrapping a modular constraint interpreter through exploration, our model scales easily to multiple constraints and to shifts in the environment’s reward structure, all while exploring new environments safely. We applied POLCO within HAZARDWORLD to train an agent that navigates safely by obeying natural language constraints. This agent is a step towards creating applications like cleaning robots that can obey free form constraints, such as “don’t get too close to the TV” – a relational constraint in our formulation.
  No model is without limitations. The absolute scores of POLCO on HAZARDWORLD still leave a lot of room for improvement using better models or training techniques. The current version of HAZARDWORLD is also not all-encompassing – we envision it as a benchmark that evolves over time, with the addition of new types of constraints and new environments. Future work can investigate training without explicit labels for the constraint interpreter, potentially using techniques like Gumbel softmax [60], or extending POLCO to tasks with more realistic visuals.
sections.heading: Acknowledgments
sections.text: The authors would like to thank members of the Princeton NLP Group, the anonymous reviewers, and the area chair for their comments and feedback. Tsung-Yen Yang thanks Siemens Corporation, Corporate Technology for their support.
  Funding Transparency Statement
  Funding in direct support of this work: scholarship by Siemens Corporation, Corporate Technology, and the Princeton CSML Summer Research Award.
title: Safe Reinforcement Learning with Natural Language Constraints