summaries: The authors propose to apply HER to image-based domain, assigning rewards based only on exact equality (and not using an epsilon-ball). The authors also propose to (1) filter transitions that are likely to cause false negative rewards and (2) balance the goal relabeling so that the number of positive and negative rewards are equal. The authors demonstrate that these two additions result in faster and better learning on a simulated 2d and 3d reaching, as well as a rope task. The authors also show that the method works on training a real-world robot to reach different positions from images.

summaries: The paper tackles the problem of self-supervised reinforcement learning through the lens of goal-conditioned RL, which is in line with recent work (Nair et al, Wade-Farley et al, Florensa et al, Yu et al.). The proposed approach is a simple one - it uses the relabeling trick from (Kaelbling, 1993; Andrychowicz et al., 2017) to assign binary rewards to the collected trajectories. They apply two simple tricks on top of relabeling:

summaries: In this paper, the authors focus on the problem of goal conditioned reinforcement learning. Specifically, the authors consider the setting where the agent only observes vision as input and the ground truth state is not observable by the agent. In this setting, it is hard to specify a reward function since the reward function has to compute rewards from images.

abstractText: To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, especially in environments with deformable objects. An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor inputs to actions. However, if this policy is trained with reinforcement learning, then without a state estimator, it is hard to specify a reward function based on high-dimensional observations. To meet this challenge, we propose a simple indicator reward function for goal-conditioned reinforcement learning: we only give a positive reward when the robot’s observation exactly matches a target goal observation. We show that by relabeling the original goal with the achieved goal to obtain positive rewards (Andrychowicz et al., 2017), we can learn with the indicator reward function even in continuous state spaces. We propose two methods to further speed up convergence with indicator rewards: reward balancing and reward filtering. We show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards. We show that our method can perform complex tasks in continuous state spaces such as rope manipulation from RGB-D images, without knowledge of the ground-truth state.

authors:
Pulkit Agrawal
Ashvin V Nair
Pieter Abbeel
Jitendra Malik
Sergey Levine

title: Learning to poke by poking: Experiential learning of intuitive physics

venue: In NIPS,

authors:
Marcin Andrychowicz
Filip Wolski
Alex Ray
Jonas Schneider
Rachel Fong
Peter Welinder
Bob McGrew
Josh Tobin
OpenAI Pieter Abbeel
Wojciech Zaremba

title: Hindsight experience replay

authors:
Marcin Andrychowicz
Bowen Baker
Maciek Chociej
Rafal Jozefowicz
Bob McGrew
Jakub Pachocki
Arthur Petron
Matthias Plappert
Glenn Powell
Alex Ray

title: Learning dexterous in-hand manipulation

venue: arXiv preprint arXiv:1808.00177,

title: Learning dexterous in-hand manipulation

authors:
Yevgen Chebotar
Ankur Handa
Viktor Makoviychuk
Miles Macklin
Jan Issac
Nathan Ratliff
Dieter Fox

title: Closing the sim-to-real loop: Adapting simulation randomization with real world experience

venue: arXiv preprint arXiv:1810.05687,

authors:
Marco Cusumano-Towner
Arjun Singh
Stephen Miller
James F O’Brien
Pieter Abbeel

title: Bringing clothing into desired configurations with limited perception

venue: In ICRA,

authors:
Frederik Ebert
Chelsea Finn
Alex X Lee
Sergey Levine

title: Self-supervised visual planning with temporal skip connections

venue: arXiv preprint arXiv:1710.05268,

authors:
Frederik Ebert
Sudeep Dasari
Alex X Lee
Sergey Levine
Chelsea Finn

title: Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning

venue: arXiv preprint arXiv:1810.03043,

authors:
Frederik Ebert
Chelsea Finn
Sudeep Dasari
Annie Xie
Alex Lee
Sergey Levine

title: Visual foresight: Model-based deep reinforcement learning for vision-based robotic control

venue: arXiv preprint arXiv:1812.00568,

authors:
Kuan Fang
Yuke Zhu
Animesh Garg
Andrey Kurenkov
Viraj Mehta
Li Fei-Fei
Silvio Savarese

title: Learning task-oriented grasping for tool manipulation from simulated self-supervision

authors:
Chelsea Finn
Sergey Levine

title: Deep visual foresight for planning robot motion

venue: In Robotics and Automation (ICRA),

authors:
Chelsea Finn
Ian Goodfellow
Sergey Levine

title: Unsupervised learning for physical interaction through video prediction

venue: In Advances in neural information processing systems,

authors:
Chelsea Finn
Xin Yu Tan
Yan Duan
Trevor Darrell
Sergey Levine
Pieter Abbeel

title: Deep spatial autoencoders for visuomotor learning

authors:
Chelsea Finn
Tianhe Yu
Tianhao Zhang
Pieter Abbeel
Sergey Levine

title: One-shot visual imitation learning via meta-learning

venue: arXiv preprint arXiv:1709.04905,

authors:
Carlos Florensa
Jonas Degrave
Nicolas Heess
Jost Tobias Springenberg
Martin Riedmiller

title: Self-supervised learning of image embedding for continuous control

venue: Workshop at NIPS

authors:
Shixiang Gu
Ethan Holly
Timothy Lillicrap
Sergey Levine

title: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

authors:
Sandy H Huang
Jia Pan
George Mulcaire
Pieter Abbeel

title: Leveraging appearance priors in nonrigid registration, with application to manipulation of deformable objects

venue: In IROS,

authors:
Shervin Javdani
Sameep Tandon
Jie Tang
James F O’Brien
Pieter Abbeel

title: Modeling and perception of deformable one-dimensional objects

title: Learning to achieve goals

venue: In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, Chambery, France,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: Proceedings of the 3rd International Conference on Learning Representations (ICLR),

authors:
Petar Kormushev
Sylvain Calinon
Darwin G Caldwell

title: Robot motor skill coordination with em-based reinforcement learning

authors:
Alex X Lee
Max A Goldstein
Shane T Barratt
Pieter Abbeel

title: A non-rigid point and normal registration algorithm with applications to learning from demonstrations

venue: In ICRA, pp

authors:
Sergey Levine
Chelsea Finn
Trevor Darrell
Pieter Abbeel

title: End-to-end training of deep visuomotor policies

venue: The Journal of Machine Learning Research,

authors:
Timothy P Lillicrap
Jonathan J Hunt
Alexander Pritzel
Nicolas Heess
Tom Erez
Yuval Tassa
David Silver
Daan Wierstra

title: Continuous control with deep reinforcement learning

authors:
Stephen Miller
Mario Fritz
Trevor Darrell
Pieter Abbeel

title: Parametrized shape models for clothing

authors:
Ashvin V Nair
Vitchyr Pong
Murtaza Dalal
Shikhar Bahl
Steven Lin
Sergey Levine

title: Visual reinforcement learning with imagined goals

venue: Advances in Neural Information Processing Systems,

authors:
Xue Bin Peng
Angjoo Kanazawa
Sam Toyer
Pieter Abbeel
Sergey Levine

title: Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow

venue: arXiv preprint arXiv:1810.00821,

authors:
Calder Phillips-Grafflin
Dmitry Berenson

title: A representation of deformable objects for motion planning with no physical simulation

venue: ICRA

authors:
Lerrel Pinto
Marcin Andrychowicz
Peter Welinder
Wojciech Zaremba
Pieter Abbeel

title: Asymmetric actor critic for image-based robot

authors:
Fereshteh Sadeghi
Sergey Levine

title: Cad2rl: Real single-image flight without a single real image

venue: arXiv preprint arXiv:1611.04201,

authors:
Tom Schaul
Daniel Horgan
Karol Gregor
David Silver

title: Universal value function approximators

authors:
Connor Schenck
Dieter Fox

title: Guided policy search with delayed sensor measurements

venue: arXiv preprint arXiv:1609.03076,

authors:
John Schulman
Alex Lee
Jonathan Ho
Pieter Abbeel

title: Tracking deformable objects with point clouds

authors:
Pierre Sermanet
Kelvin Xu
Sergey Levine

title: Unsupervised perceptual rewards for imitation learning

venue: arXiv preprint arXiv:1612.06699,

authors:
Jie Tan
Tingnan Zhang
Erwin Coumans
Atil Iscen
Yunfei Bai
Danijar Hafner
Steven Bohez
Vincent Vanhoucke

title: Sim-to-real: Learning agile locomotion for quadruped robots

authors:
Josh Tobin
Rachel Fong
Alex Ray
Jonas Schneider
Wojciech Zaremba
Pieter Abbeel

title: Domain randomization for transferring deep neural networks from simulation to the real world

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

venue: IROS

authors:
Ping Chuan Wang
Stephen Miller
Mario Fritz
Trevor Darrell
Pieter Abbeel

title: Perception for the manipulation of socks

authors:
David Warde-farley
Tom Van De Wiele
Tejas Kulkarni
Catalin Ionescu
Steven Hansen
Volodymyr Mnih

title: Unsupervised Control through Non-parameteric Discriminative Rewards

venue: ICLR, pp

authors:
Ali Yahya
Adrian Li
Mrinal Kalakrishnan
Yevgen Chebotar
Sergey Levine

title: Collective robot reinforcement learning with distributed asynchronous guided policy search

authors:
Tianhe Yu
Gleb Shevchuk
Dorsa Sadigh
Chelsea Finn

title: Unsupervised visuomotor control through distributional planning networks. https://github.com/tianheyu927/dpn, 2019a

title: Unsupervised visuomotor control through distributional planning networks. Robotics: Science and Systems, 2019b

authors:
Yuke Zhu
Ziyu Wang
Josh Merel
Andrei Rusu
Tom Erez
Serkan Cabi
Saran Tunyasuvunakool
János Kramár
Raia Hadsell
Nando de Freitas

title: Reinforcement and imitation learning for diverse visuomotor skills

venue: arXiv preprint arXiv:1802.09564,

text:
1 INTRODUCTION
To perform robot manipulation tasks, a low dimensional state of the environment typically needs to be estimated. In reinforcement learning, this state is also used to compute the reward function. However, designing a state estimator can be difficult, especially in environments with deformable objects, as shown in Figure 1. An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor input to actions. However, without a state estimator, it is hard to specify a reward function based on high-dimensional observations.
Past efforts to use reinforcement learning for robotics have avoided this issue in a number of ways. One common approach is to use extra sensors to determine the state of the environment during training, even if such sensors are not available at test time. Examples of this include using another robot arm to hold all relevant objects (Levine et al., 2016), placing an IMU sensor (Gu
et al., 2017; Yahya et al., 2017), motion capture markers on such objects (Kormushev et al., 2010), or ensuring that all relevant objects are placed on scales (Schenck & Fox, 2016).
However, such instrumentation is not always easy to set up for each task. This is especially true for deformable object manipulation, such as rope or cloth manipulation, in which every part of the object must be instrumented in order to measure the full state of the entire object. Attaching such sensors to food or granular material would present additional difficulties.
We present an alternative approach for goal-conditioned reinforcement learning for specifying rewards using raw (e.g. high-dimensional and continuous) observations without requiring explicit state estimation or access to the ground-truth state of the environment. We achieve this using a
simple indicator reward function, which only gives a positive reward when the robot’s observation exactly matches a target goal observation. Naturally, in continuous state spaces, we do not expect any two observed states to be identical. Surprisingly, we show that we can learn with such an indicator reward, even in continuous state spaces, if we use goal relabeling (Kaelbling, 1993; Andrychowicz et al., 2017), which relabels the original goal with the achieved observation such that a positive reward is given. As the indicator rewards produce extreme sparse positive rewards, we further introduce reward balancing to balance the positive and negative rewards, as well as reward filtering to filter out uncertain rewards.
We show theoretically that the indicator reward results in a policy with bounded suboptimality compared to the ground-truth reward. We also empirically show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards, even though our method only operates on raw observations and does not have access to the ground-truth state. We demonstrate that an indicator reward can be used to teach a robot complex tasks such as rope manipulation from RGB-D images, without knowledge of the ground-truth state during training. Videos of our method can be found at https://sites.google.com/view/image-rl.

heading: 2 RELATED WORK

heading: 2.1 OBTAINING GROUND-TRUTH STATE FOR TRAINING

text:
Adding sensors: To obtain ground-truth states for calculating rewards, one approach is to perform state estimation. However, such an approach can be noisy and challenging to implement, especially for the deformable objects that we study in this work. Another approach is to add extra sensors during training to accurately record the state. For example, in past work, one robot arm (covered with a cloth at training time) is used to rigidly hold and move an object, while another robot arm learns to manipulate the object (Levine et al., 2016). In such a case, the object position can be inferred directly from the position of the robot gripper that is holding it. In other work on teaching a robot to open a door, an IMU sensor is placed on the door handle to determine the rotation angle of the handle and whether or not the door has been opened (Gu et al., 2017; Yahya et al., 2017). One can also ensure that all relevant objects for a task are placed on scales (Schenck & Fox, 2016) or affixed with motion capture markers to obtain a precise estimate of their position (Kormushev et al., 2010). However, such instrumentation is challenging for deformable objects, granular material, food, or other settings. Further, such instrumentation is costly and time-consuming to setup; hence most of these previous approaches assume that such instrumentation is only available at training time and these methods do not allow further fine-tuning of the policy after deployment.
Training in simulation: Another common approach is to train the policy entirely in simulation in which ground-truth state can be obtained from the simulator (Fang et al., 2018; Andrychowicz et al., 2018b; Zhu et al., 2018; Sadeghi & Levine, 2016; Pinto et al., 2018). Many approaches have been explored to try to transfer such a policy from simulation to the real world, such as domain randomization (Tobin et al., 2017) or building a more accurate simulator (Tan et al., 2018; Chebotar et al., 2018). However, obtaining an accurate simulator is often very challenging, especially if the simulator differs from the real-world in unknown ways. Further, building the simulator itself can be fairly complex. Because these methods require the ground-truth state to obtain the reward function, they require training in a simulator and do not allow further fine-tuning after deployment in the real world; our method, in contrast, does not require the ground-truth state for the reward function.

heading: 2.2 ROBOT LEARNING WITHOUT GROUND-TRUTH STATE

text:
Learning a reward function without supervision: One line of work for learning a reward function is to first learn a latent representation and then derive the reward function based on the distance in the embedding space, such as cosine similarity. The representation can be learned by maximizing the mutual information between the achieved goal and the intended goal (Warde-farley et al., 2018), reconstruction of the observation with VAE (Nair et al., 2018), or learning to match keypoints with spatial autoencoders (Finn et al., 2016b). Recent work also explicitly learns a representation that is suitable for gradient based optimizer and then use it for specifying rewards (Yu et al., 2019b). However, there is no guarantee that these learned representation are suitable for deriving rewards for control. In addition, if the representation is pre-trained, it may also be incorrect in some parts of the observation space which can be exploited by the agent. Our approach is much simpler in that
the reward function does not have any parameters that need to be learned and we empirically show better performance to some reward learning approaches.
Changing the optimization: Another approach is to forego maximizing a sum of rewards as is typically done in reinforcement learning and instead optimize for another objective. For example, one method is to choose one-step greedy actions based on a learned one-step inverse dynamics model; after training, the policy is then applied directly to a multi-step goal (Agrawal et al., 2016). An alternative method is to learn a predictive forward dynamics model directly in a high-dimensional state space and use visual model-predictive control (Finn et al., 2016a; Finn & Levine, 2017; Ebert et al., 2017; 2018a;b). Although these methods have shown some promise, predicting future highdimensional observations (such as images or depth measurements) is challenging. Another approach is to obtain expert demonstrations and define an objective as trying to imitate the expert (Sermanet et al., 2018; 2016; Peng et al., 2018; Finn et al., 2017). Our approach, however, applies even when demonstrations are not available.

heading: 2.3 MANIPULATING DEFORMABLE OBJECTS

text: Deformable object manipulation presents many challenges for both perception and control. One approach to the perception problem is to perform non-rigid registration to a deformable model of the object being manipulated (Huang et al., 2015; Lee et al., 2015; Schulman et al., 2013; Wang et al., 2011; Javdani et al., 2011; Miller et al., 2011; Cusumano-Towner et al., 2011; Phillips-Grafflin & Berenson, 2014). However, such an approach is often slow, leading to slow policy learning, and can produce errors, leading to poor policy performance. Further, such an approach often requires a 3D deformable model of the object being manipulated, which may be difficult to obtain. Our approach applies directly to high-dimensional observations of the deformable object and does not require a prior model of the object being manipulated.

heading: 3 PROBLEM FORMULATION

text: In reinforcement learning, an agent interacts with the environment over discrete time steps. In each time step t, the agent observes the current state st and takes an action at. In the next time step, the agent transitions to a new state st+1 based on the transition dynamics p(st+1|st, at) and receives a reward rt+1 = r(st, at, st+1). The objective for the agent is to learn a policy π(at|st) that maximizes the expected future return R = E [∑∞ t=0 γ trt+1 ] , where γ is a discount factor.

heading: 3.1 GOAL-REACHING REINFORCEMENT LEARNING

text:
In order for the agent to learn diverse and general skills, we define a goal reaching MDP (Schaul et al., 2015; Andrychowicz et al., 2017) as follows: In the beginning of each episode, a goal state sg is sampled from a goal distribution G. We learn a goal conditioned policy π(at|st, sg) that tries to reach any goal state from the goal distribution. We use a goal conditioned reward function rt = r(st+1, sg) and optimize for Esg∼G [∑∞ t=0 γ trt ] . The transition dynamics p(st+1|st, at) of the environment remain independent of the goal.
In many real-world scenarios, it is often difficult to construct a well-shaped reward function. Past work has shown that sparse rewards, combined with an appropriate learning algorithm, can achieve better performance than poorly-shaped dense rewards in goal-reaching environments (Andrychowicz et al., 2017). We thus define a sparse reward function that only makes the binary decision of whether the goal is reached or not. Specifically, let S+(sg) be a subset of the state space such that any state in this set is determined to be sufficiently close to sg (in some unknown metric); in other words, if the environmental state is within S+(sg), then the task of reaching sg can be considered to be achieved.1 Naturally, we can assume that sg ∈ S+(sg). A binary reward function can then be defined as
r(st+1, sg) = { R+ st+1 ∈ S+(sg) R− st+1 /∈ S+(sg),
(1)
where R+ and R− are constants representing the rewards received for achieving the goal and failing to achieve the goal, respectively.
1S+ is a function that maps from the state space to a subset of the space.

heading: 3.2 REWARDS FROM IMAGES

text:
In many cases, the ground-truth state st is unknown and we cannot directly use the true reward function defined in Equation 1. Instead, the agent observes high-dimensional observations ot from sensors, from which we must instead define a proxy reward function r̂(ot+1, og). The question now becomes how to choose r̂ to be optimal for reinforcement learning?
The most common approach in robotics is to perform state estimation. However, in many cases, the state estimator might be hard to obtain, such as for deformable object manipulation, e.g. laundry folding or food preparation. We therefore investigate whether an alternative reward function that does not depend on state estimation can be used. Specifically, let us consider a general reward function defined in observation space of the form
r̂(ot+1, og) = { R+ ot+1 ∈ Ô+(og) R− ot+1 /∈ Ô+(og),
(2)
where og is a representation of the goal in observation space and Ô+(og) is a subset of the observation space for which we will give positive rewards. A number of past approaches have proposed various methods for learning an observation-based reward function r̂(ot+1, og) (Warde-farley et al., 2018; Nair et al., 2018; Florensa et al., 2019; Yu et al., 2019b). However, these approaches do not analyze the properties needed by such a reward function to enable optimal learning. Next we will investigate trade-offs between different choices of Ô+(og) and how they will affect policy training time when trained with rewards of r̂(ot+1, og).

heading: 4 REWARD MISCLASSIFICATIONS

text:
We will now investigate how to design a good proxy reward function r̂(ot+1, og), based on raw sensor observations, that we can use to train the policy; we desire for the policy trained with r̂(ot+1, og) to optimize the original reward r(st+1, sg) based on the ground-truth state (which we do not have access to). Our first insight into choosing a good
proxy reward function r̂(ot+1, og) is that we should think about reward functions in terms of false positives and false negatives. Let us define a false positive reward to occur when the agent receives a positive reward based on our proxy reward function r̂(ot+1, og) when it would have received a negative reward based on the original reward function r(st+1, sg). In other words, suppose that an unknown function f maps from an observation to its corresponding ground-truth state, st = f(ot). The function f exits as we assume that the environment is fully observable and no two states will have the same observation. Then a false positive reward occurs when ot+1 ∈ Ô+(og) while f(ot+1) /∈ S+(sg). Similarly, a false negative reward can be defined. Intuitively, both false positive rewards and false negative rewards can negatively impact learning. However, for any estimated reward function r̂(ot+1, og), we will have either false positives or false negatives (or both) unless we have access to a perfect state estimator. To design a good proxy reward function, we must ask: which will more negatively affect learning: false positives or false negatives?
The two types of mistakes are not symmetric. As we will see, a false positive reward can significantly hurt policy learning, while a false negative reward is much more tolerable. Under a false positive reward, the agent receives a positive reward (under the proxy reward function r̂(o, og)) for reaching some observation o, even though the agent should receive a negative reward based on the corresponding ground-truth state f(o) under the original reward function r(st, sg). This false positive reward will encourage the agent to continue to try to reach the state f(o), even though reaching this state does not achieve the original task since f(o) /∈ S+(sg). On the other hand, false negative rewards are much more tolerable. Under a false negative, the agent observes some observation o such that f(o) ∈ S+(sg) but the agent receives a negative reward.
However, if the agent still receives a positive reward for some other observation o′ such that f(o′) ∈ S+(sg), then the agent can still learn to reach the goal states S+(sg), though learning might be slower and the learned policy may be suboptimal.
We provide a simple example to verify this intuition. Consider a robot arm reaching task, with the observation space O ∈ R3 being the 3D position of the end-effector (EE). Note that here the observation space coincides with the state space and the agent directly observes the states. The action space A ∈ R3 controls the position of EE. The true reward is defined by S+(sg) = {o | ‖st − sg‖2 < }. We define two types of noisy reward functions used for training. The reward function r̂FP gives the same rewards as the true reward function, except that with a probability of pFP (False Positive Rate), a negative reward will be flipped to a positive reward. The reward function r̂FN can be similarly defined, where a positive reward will be flipped to a negative reward with a probability of pFN (False Negative Rate). For this experiment, we use a standard reinforcement learning algorithm DDPG (Lillicrap et al., 2016) combined with goal relabeling (Andrychowicz et al., 2017). The learning performance of this same algorithm with different noisy rewards can be shown in Figure 2. We can see that the agent is able to learn the task even with a very large false negative rate. But when the false positive rate increases, the performance sharply decreases.

heading: 5 APPROACH

heading: 5.1 INDICATOR REWARDS

text:
Following this idea, we propose using a proxy reward function that does not have any false positive rewards. To do so, we will use an extreme reward function of Ô+(og) = {og}. In other words, we will use an indicator reward function:
r̂ind(ot+1, og) = { R+ ot+1 = og R− ot+1 6= og,
(3)
It should be clear that this reward function will have no false positives, since the reward is positive only if ot+1 = og , which implies that f(ot+1) = f(og), or equivalently, st+1 = sg . As sg ∈ S+(sg) by definition, all positive rewards are true positives. However, this reward function is extremely sparse and has many false negatives. In fact, without goal relabeling, in continuous state spaces, we would expect all rewards to be negative under this indicator reward function, as no two observations in continuous spaces will ever be identical. Next, we will describe how to learn with this reward function with goal relabeling.

heading: 5.2 GOAL RELABELING FOR OFF-POLICY LEARNING

text: Fortunately, for off-policy multi-goal learning, we can adopt the goal relabeling technique introduced in (Kaelbling, 1993; Andrychowicz et al., 2017) to learn the goal-conditioned Q-function. Suppose that some transitions (ot, at, ot+1) are observed when the agent takes an action at ∼ π(ot, og) with a goal of og . Because Q-learning is an off-policy reinforcement learning algorithm, we can replace the goal observation og with any other observation og′ in our Bellman update of the Q-function. This works because the transition dynamics p(st+1|st, at), and likewise the observation transition dynamics p(ot+1|ot, at), are independent of the goal og . Specifically, for some transitions, we will choose to replace og with the observation ot+1. By re-labeling og with ot+1 and using our indicator reward function, we will have that r̂ind(ot+1, og) = r̂ind(ot+1, ot+1) = R+. Thus, using goal relabeling, we can get positive rewards, even when using an indicator reward function in continuous state spaces.

heading: 5.3 REWARD BALANCING AND FILTERING

text:
As mentioned above, after sampling a batch of data, we train the Q-function with goal relabeling. We use three different strategies for choosing which goals to use for relabeling: with probability p1, we relabel og with ot+1, which will receive a positive reward under our indicator reward function. With probability p2, we relabel the goal og with ot′ with an observation from some future time t′ step within the episode. The indicator reward function will most likely give a negative reward in this case, which is possibly a false negative because the new goal is possibly considered achieved based on the state-based, ground-truth reward funciton. Finally, with probability p3, we use the original
goal (with no relabeling), which will again most likely give a negative reward under the indicator reward function; as before, this might be a false negative.
Reward balancing: We refer to “reward balancing” as setting p1 = p2 = 0.45 and p3 = 0.1, leading us to receive positive rewards approximately 0.45 of the time and negative rewards approximately 0.55 of the time. Thus the ratio of positive and negative rewards that we use to train the Q-function are approximately balanced, even with indicator rewards. From another perspective, p1 and p2 determine the relative frequency between providing positive rewards and propagating rewards to other timesteps in the episode. Additionally, training with a small fraction of the original goals (i.e. p3) can be seen as a regularization which ensures that the distribution of the relabeled goals moves towards the original goal distribution.
Reward filtering: While false negative rewards do not hurt learning as much as false positives, we still wish to avoid them if possible to improve the convergence time of the learned policy. We achieve this using “reward filtering,” in which we filter out transitions that we suspect of having a high chance of being false negatives. We refer to “reward filtering” as discarding a sampled transition if its Q value is above a threshold q0. If the assigned reward is negative based on the proxy reward function but the Q-value is sufficiently high, then there is a chance that this reward a false negative. To reduce the fraction of false negatives, we filter out such transitions and do not use them for training.
We can estimate how to set the threshold q0 as follows: for a given transition (ot, at, ot+1), if ot+1 = og , we know that r̂ind(ot+1, og) = R+. In this case, Q∗(ot, at, og) = R+/(1 − γ), where Q∗ is the optimal Q-function, assuming the optimal policy will continue to receive (discounted) positive rewards in the future. Similarly, if ot+1 6= og , then r̂ind(ot+1, og) = R−. Since we know that the policy starting from ot will thus receive at least one negative reward before receiving positive rewards, then Q∗(ot, at, og) ≤ R− + γR+/(1 − γ). Thus, we can set a threshold q0, where R− + γR+/(1 − γ) < q0 < R+/(1 − γ); if we find that Q(ot, og, at) > q0, then the corresponding reward r̂ind(ot+1, og) is likely to be a false negative (assuming that the Q-function has been trained well); we thus filter out such rewards, to reduce the number of false negatives that we use for training. We can see that q0 is set to a rather conservative fitlering value. Additionally, the Q-function is initialized to a relatively low value to avoid overestimation of the Q-function which can lead to incorrect filtering in the beginning of the training.

heading: 6 ANALYSIS

text: In this section, we analyze the performance of learning with indicator rewards. We first interpret the goal conditioned Q-function as a measure of the time to reach one observation from another.

heading: 6.1 MINIMUM REACHING TIME INTERPRETATION

text:
Let us define d = Dπ(ot, at, Ô+(og)) as the number of time steps it takes for the policy π to go from the current observation ot, starting with action at, to reach the set Ô+(og) of goal observations. For simplicity, we assume that, once the agent receives a positive reward, it will take actions to continue to receive positive rewards. The Q-function can be written as
Qπ(ot, at, og) = R− + γR− + ...+ γ d−1R− + γ dR+ + γ d+1R+ + ...
= γd
1− γ (R+ −R−) +
1
1− γ R−.
(4)
Now it can be easily seen that, as long as R+ > R−, Qπ is strictly monotonically decreasing w.r.t. d. As such, maximizing Qπ over π is equivalent to minimizing the time the agent takes to reach the goal Ô+(og). Note that this is true for varying definitions of Ô+(og); thus the policy trained under the true reward function (Equation 1) will minimize Dπ(ot, at, O+(og)) whereas the policy trained under the indicator reward function will minimize Dπ(ot, at, {og}) (slightly overloading notation for Dπ). Below we will show how this interpretation of the policy’s behavior at convergence can lead to a simple analysis of the suboptimality of the learned policy under the indicator reward.

heading: 6.2 ANALYSIS OF SUB-OPTIMALITY

text:
Due to the false negative rewards given by the indicator function r̂ind, the learned policy may not be optimal with respect to the original reward function r(st+1, sg) defined in Equation 1. Here we
give the worst case bound for the policy learned with the indicator reward. Following the minimum reaching time interpretation of the previous section, we evaluate the performance of the policy in terms of the time it takes to reach the set of goal observations O+(og) from the current observation. Given ot, og , denote t1 as the minimum number of time steps to reach from ot to the set of true goal observations, i.e. t1 = D(ot, O+(og)). Let t2 = D(ot, og) be the minimum time to reach from ot to og . Define the diameter of this goal observation set as d = max{D(o1, o2)|o1, o2 ∈ O+(og)}. From the optimality of t2, we know that
t2 ≤ D(ot, o) +D(o, og),∀o ∈ O+(og).
Thus,
t2 ≤ min o∈O+(og) D(ot, o) +D(o, og) ≤ t1 + d. (5)
From the analysis in the previous section, the optimal policy which optimizes the indicator reward will reaches og in t2 time steps; since og ∈ O+(og), we know that this policy will reach O+(og) in some time t3 ≤ t2. Also recall that we have defined t1 such that the optimal policy under the true reward function of Equation 1 will reach O+(og) in t1 steps. Thus t3/t1 ≤ (t1 + d)/t1 is an upper bound on the suboptimality of the policy trained under the indicator reward, at convergence.

heading: 7 EXPERIMENTS

text:
Our experiments address the following questions:
1. In the case of visual input, how much are the sample efficiency and the final performance affected without assuming access to the ground-truth reward?
2. How much does reward balancing and filtering improve learning efficiency? 3. Can our method scale to real world robotic tasks?
We denote our method, which uses indicator rewards with reward balancing and filtering, as Indicator+Balance+Filter. We compare our method with the following baselines:
• Oracle: This method assumes access to the ground-truth reward from state space r(st, sg). • Indicator: This is an ablation of our method, without reward balancing and filtering. • Auto Encoder (AE): We train an autoencoder with an L2 reconstruction loss of the im-
age observation, jointly with the RL agent. We then use cosine similarity in the learned embedding space to provide dense rewards, as similarly compared in (Warde-farley et al., 2018). Specifically, assuming the learned encoding of an observation o is φ(o) after L2 normalization, the reward will be r(o, og) = max(0, φ(o)Tφ(og)).
• Variational Auto Encoder (VAE): Similarly to the AE baseline, a VAE is jointly trained with the RL agent to provide rewards, as done in (Nair et al., 2018). For a fair comparison, the goal sampling strategy for this baseline is kept the same a other approaches.
• Distributional Planning Network (DPN) (Yu et al., 2019b): DPN aims to learn a representaiton that is suitable for a gradient based planner to reach a goal observation. Following (Yu et al., 2019b), we first pre-train DPN using randomly generated samples and then use the learned representation for giving rewards.
Only Oracle uses the ground-truth, state-based reward function. We use the standard off-policy learning algorithm DDPG (Lillicrap et al., 2016) with goal relabeling (Andrychowicz et al., 2017). For methods without reward balancing, we re-label the current goal with an achieved goal sampled uniformly from one of the future time steps within the same episode with a probability of 0.9; otherwise, the original goals are used. For all the environments, the ground-truth rewards are based on the L2 distance in the state space: R+ if ‖st+1 − sg‖ ≤ and R− otherwise. More details on algorithms, architectures, and hyperparameters can be found in the Appendix.
We first evaluate all the methods in a set of simulated environments in MuJoCo (Todorov et al., 2012), where both current and goal observation are given by RGB-D images:
• Reacher: Teach a two-link arm to reach a randomly located position in 2D space.
• FetchReach: Move the end effector of the Fetch robot to a random position in 3D.
• RopePush: Push a rope from a random initial configuration into a target configuration.
The first two environments above are standard environments from Gym (Andrychowicz et al., 2018a). For the more complex RopePush task, the robot needs to push a 15-link rope to a targeted pose, as shown in Figure 1. To accelearate learning, we fix the orientation of the gripper and parameterize the action as (x1, y1, x2, y2) ∈ R4, denoting the starting and ending position of one push from the gripper. We generate the initial rope pose by giving the rope a random push from a fixed location. The goal poses are generated by giving the rope two more pushes based on the initial push (these pushes are hidden from the policy). The robot can give three pushes to push the rope to the goal pose. More details on environments can be found in Appendix A.
The results are shown in Figure 3. We see that our method (Indicator+balance+Filter) achieves nearly the same performance as the Oracle even though our method operates only from RGB-D images and does not have access to the ground-truth state. For the RopePush environment, only the Oracle and our method are able to learn to achieve the task to a reasonable accuracy. For AE, VAE and DPN, the learned representation may not lead to a perfect reward function everywhere and the agent will exploit states that yield a high proxy rewards, even though the goal is not achieved. Instead, our method does not require learning a reward function and outperforms these baselines.
In Appendix C, we perform an ablation where we removed either Balance or Filter from our method. These experiments show that both Balance and Filter are important for optimal performance across the environments tested. In Appendix D, we show experiments in which the ground-truth states are used as inputs to the policy but are not used to compute the rewards. In these experiments, we also see that our method has a similar performance to the Oracle and outperforms the other approaches.

heading: 7.1 INDICATOR REWARDS WITH REAL IMAGES

text: Using RGB-D observations and goals, we train a Sawyer robot for a 3-dimensional reaching task. Figure 4 shows example observation and goal images as well as the distance to goal throughout training. As before, our method (Indicator+Balance+Filter) performs similarly to the Oracle in terms of final goal distance; the baseline of Indicator Rewards without balancing or filtering performs significantly worse and diverges in the end due to many negative rewards, many of which are false negative.

heading: 8 CONCLUSION

text:
In this work, we show that we can train a robot to perform complex manipulation tasks directly from high-dimensional images, without requiring access to the ground-truth state in either the policy input or the reward function. We empirically show that our method enables a robot to learn complex skills for manipulating deformable objects, for which state estimation is often challenging, including a real-world experiment.
We provide a theoretical analysis which shows that the optimal policy under the indicator reward has a bounded sub-optimality compared to the optimal policy under the ground-truth reward. We hope that our method will enable robot learning in the real world in cases where it is difficult to add extra sensors or accurately simulate the environment.

heading: A ENVIRONMENT DETAILS

text:
During evaluation, for all environments, a binary sparse reward is given at each time step. A positive reward R+ = 1 is given when the goal is reached, i.e. ||st+1 − sg|| ≤ and a negative reward R− = −1 is given otherwise. Other environment details are summarized in Table 1. For the RopePush environment, to save the time for computing initial and goal configuration, 10,000 initial configurations of the rope are pre-computed and cached which are later used for training.
Sawyer robot experiment details: The observation is recorded with an Intel RealSense D435 depth camera. The goal observations are sampled by moving the robot arm to a uniformly sampled location in a cuboid of diagonal length 1.3m. The episode was considered successful if the end effector moved to within 0.1 m from goal location at the end of the episode (we used a time horizon of 25 steps). The trained policy performs position control and outputs end effector displacement within a range of -0.05m to 0.05m in each direction.

heading: B HYPER-PARAMETERS

text: All the experiments are run for two random seeds. The hyper-parameters of the training algorithm with indicator rewards are summarized in Table 2. For all experiments with visual observation, the parameters of the convolution layers are shared among the observation input and goal input. Due to the complexity of the RopePush environment, a spatial softmax layer (Levine et al., 2016) with an output size of 32 is applied before the fully connected layers.

heading: C ABLATION ANALYSIS

text:
We show different ablations of our methods in Figure 5. We can see that, in the RopePush environment, filtering is required for the policy to learn. On the other hand, the Reacher and FetchReach environments show that balancing is required for optimal performance. In all cases, Indicator+Balance+Filter consistently performs better than all the ablated methods. Thus, these results show that both balancing and filtering are important for optimal performance across a range of tasks.
With indicator rewards, we do not have any false positive rewards but may have many false negative rewards. In Figure 6, we show the accuracy of the given rewards using different approaches with indicator rewards. The ground-truth rewards are calculated based on the ground-truth states which we do not have during training and are only used for analysis. We can see that with a longer time horizon, reward balancing is more important for the Visual Reacher and Visual FetchReach environment, which significantly lower the false negative rates. On the other hand, both reward filtering and balancing are important in the Visual RopePush environment. In all the cases, the accuracy of the rewards are improved a lot and sometimes almost perfect with reward balancing and filtering and we can see that this is necessary for the good performance of using indicator rewards.

heading: D LEARNING WITH INDICATOR REWARDS WITH STATE INPUT

text:
The performances of different learning methods when learning with low dimensional state representation are shown in Figure 7. In all the environments, using indicator rewards with reward balancing and filtering achieves comparable performances to Oracle. Compare this method to the Indicator baseline which does not have reward balancing and filtering, Indicator+Balance+Filter achieves a better sample efficiency. Interestingly, in the FetchReach environment, the default distance threshold for receiving an R+ reward is set to 0.05. Thus, the policy that learns with this reward stops learning when the policy reach to such a distance to the goal, while the policy learned with indicator rewards keep reaching closer to the goals. This shows another benefit of using the indicator reward that the user does not need to tune the hyper-parameter to achieve the best performance.
E IMPLEMENTATION DETAILS OF BASELINES
• Auto Encoder (AE): The network architecture for the encoder is four convolutional layers each followed by a max-pooling layer. The latent code as a dimension of 32. The decoder is four up-pooling and up-convolution layers. The auto encoder is trained jointly with the RL agent by the reconstruction loss of the observation images in the sampled transition. The learning rate is set to 0.001. We then use cosine similarity in the learned embedding space to provide dense rewards, as similarly compared in (Warde-farley et al., 2018). Specifically, assuming the learned encoding of an observation o is φ(o) after L2 normalization, the reward will be r(o, og) = max(0, φ(o)Tφ(og)).
• Variational Auto Encoder (VAE) The VAE uses the same architecture and training procedure as the AE baseline. Following (Nair et al., 2018), we use a variant of the VAE, β-VAE with β = 5. Following (Nair et al., 2018), given the latent representation z, zg of the current observation o and the goal observation og , the rewards are given by
r(o, og) = −||z − zg||2,
which is the negative of the Euclidean distance.
• Distributional Planning Network (DPN) (Yu et al., 2019b) We use the released implementation (Yu et al., 2019a) and use the default hyperparameters. The network architecture for encoder is three convolutional layers, each with kernel size 5, stride 2. After each convolutional layer, there is layer normalization and sigmoid non-linearity. The output representation from CNN goes into a fully connected layer with latent code dimension of 128. For each environment, we collected 20,000 transitions from a random policy and performed 100,000 minibatch updates. The input dimension for each of the three environments is the same as the Rendered Dimension column specified in Table 1. Once the DPN is pre-trained, the representation is fixed during the training of the RL agent. Following (Yu et al., 2019b), given the latent representation z, zg of the current observation and the goal observation, the
rewards are given by
r(o, og) = − exp(||dDPN(z − zg, δ)||1),
where
dDPN(x, δ)i =
{ 1 2x 2 i for |xi| ≤ δ
δ|xi| − 12δ 2 otherwise
.
δ is set to 0.85. To avoid overflow of the rewards during the exponential, we normalize z and zg such that they have a norm of 1 after the representation is trained.

heading: F SENSITIVITY ON FILTERING THRESHOLD

text: In Section 5.3, we give the range of the filtering threshold q0. In previous experiments, we use the middle value of this range. Here, we train with different values of q0 within this range. The results for the rope pushing task is shown in Figure 8. We can see that the learning is very stable with different values of q0.