summaries: This is a well-written paper that discusses how to learn disentangled representations for the learning from demonstrations (LfD) task in robotics. It is shown that using weak-supervision on top of unsupervised learning frameworks (that use the variational autoencoder for instance) can work well in this case. These disentangled factors of variation in the data are shown to correspond well to the 'abstract concepts' of the human demonstrations. This is shown in the example of the PR2 robot dabbing demonstrations, including visual data as well robot trajectories.  Under the context of learning from demonstrations, the paper studies the problem of leaning interpretable low dimensional representations from high dimensional multimodal inputs using weak supervision. Paper argues that since robots and humans have different levels of abstractions and mechanisms, observation+action spaces between them are greatly misaligned which complicates learning by directly observing humans. However, the underlying concepts essential for tasks lie in a much lower-dimensional manifold. Learning this manifold effectively and in an interpretable way, especially using weak supervision, can significantly change how robots can acquire skills from demonstrations and generalize them to new unseen scenarios. Towards this end, the paper proposes to learn probabilistic generative models capturing high-level notions from demonstrations using variational inference. The strength of the paper is in demonstrating that conditional latent variable models can learn disentangled low dimensional represented using weak supervision; which authors effectively demonstrated using real-world experiments. My main reservations are in terms of the technical novelty of the paper and the narrow scope of experimental evaluation. This paper presents a way to learn from demonstrations with weak or no labels. The premise behind this paper is that even when humans provide labels during a demonstration, those labels often do not fully describe the data (e.g., the human may say "soft" when "fast" would also apply). This paper presents a technique that uses latent variables to model the uncertainty over a group of class labels that could describe the task (e.g., slow, soft, left-of-object). The variables are modeled such that the observation is conditionally independent of the human provided labels given the latent variables. This allows the human provided labels to be decoupled (or disentangled as the paper calls it) from the observations. By doing so, it is possible to have only partial labels (weak labels). This model was applied to a task where a human would teleoperate a robot arm and apply a dabbing motion in relation to an object in the scene. The operator would provide only one of several possible applicable labels for each demonstration. The results show that the models using the weak labeling out-performed models with no labeling.
abstractText: Robotic manipulation tasks, such as wiping with a soft sponge, require control from multiple rich sensory modalities. Human-robot interaction, aimed at teaching robots, is difficult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams. We treat the task of interpretable learning from demonstration as an optimisation problem over a probabilistic generative model. To account for the high-dimensionality of the data, a high-capacity neural network is chosen to represent the model. The latent variables in this model are explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations. We show that such alignment is best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables. Our approach is evaluated in the context of two table-top robot manipulation tasks performed by a PR2 robot – that of dabbing liquids with a sponge (forcefully pressing a sponge and moving it along a surface) and pouring between different containers. The robot provides visual information, arm joint positions and arm joint efforts. We have made videos of the tasks and data available see supplementary materials at: https://sites.google.com/view/weak-label-lfd.
authors.name: Yordan Hristov
authors.name: Subramanian Ramamoorthy
references.authors: Paszke Adam Gross Sam Chintala Soumith Chanan Gregory Yang Edward D Zachary Lin Zeming Desmaison Alban Antiga Luca Lerer Adam
references.title: Automatic differentiation in pytorch
references.venue: In Proceedings of Neural Information Processing Systems,
references.authors: Samuel K. Ainsworth Nicholas J. Foti Adrian K.C. Lee Emily B. Fox
references.title: oi-VAE: Output interpretable VAEs for nonlinear group factor analysis
references.venue: Proceedings of the 35th International Conference on Machine Learning,
references.authors: Brenna D Argall Sonia Chernova Manuela Veloso Brett Browning
references.title: A survey of robot learning from demonstration
references.venue: Robotics and autonomous systems,
references.authors: Shaojie Bai J Zico Kolter Vladlen Koltun
references.title: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling
references.venue: arXiv preprint arXiv:1803.01271,
references.authors: Apratim Bhattacharyya Bernt Schiele Mario Fritz
references.title: Accurate and diverse sampling of sequences based on a “best of many
references.venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Tian Qi Chen Xuechen Li Roger B Grosse David K Duvenaud
references.title: Isolating sources of disentanglement in variational autoencoders
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Xi Chen Yan Duan Rein Houthooft John Schulman Ilya Sutskever Pieter Abbeel
references.title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Adrià Colomé Gerhard Neumann Jan Peters Carme Torras
references.title: Dimensionality reduction for probabilistic movement primitives
references.venue: In 2014 IEEE-RAS International Conference on Humanoid Robots,
references.authors: Charles Fefferman Sanjoy Mitter Hariharan Narayanan
references.title: Testing the manifold hypothesis
references.venue: Journal of the American Mathematical Society,
references.authors: Stevan Harnad
references.title: The symbol grounding problem
references.venue: Physica D: Nonlinear Phenomena,
references.authors: Irina Higgins Loic Matthey Arka Pal Christopher Burgess Xavier Glorot Matthew Botvinick Shakir Mohamed Alexander Lerchner
references.title: beta-vae: Learning basic visual concepts with a constrained variational framework
references.authors: Yordan Hristov Daniel Angelov Michael Burke Alex Lascarides Subramanian Ramamoorthy
references.title: Disentangled relational representations for explaining and learning from demonstration
references.venue: In Conference on Robot Learning (CoRL),
references.authors: Dirk Husmeier
references.title: Introduction to learning bayesian networks from data
references.authors: Eric Jang Shixiang Gu Ben Poole
references.title: Categorical reparametrization with gumble-softmax
references.venue: In International Conference on Learning Representations (ICLR 2017). OpenReview
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Diederik P Kingma Max Welling
references.title: Auto-encoding variational bayes
references.venue: arXiv preprint arXiv:1312.6114,
references.authors: Durk P Kingma Shakir Mohamed Danilo Jimenez Rezende Max Welling
references.title: Semi-supervised learning with deep generative models
references.venue: In Advances in neural information processing systems,
references.authors: Jens Kober Betty Mohler Jan Peters
references.title: Learning perceptual coupling for motor primitives
references.venue: In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems,
references.authors: John E Laird Kevin Gluck John Anderson Kenneth D Forbus Odest Chadwicke Jenkins Christian Lebiere Dario Salvucci Matthias Scheutz Andrea Thomaz Greg Trafton
references.title: Interactive task learning
references.venue: IEEE Intelligent Systems,
references.authors: Francesco Locatello Stefan Bauer Mario Lucic Gunnar Raetsch Sylvain Gelly Bernhard Schölkopf Olivier Bachem
references.title: Challenging common assumptions in the unsupervised learning of disentangled representations
references.venue: In International Conference on Machine Learning,
references.authors: Michael D. Noseworthy Rohan Paul Subhro Roy Daehyung Park Nicholas Roy
references.title: Taskconditioned variational autoencoders for learning movement
references.authors: Alexandros Paraschos Christian Daniel Jan R Peters Gerhard Neumann
references.title: Probabilistic movement primitives
references.venue: In Advances in neural information processing systems,
references.authors: Antti Rasmus Mathias Berglund Mikko Honkala Harri Valpola Tapani Raiko
references.title: Semisupervised learning with ladder networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: MH Reibert
references.title: Hybrid position/force control of manipulators
references.venue: ASME, J. of Dynamic Systems, Measurement, and Control,
references.authors: Stéphane Ross Geoffrey Gordon Drew Bagnell
references.title: A reduction of imitation learning and structured prediction to no-regret online learning
references.venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,
references.authors: Elmar Rueckert Jan Mundo Alexandros Paraschos Jan Peters Gerhard Neumann
references.title: Extracting low-dimensional control variables for movement primitives
references.venue: In 2015 IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Stefan Schaal
references.title: Dynamic movement primitives-a framework for motor control in humans and humanoid robotics
references.venue: In Adaptive motion of animals and machines,
references.authors: Kihyuk Sohn Honglak Lee Xinchen Yan
references.title: Learning structured output representation using deep conditional generative models
references.venue: In Advances in neural information processing systems,
references.authors: Paul Vogt
references.title: The physical symbol grounding problem
references.venue: Cognitive Systems Research,
references.authors: Jacob Walker Carl Doersch Abhinav Gupta Martial Hebert
references.title: An uncertain future: Forecasting from static images using variational autoencoders
references.venue: In European Conference on Computer Vision,
references.authors: Nick Watters Loic Matthey Chris P Burgess Alexander Lerchner
references.title: Spatial broadcast decoder: A simple architecture for disentangled representations in vaes
sections.heading: 1 INTRODUCTION
sections.text: Learning from Demonstration (LfD) (Argall et al., 2009) is a commonly used paradigm where a potentially imperfect demonstrator desires to teach a robot how to perform a particular task in its environment. Most often this is achieved through a combination of kinaesthetic teaching and supervised learning—imitation learning (Ross et al., 2011). However, such approaches do not allow for elaborations and corrections from the demonstrator to be seamlessly incorporated. As a result, new demonstrations are required when either the demonstrator changes the task specification or the agent changes its context—typical scenarios in the context of interactive task learning (Laird et al., 2017). Such problems mainly arise because the demonstrator and the agent reason about the world by using notions and mechanisms at different levels of abstraction. An LfD setup, which can accommodate abstract user specifications, requires establishing a mapping from the high-level notions humans use—e.g. spatial concepts, different ways of applying force—to the low-level perceptive and control signals robot agents utilise—e.g. joint angles, efforts and camera images. With this in place, any constraints or elaborations from the human operator must be mapped to behaviour on the agent’s side that is consistent with the semantics of the operator’s desires.
  Concretely, we need to be able to ground (Vogt, 2002; Harnad, 1990) the specifications and symbols used by the operator in the actions and observations of the agent. Often the actions and the observations of a robot agent can be high-dimensional—high DoF kinematic chains, high image resolution, etc.—making the symbol grounding problem non-trivial. However, the concepts we need to be able to ground lie on a much-lower-dimensional manifold, embedded in the high-dimensional data space (Fefferman et al., 2016). For example, the concept of pressing softly against a surface manifests itself in a data stream associated with the 7 DoF real-valued space of joint efforts, spread across multiple time steps. However, the essence of what differentiates one type of soft press from another nearby concept can be summarised conceptually using a lower-dimensional abstract space. The focus of this work is finding a nonlinear mapping (represented as a high-capacity neural model) between such a
  low-dimensional manifold and the high-dimensional ambient space of cross-modal data. Moreover, we show that, apart from finding such a mapping, we can also shape and refine the low-dimensional manifold by imposing specific biases and structures on the neural model’s architecture and training regime.
  In this paper, we propose a framework that allows human operators to teach a PR2 robot about different spatial, temporal and force-related aspects of a robotic manipulation task, using tabletop dabbing and pouring as our main examples. They serve as concrete representative tasks that incorporate key issues specific to robotics (e.g. continuous actions, conditional switching dynamics, forceful interactions, and discrete categorizations of these). Numerous other applications require the very same capability. Our main contributions are:
  • A learning method which incorporates information from multiple high-dimensional modalities—vision, joint angles, joint efforts—to instill a disentangled low-dimensional manifold (Locatello et al., 2019). By using weak expert labels during the optimisation process, the manifold eventually aligns with the human demonstrators’ ‘common sense’ notions in a natural and controlled way without the need for separate post-hoc interpretation. • We release a dataset of subjective concepts grounded in multi-modal demonstrations. Us-
  ing this, we evaluate whether discrete latent variables or continuous latent variables, both shaped by the discrete user labels, better capture the demonstrated continuous notions.
sections.heading: 2 IMAGE-CONDITIONED TRAJECTORY & LABEL MODEL
sections.text: In the task we use to demonstrate our ideas, which is representative of a broad class of robotic manipulation tasks, we want to control where and how a robot performs an action through the use of a user specification defined by a set of coarse labels—e.g. “press softly and slowly behind the cube in the image”. In this context, let, x denote a K × T dimensional trajectory for K robot joints and a fixed time length T , y denote a set of discrete labels semantically grouped in N label groups G = {g1, . . . , gN} (equivalent to multi-label classification problem) and i denote an RGB image1. The labels y describe qualitative properties of x and x with respect to i—e.g. left dab, right dab, hard dab, soft dab, etc. We aim to model the distribution of demonstrated robot-arm trajectories x and corresponding user labels y, conditioned on a visual environment context i. This problem is equivalent to that of structured output representation (Walker et al., 2016; Sohn et al., 2015; Bhattacharyya et al., 2018)—finding a one-to-many mapping from i to {x,y} (one image can be part of the generation of many robot trajectories and labels). For this we use a conditional generative model, whose latent variables c = {cs, ce, cu} can accommodate the aforementioned mapping—see Figure 2. The meaning behind the different types of latent variables—cs, ce and cu—is elaborated in section 3.
  1What i actually represents is a lower-dimensional version of the original RGB image I. The parameters of the image encoder are jointly optimised with the parameters of the recognition and decoder networks
  We want the variability in the trajectory and label data to be concisely captured by c. We choose by design the dimensionality of the latent variables to be much lower than the dimensionality of the data. Therefore, c is forced to represent abstract properties of the robot behaviour which still carry enough information for the behaviour to be usefully generated—absolute notions like speed, effort, length, and relative spatial notions which are grounded with respect to the visual context. Another way to think of c is as a continuous version of y which can therefore incorporate nuances of the same label. In the existing literature discrete labels are usually represented as discrete latent variables—e.g. digit classes (Kingma et al., 2014).
  However, the discrete labels humans use are a rather crude approximation to the underlying continuous concepts—e.g. we may have the notion for a soft and a softer dab even though both would be alternatively labelled as soft. For this reason, we use continuous latent variables, shaped by discrete labels, to represent these subjective concepts.
  The joint distribution over x and y, conditioned on i, is modelled according to Eq. 1. A prior of choice—isotropic Gaussian, Uniform, Categorical—is placed over the latent variables c which are independent of the image.
  p(x,y|i) = ∫ p(x,y|i, c)p(c)dc = ∫ p(x|c, i)p(y|c)p(c)dc (1)
  We choose to represent the distribution over x and y as Gaussian and Multinomial distributions respectively. The parameters of these distributions are defined as nonlinear functions (of the image context and latent variables) which are represented as the weights θ of a neural network pθ—µθ(·) is a vector of means, σθ(·) is a vector of log variances and πθ(·) is a vector of probabilities.
  pθ(x|c, i) = N (x|µθ(c, i),σθ(c, i)) (2) pθ(y|c) = Cat(y|πθ(c)) (3)
  The parameters of pθ are optimised by maximising the variational lower bound (VLB) of the data distribution—see Eq. 4. We additionally optimise a recognition network qφ, parametrised by φ, which acts as an approximation to the posterior of c, also modelled as a Gaussian. The recognition network qφ is used to find good candidates of c, for given x and i, thus making the optimisation of θ, tractable (Kingma & Welling, 2013). The posterior is conditioned on the environment context as well, since i and c are conditionally dependent, once x is observed (Husmeier, 2005). Intuitively, we want some of the latent variables to represent relative spatial concepts—e.g. left of the cube in the image. For such concepts, under the same robot trajectory x we should get different latent codes, given different contexts i. The omission of y from qφ means that we might not be able to fully close the gap between the data distribution and its VLB, leading to a less efficient optimisation procedure (due to higher variance). That is mainly owing to the fact that we use weak supervision—only for some demonstrations do we have labels y. The derivation of and commentary on the optimised VLB are provided in the Supplementary Materials.
  log p(x,y|i) ≥ Eqφ(c|x,i)(log pθ(x,y|c, i))− DKL(qφ(c|x, i)||p(c)) (4) qφ(c|x, i) = N (c|µφ(x, i),σφ(x, i)) (5)
sections.heading: 3 WEAK LABELLING AND USER SPECIFICATION
sections.text: Interpretability through Weak Labels Even though the latent variables are constrained to be useful both for the tasks of trajectory and label generation, nothing forces them to carry the intuitive absolute and relative notions described in Section 2 in a factorised fashion—e.g. for dabbing, c1 to correspond to the effort applied when pressing against the table, c2 to correspond to how quickly the robot presses against the table, etc. Under an optimised model, those notions can be captured in c but with no guarantee that they will be represented in an interpretable way— i.e. c needs to be
  disentangled. We achieve disentanglement in the latent space by establishing a one-to-one correspondence between a subset of the latent axes—cs, ce—and the concept groups in G (Hristov et al., 2018). cs is optimised to represent spatial notions, ce—effort-related and temporal notions. The rest of the latent dimensions—cu—encode features in the data which don’t align with the semantics of the labels and their concept groups but are still necessary for good trajectory reconstruction (Figure 2). Under these assumptions the label likelihood from Eq. 3 becomes:
  pθ(y|{cs, ce}) = |G|∏ j 1{yj 6=∅}Cat(yj |πj(cj)) (6)
  Labelling is weak/restricted since each demonstration is given just a single label from a single concept group. This is meant to represent a more natural LfD scenario, where the main task of the expert is still to perform a number of demonstrations, rather than to exhaustively annotate each demonstration with relevant labels from all groups. For example, a demonstrator might say “this is how you press softly” and proceed to perform the demonstration (as is common, say, in an instructional video). The demonstrated behaviour might also be slow and short but the demonstrator may not have been explicitly thinking about these notions, hence may not have labelled accordingly. The missing label values are incorporated with an indicator function in Eq. 6.
  Condition on User Specification Apart from being used in the optimisation objective, the weak labels are also used in an additional conditioning step at test time. The generative process we have consists of first sampling values for c from its prior. These are then passed together with i through pθ in order to get distributions over x and y, from which we can sample. However, we are really interested in being able to incorporate the provided information about the labels y into the sampling of semantically aligned parts of c. Specifically, we are interested in being able to generate robot behaviours which are consistent with chosen values for y (what we call a user specification). Postoptimising the θ and φ parameters, we choose to approximate the posterior over c for each label l in each concept group j with a set of Gaussian distributions N (µjl,Σjl). We use Maximum Likelihood Estimation for µjl and Σjl over a set of samples from c. The samples are weighted with the corresponding label likelihood pθ(yj = l|πj(cj)). As a result, the process of generating a trajectory x ∼ pθ(x|i, c) can be additionally conditioned on an optional user specification y = {y1, . . . , y|G|}, through c, in a single-label fashion (Eq. 7) or a compositional fashion (Eq. 8). Pseudo-code for the generation of trajectories for both types of conditioning is provided in the Supplementary Materials.
  c ∼ p(c|yj = l) = N (c|µjl,Σjl) (7) ∀j ∈ {1, . . . , |c|} : cj ∼ p(cj |yj = l) = {
  prior, if yj = ∅ or j > |G| N (c|µjl,Σjl)(j), otherwise
  (8)
sections.heading: 4 METHODOLOGY
sections.text: In terms of a concrete model architecture, our model is a Conditional Variational Autoencoder (CVAE) (Sohn et al., 2015) with an encoding network qφ and a decoding network pθ. The parameters of both networks are optimised jointly using methods for amortised variational inference and a stochastic gradient variational bayes estimation (Kingma & Welling, 2013).2
  Encoding & Decoding Networks Due to the diversity of modalities that we want to use, qφ is implemented as a combination of 2D convolutions (for the image input), 1D convolutions (for the trajectory input) and an MLP that brings the output of the previous two modules to the common concept manifold c. For the decoding network, we implement a Temporal Convolutional Network (TCN) (Bai et al., 2018) which operates over the concatenation h of a single concept embedding c and a single image encoding i. However, the TCN takes a sequence of length T and transforms it to another sequence of length T , by passing it through a series of dilated convolutions, while h is equivalent to a sequence of length 1. Thus, we tile the concatenated vector T times and to each
  2The models are implemented in PyTorch (Adam et al., 2017) and optimised using the Adam optimiser (Kingma & Ba, 2014)
  instance of that vector hi, i ∈ {1, . . . , T}, we attach a time dimension ti = iT . This broadcasting technique has previously shown to achieve better disentanglement in the latent space both on visual (Watters et al., 2019) and time-dependent data (Noseworthy et al., 2019). We only set up the mean µθ of the predicted distribution over x to be a non-linear function of [h; t] while σ is fixed, for better numerical stability during training. This allows us to use the L2 distance—Mean Squared Error (MSE)—between the inferred trajectories µθ, acting as a reconstruction, and the ground truth ones.
  Label Predictors For each concept group gj we take values from the corresponding latent axis cj and pass it through a single linear layer, with a softmax activation function, to predict a probability vector πj . Maximising the label likelihood is realised as a Softmax-Cross Entropy (SCE) term in the loss function. Optimising this term gives us a better guarantee that some of the latent axes will be semantically aligned with the notions in the label groups. The fact that some labels might be missing is accounted for with an indicator function which calculates ∇θ,∇φ with respect to the SCE, only if yj is present.
  Weighted Loss The full loss function that we optimise is presented in Eq. 9. It is composed of three main terms—the trajectory MSE, equivalent to the negative trajectory log-likelihood, the label SCE, equivalent to the negative weak label log-likelihood and the KL divergence between the fit amortised posterior over c and its prior. The concrete values of the three coefficients—α, β, γ—are discussed in the Supplementary Materials.
  min θ,φ,w L(x,y, I) = βDKL(qφ(c|x, I)︸ ︷︷ ︸ amortised posterior || pθ(c)︸ ︷︷ ︸ prior )+αEqφ(c|x,I)(log pθ(x|c, I)︸ ︷︷ ︸ MSE )+γ
  |G|∑ i 1{yi 6=∅}H(ciw T i , yi)︸ ︷︷ ︸
  SCE
  (9)
sections.heading: 5 EXPERIMENTS
sections.text: The models we experiment with include a Variational Autoencoder (Kingma & Welling, 2013) with a Gaussian prior (VAE), an Adversarial Autoencoder (Makhzani et al., 2015) with a Uniform prior (AAE) and a VAE with mixed continuous and categorical latent variables (GS). For the last model, the continuous variables have a Gaussian prior and the categorical variables utilise the GumbelSoftmax reparametrisation (Jang et al., 2017). Each of the three models, is trained while utilising the weak labels—VAE-weak, AAE-weak, GS-weak—and with no label information—VAE, AAE, GS. Whether or not labels are used during training is controlled by setting γ = 0 in Eq. 9.
  The setup used in the experiments consists of a PR2 robot, an HTC Vive controller and headset, and a tabletop with a single object on it—a red cube. Multiple pressing motions on the surface of the table are demonstrated by teleoperating the end effector of the robot. The pose of the end effector, with respect to the robot’s torso, is mapped to be the same as the pose of the controller (held by the demonstrator) with respect to the headset (behind the demonstrator in Figure 4).
  Data A total of 100 demonstrations were captured from the PR2, consisting of: • Initial frame of each demonstration:
  128x128-pixel Kinect2 image (RGB); • 7-DoF joint angle positions;
  • 7-DoF joint efforts; • 5 groups of discrete weak labels;
  Each demonstration has a single label attached to it. In total, we have 4 spatial symbols—where in the image, with respect to the red cube, do we dab—and 6 force-related symbols—how do we dab (Table 1). Figure 3 provides a graphical depiction demonstrating the differences between the different force-relate groupings qualitatively.
  All trajectories are standardised to be of fixed length—the length of the longest trajectory (240 timesteps)—by padding them with the last value for each channel/joint. Additionally, both the joint posi-
  tions and efforts are augmented with Gaussian noise or by randomly sliding them leftwards (simulating an earlier start) and padding accordingly. The size of the total dataset after augmentation is 1000 demonstrations which are split according to a 90-10 training-validation split. The size of the latent space is chosen to be |c| = 8. In the context of the problem formulation in Section 2, cs = {c0, c1}, ce = {c2, c3, c4} and cu = {c5, c6, c7}.
  plementary Materials. Average accuracy across all images and all samples is reported for each label. For the labels of a spatial nature, for each 7-dimensional (7 DoFs) robot state xi,x = {x1, . . . , xT }, we compute the corresponding end-effector pose pi through the forward kinematics K of the robot. Using the camera model of the Kinect2 sensor, each pose pi is projected in the image we condition on. We report the Mean Absolute Error (MAE) in normalised pixel coordinates between the dab location and the landmark location (in pixel coordinates) for the misclassified trajectories. This gives us a sense of how far from the desired ones the generated samples are. The dab location is inferred as the point in the end-effector pose trajectory with the lowest z coordinate. For example, if we have conditioned on the left label but some of the sampled trajectories result in a dab to the right of the cube in the image, we report how far from the true classification of left was the robot when it touched the table.
  Conditioning on a Composition of Labels Since the concepts we encode in the latent space of the model should be independent of each other they can be composed together—e.g. a trajectory can be simultaneously hard, long and fast. We examine how well is this conceptual independence preserved in the latent space by generating trajectories conditioned on all possible combinations of labels regarding effort, speed and length of press—8 possible combinations in total. We still use the Gaussian distributions for each label in each concept group but only take the values along the relevant latent axis—Eq. 8. To fully close the loop, the trajectories which we sample from the model could further be executed on the physical robot through a hybrid position/force controller (Reibert, 1981). However, such an evaluation is beyond the scope of the paper.
sections.heading: 6 RESULTS & DISCUSSION
sections.text: Our experiments demonstrate that the models which utilise weak user labels within our proposed setup can more reliably generate behaviours consistent with a given spatial or force-related user specification, in contrast to baseline models which do not benefit from such information. Moreover, we find that continuous latent variables better capture the independent continuous notions which underlie the discrete user labels in comparison to discrete latent variables. Below we present quantitative results from the experiments with further qualitative results in the Supplementary Materials.
  When conditioning on a single force label—Table 3—the best-performing models are VAE-weak and AAE-weak. In the worse-performing cases—VAE, GS—it would appear that the models suffer from partial mode collapse. For instance, in Table 3, penultimate row, the VAE can not represent soft trajectories, most samples are hard. Interestingly AAE and AAE-weak perform similarly well. We attribute that to the fact the Uniform prior used does not have a discrete number of modes to which the optimised posterior might potentially collapse. Simultaneously, regardless of their prior, none of the weak models suffers from mode collapse, since the label information helps for the corresponding notions to be more distinguishable in the latent space. All models have high accuracy, when conditioned on a spatial label—Table 2. However, GS-weak, VAE-weak and AAEweak appear to be closer to the true conditioned-on concept when a bad trajectory is generated, as measured by the lower Mean Absolute Error of the image-projected end-effector positions.
  While single-label conditioning is possible under all models, it is only VAE-weak, AAE-weak and GS-weak which provide a reliable mechanism for composing labels from different independent concept groups. That is possible due to the explicit alignment of each concept and its labels with a single latent axis. Trajectory generation is done after conditioning on compositions of labels from the three independent concept groups designating effort—see Table 1 (right column). Specifically, the difference between the sampled latent values for hard long slow and hard long fast, for example, are only along the axis used to predict slow from fast—c4 in this case. If the generated trajectories match their specification (high measured accuracy) then indeed the perturbed axis is aligned with the notion behind the perturbed labels. Results from this experiment are presented in Figure 5. The better-performing models are the ones with continuous latent variables. While the VAE-weak and AAE-weak perform equally well for most label combinations and do manage to
  encode independent concepts as such (along specific latent axes), the model with the mix of continuous and categorical latent variables (GS-weak) does not match their results. Taking the second row in Table 3 into account, this means that the categorical model (which has more constrained capacity than the VAE-weak and AAE-weak) can capture the data distribution (shows good singlelabel-conditioning reconstructions) but in an entangled fashion. Namely, going from slow to fast corresponds to perturbing more than a single latent axis simultaneously. This further supports our initial hypothesis that while discrete, the weak labels only provide a mechanism for shaping the latent representation of what are otherwise continuous phenomena.
sections.heading: 7 POURING EXPERIMENT
sections.text: An additional pouring task is demonstrated where the manner of execution varies spatially and in terms of how it is executed—see Table 4 and Figure 6. 20 demonstrations are captured for each label. The size of the latent space is kept to be |c| = 8. In the context of the problem formulation in Section 2, cs = {c0}, ce = {c1, c2} and cu = {c3, c4, c5, c6, c7}.
  When compared to the dabbing setup, this task presents a more significant challenge in identifying the switching latent dynamics for pouring in each cup (compared to the single red cube in the previous task, acting as a reference point for resolving spatial prepositions).
  Interpolation experiments for the VAE-weak model are presented in Figure 7 for 4 test cup configurations. The first latent dimension of the model—c0, optimised to predict in which cup to pour—is linearly interpolated in the range [−2, 2]. In all four rows, we can see the end effector projections in the image plane, through the robot kinematics and camera model, showing a gradual shift from initially pouring in the red cup to eventually pouring in the blue cup. The color evolution of each end-effector trajectory from cyan to magenta signifies time progression. Moreover, given the disentangled nature of the VAE-weak model, we also generate robot trajectories conditioned on specific values for the 3 latent axes (c0, c1, c2), consistent with example label compositions—see Figure 8. For picking the right value for each axis we use the post-training-estimated Normal distributions for each label (consult Section 3). Additional qualitative results and sensitivity analysis of the model to the heterogeneous input modalities are presented in Appendix E and the supplementary website 3.
  3https://sites.google.com/view/weak-label-lfd
sections.heading: 8 RELATED WORK
sections.text: Methods that utilise high-parameter neural models to learn disentangled low-dimensional representation achieve that by either tuning the optimised loss function (Higgins et al., 2017; Chen et al., 2018), imposing crafted priors over the latent space (Chen et al., 2016) or using weak supervision (Hristov et al., 2019; Rasmus et al., 2015). While being able to produce interpretable manifolds, most of these approaches focus on modelling visual data and respectively visually-manifested concepts, with minor exceptions—e.g. modelling motion capture data (Ainsworth et al., 2018). In contrast, we solve a problem involving multiple heterogeneous modalities—vision, forces, joint angles. Moreover, in addition to separating factors of variation in the data, to achieve latent space interpretability, we also deal with the problem of entanglement across modalities—i.e. which factors relate to which data modalities.
  Dynamic Movement Primitives (Schaal, 2006) and Probabilistic Movement Primitives (Paraschos et al., 2013) are commonly used to model dynamical systems as ‘point-mass attractors with a nonlinear forcing term’. However, the resulting control parameters can be high-dimensional and unintuitive for end-user manipulation. Performing dimensionality reduction (Colomé et al., 2014) or imposing hierarchical priors (Rueckert et al., 2015) are both ideas seen in the literature as a means of making the high-dimensional parameter space more meaningful to the human demonstrator. These have the advantage of yielding a clearer idea of how changes in the optimised parameters result in deterministic changes in the generated behaviours. However, most of these approaches limit themselves to end-effector trajectories, or – at most – making use of visual input (Kober et al., 2008).
  Noseworthy et al. (2019) explore disentangling task parameters from the manner of execution parameters, in the context of pouring. They utilise an adversarial training regime, which facilitates better separation of the two types of parameters in the latent space. However, it is assumed that the task parameters are known a priori. Interpretation of the learned codes is achieved post-training by perturbing latent axis values and qualitatively inspecting generated trajectories (standard evaluation technique for models which are trained unsupervised). We argue that through the use of weak supervision through discrete labels we have finer control over the ‘meaning’ of latent dimensions.
sections.heading: 9 CONCLUSION
sections.text: We recontextualise the problem of interpretable multi-modal LfD as a problem of formulating a conditional probabilistic model. In the example task of table-top dabbing, we utilise weak discrete labels from the demonstrator to represent abstract notions, manifested in a captured multi-modal robot dataset. Through the use of high-capacity neural models and methods from deep variational inference, we show how to align some of the latent variables of the formulated probabilistic model with high-level notions implicit in the demonstrations.
sections.heading: ACKNOWLEDGMENTS
sections.text: This work is partly supported by funding from the Turing Institute, as part of the Safe AI for surgical assistance project. Additionally, we thank the members of the Edinburgh Robust Autonomy and Decisions Group for their feedback and support.
sections.heading: B LABEL CONDITIONING & EVALUATION HEURISTICS
sections.text: Algorithms 1 and 2 provide pseudo-code for the trajectory generation procedures described in Section 3. The main difference can be summarised as following—when the generative process is conditioned on a single label l, all 8 dimensions of the latent samples are drawn from a single 8-dimensional Gaussian distribution associated with that label and its concept group—line 2 in Algorithm 1. However, when the process is conditioned on a composition of labels from different groups—a user specification—the sampling procedure differs as c is iteratively built. For each latent axis cj , if a label is specified for the concept group aligned with cj , only the values along the j-th axis are taken of a sample from the corresponding 8-dimensional Gaussian. If no label is given, cj is sampled from a Standard Normal distributionN (0, 1) (or the respective prior of choice.
  Same applies for latent dimensions which have not been aligned with any concept groups, since |c| > |G|—lines 2 to 9 in Algorithm 2.
  Algorithm 1: Trajectory generation conditioned on a single label Input: Visual Scene image I Input: Single user label y from concept group g Input: Image encoder, part of pθ Input: Decoder network qφ Input: set of Gaussian distribution for each label in each concept group: K = {{N (µjl,Σjl)}, j ∈ {1, . . . , |G|}, l ∈ {1, . . . , |gj |} Output: Generated trajectory x̂ of time length T
  1 encode image input i← pθ(I); 2 sample latent values c ∼ N (c|µjl,Σjl), such that j = g, l = y; 3 concatenate c and i: h← [c; i]; 4 tile h T times; 5 attach a last time dimension h = [h; t], t = {t1, . . . , tT }, ti = iT ; 6 generate trajectory x̂← qφ(x|h); 7 return x̂;
  Algorithm 2: Trajectory generation conditioned on a composition of labels Input: Visual Scene image I Input: User label specification y = {y1, . . . , y|G|}, potentially one label for each concept group Input: Image encoder, part of pθ Input: Decoder network qφ Input: set of Gaussian distribution for each label in each concept group: K = {N (c|µjl,Σjl)}, j ∈ {1, . . . , |G|}, l ∈ {1, . . . , |gj |} Output: Generated trajectory x̂ of time length T
  1 encode image input i← pθ(I); 2 init c = []; 3 for each latent dimension cj ∈ c do 4 if yj 6= ∅ and j <= |G| then 5 sample w ∼ N (w|µjl,Σjl), such that, l = yj ; 6 only take values along the j-th dimension: append w[j] to c; 7 else 8 sample w ∼ N (w|0, 1); 9 append w to c;
  10 concatenate c and i: h← [c; i]; 11 tile h T times; 12 attach a last time dimension h = [h; t], t = {t1, . . . , tT }, ti = iT ; 13 generate trajectory x̂← qφ(x|h); 14 return x̂;
  The hand-designed heuristics used for reporting accuracy on the the generated trajectories have concrete semantics and values which have been chosen after closely inspecting the captured dataset, the given user labels and the robot configuration during the demonstrations. :
  • spatial labels—left, right, front, behind—the location of the red object in the scene is extracted using standard computer vision techniques based on color segmentation. The position of the cube is inferred as the center of mass of all red pixels detected. From the overall end-effector trajectory, derived through the robot forward kinematics, the pressing location is chosen as the end-effector position with the lowest z coordinate. That position is then projected into the image plane in order to determine where with respect to the detected object position did the robot touch the table.
  • soft vs hard trajectories - the generated trajectories have normalised values in the range [−1, 1]. A soft trajectory is considered as such if specific joint efforts are below a particular threshold. More specifically:
  – shoulder lift joint efforts are below 0.5. – upper arm roll joint effort are above -0.5.
  • short vs long trajectories - the generated trajectories have length of T = 240. A short trajectory is considered as such if efforts in the range of [max effort-0.2, max effort] are maintained for more than 50 time steps. Otherwise the trajectory is deemed long. The heuristic operates over the following joints:
  – shoulder lift joint efforts – upper arm roll joint efforts
  • slow vs fast trajectories - a slow trajectory is considered as such if the time it takes to reach maximal joint effort is more than 80 time steps. Otherwise the trajectory is deemed fast. The heuristic operates over the following joints:
  – shoulder lift joint efforts – upper arm roll joint efforts
  For resolving robot joint names, please consult Figure 9 below:
sections.heading: C ARCHITECTURE DETAILS
sections.text: The model architectures are implemented in the PyTorch framework4 and is visually presented in Figure 10. The image encoder network takes as input a single RGB 128x128 pixel image. The trajectory encoder takes a single 14-dimensional, 240 time-step-long trajectory. Their outputs both feed into an MLP network which, after a series of nonlinear transformations, outputs parameters of a distribution over the latent space c. Through the reparametrisation trick Kingma & Welling (2013) we sample values for c. Through a residual connection, the output of the image encoder is concatenated to the sampled latent values. The resultant vector is tiled 240 times, extended with a time dimension, and fed into a TCN decoder network to produce reconstructions for the original 14-dimensional input trajectories.
  Across all experiments, training is performed for a fixed number of 100 epochs using a batch size of 8. The dimensionality of the latent space |c| = 8 across all experiments. The Adam optimizer (Kingma & Ba, 2014) is used through the learning process with the following values for its parameters—(learningrate = 0.001, β1 = 0.9, β2 = 0.999, eps = 1e− 08, weightdecayrate = 0, amsgrad = False).
  4https://pytorch.org/docs/stable/index.html
sections.heading: FC (256)
sections.heading: FC (256)
sections.text: For all experiments, the values (unless when set to 0) for the three coefficients from Equation 9 are:
  • α = 1, β = 0.1, γ = 10
  The values are chosen empirically in a manner such that all the loss terms have similar magnitude and thus none of them overwhelms the gradient updates while training the full model.
sections.heading: D QUALITATIVE EVALUATION FOR DABBING
sections.text: Figures 11 to 15 present the qualitative results from perturbing in a controlled way the latent space optimised under the VAE-weak model. For figure the first row of plots represents the generated joint angle positions for each of the 7 joints, for each of the 5 drawn samples, the second row—the generated joint efforts—and the third row the corresponding end-effector positions, from the forward robot kinematics, projected in the image plane. As we can see, perturbing c0 and c1 corresponds to clear and consistent movement of the end effector (through the generated joint position profiles) while there is not much change in the generated effort profiles—Figures 11 and 12. Simultaneously, we observe the opposite effect in other 3 Figures. Perturbations in c2 correspond solely changes in the output efforts in the shoulder lift and upper arm rolls joints—Figure 13 (i) and (j). Perturbations in c3 correspond to changes in length for which the joint efforts are exerted but does not change their magnitude or where they are applied on the table—Figure 14. And finally, perturbations in c4 correspond to changes on how quickly the maximum efforts are reached - most noticeable in the changing slope of the shoulder lift and upper arm rolls joints—Figure 15 (i) and (j). As a result, we can conclude that the latent dimensions, which we aligned with the semantically grouped weak labels, have indeed captured the notions and concepts which underlie the labels.
sections.heading: E MODEL SENSITIVITY ANALYSIS TO DIFFERENT DATA MODALITIES
sections.text: As an exploration step towards analysing to what extend the auxiliary loss terms guide the model to pay attention to the different data modalities it is exposed to, we train the VAE and VAE-weak models on the demonstrations for pouring in the red cup and blue cup. The first latent dimension of the VAE-weak model is optimised to represent the conceptual variation of pouring in one of the two cups. Post-training, the values of the latent vectors c for both models are fixed and trajectory samples are drawn as the models are conditioned on four different test scenes, representing different cup configurations. Figure 16 demonstrates that, averaged across the different configurations, the VAE-weak model has greater variations in the sampled trajectories for all 7 joints of the robot arm when compared to the VAE one. This suggests that unless utilised, the input image modality might be partly ignored—the VAE does not have any auxiliary tasks to force it to use the scene image input, which can result in it just auto-encoding the trajectory data through c. In contrast, predicting where the robot pours in the scene at an abstract level is both a function of the joint angle trajectory and the image inputs—utilised through the classification loss of VAE-weak. Thus, for the same specification (fixed c), conditioning on different scenes leads to different joint trajectories.