Learning useful and reusable skill, or sub-task primitives, is a long-standing problem in sensorimotor control. This is challenging because it's hard to define what constitutes a useful skill. Instead of direct manual supervision which is tedious and prone to bias, in this work, our goal is to extract reusable skills from a collection of human demonstrations collected directly for several end-tasks. We propose a weakly-supervised approach for trajectory segmentation following the classic work on multiple instance learning. Our approach is end-to-end trainable, works directly from high-dimensional input (e.g., images) and only requires the knowledge of what skill primitives are present at training, without any need of segmentation or ordering of primitives. We evaluate our approach via rigorous experimentation across four environments ranging from simulation to real world robots, procedurally generated to human collected demonstrations and discrete to continuous action space. Finally, we leverage the generated skill segmentation to demonstrate preliminary evidence of zero-shot transfer to new combinations of skills. Result videos at https: //sites.google.com/view/trajectory-segmentation/.
The paper presents a weakly supervised method for segmentation of trajectories into sub-skills inspired by multi-instance learning (MIL) in image classification by Andrews et al. (2002). This is done via training a classifier to label each observation per time-step with the probability of skills corresponding to that observation. These predictions are then accumulated throughout the trajectory to compute the probability of the skill in that trajectory. There is only a trajectory level supervision provided which specifies which skills are present with no specification of the order in which they appear. They empirically show that their model can achieve decent skill level classification scores on multiple environments provided that there is a large variety of demonstrations provided.
In its current form, I would recommend this paper to be rejected because 1) the framing and motivation of the paper does not correspond to the results and experiments reported and hence seems misleading 2) the paper is limited in scope 3) further experiments and comparisons to relevant baselines are needed to support the claims made in the paper.
The problem that they are proposing is interesting and is of clear value, however, the paper falls short in addressing this problem. In particular, the paper is framed as a way to learn re-usable useful sub-skills that can help generalize to new situations in control. However, the method presented provides a per time-step labelling of each observation with the associated most likely sub-skill. Having the per time-step labelling of the trajectory provides no indication that the data could be useful for learning reusable skills for downstream tasks later. One very basic experiment could be to train a behaviour cloning (BC) agent on the observation-action pairs conditioned on the sub-skill (or a separate network per sub-skill) and show that the learned policies can be leveraged in solving the tasks presented. For instance, one can train a meta-controller that can switch between these learned sub-skills to successfully perform the task. Training such sub-skills from weakly supervised skill annotations has been successfully done by Shiarlis et al. (2018). It should be noted that in their setting, the annotations are ordered which simplifies the problem to some extent.
Ignoring the motivation and focusing only on the problem setting addressed which is annotation of trajectories with skill labels, the experiments seem very restrictive to me. To my understanding there are at most 4-6 primitive skills present in each environment investigated. Looking at the videos linked, it feels like there is some overfitting to the trajectories provided as for example in the case of the video with the red bowl (Reach and Stir inside Cup), the classifier is predicting 'Stirring' from the first frames where there is not much information present in the scene and 'reach to object' could also be plausible for example. It would have been nice to see the logits of predictions per classes for these examples to understand the confidence of the model particularly when some of the skills could be equally likely (e.g. first few frames).
I found the experimental setup unclear. Particularly, details regarding the task setup, how many demonstrations are needed per task/skill, architectural choices and hyper-parameters are lacking and makes experiments hard to follow and understand. I would have also liked to see more analysis regarding the segmented trajectories, particularly how consistent these predictions are through time. To my understanding there is nothing that keeps a skill annotation consistent over some period of time (e.g. the model could keep switching its prediction every time-step). This would be quite unsatisfactory if one would like to use this to actually segment sub-trajectories associated with a given skill that could be useful for training policies. They report applying Gaussian smoothing to filter out noisy predictions but there are no details provided on how this is tuned and how much this affects the quality of segmentations.
Overall, the paper seems to me very limited in its scope and experimental results. The claims made throughout the paper are not supported empirically or theoretically. There is not enough evidence for me to assess the significance of the proposed method and know whether this is indeed useful in practice.