We study the important and challenging problem of controllable generation of long-term sequential behaviors. Solutions to this problem would impact many applications, such as calibrating behaviors of AI agents in games or predicting player trajectories in sports. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are significant challenges that are unique to or exacerbated by generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated temporal behavior faithfully demonstrates diverse styles? In this paper, we leverage large amounts of raw behavioral data to learn policies that can be calibrated to generate a diverse range of behavior styles (e.g., aggressive versus passive play in sports). Inspired by recent work on leveraging programmatic labeling functions, we present a novel framework that combines imitation learning with data programming to learn style-calibratable policies. Our primary technical contribution is a formal notion of style-consistency as a learning objective, and its integration with conventional imitation learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the MuJoCo physics environment, and show that our learned policies can be accurately calibrated to generate interesting behavior styles in both domains.
This paper proposes a method to train style-conditional policies. The method has three components, a dynamics model, a labeling procedure and its approximation, and a policy. The model is trained on real trajectories, while the policy is trained on both real and simulated data coming from the model. The policy is trained to both imitate and be sensitive to the style labelling of states. 
Such a method yields policies that can be executed with styles chosen externally, e.g. by a user.
This paper proposes a novel method that is an interesting take on imitation learning, but it is hard to judge how relevant this method is, as the paper has several inconsistencies and weaknesses that need to be resolved before it is accepted.
Inconsistencies: the reported NLL results do not match the trajectories shown in Figures. Some equations do not seem to reflect what is being optimized.
Weaknesses: Many quantities that should be reported are not: quality of models, diversity of policies, etc. The source of datasets should be clarified. (see detailed comments)
It's interesting that this method can leverage these very sparse or poor quality annotations, but it would be helpful to get a sense of how good the annotations provided here are. What's the threshold were annotation's quality is too low to be helpful? In the general case I suspect this threshold to be higher than what is hinted at in this paper, especially given Table 2. What happens in more realistic settings where e.g. users may specify hundreds of different labels?
Detailed comments:
- how dependent on having a variety of policies is this method? It's not clear how diverse the set of policies coming from the basketball dataset nor the Cheetah policies is, nor how this diversity affects learning. An experiment with explicitly different levels of diversity would strengthen understanding of this method. For Cheetah, it would be easy to report p(y) as a function of the target forward speed, that would give readers a sense of diversity for each label.
- For something like cheetah, with the labels that you propose being a very simple function of the state space, this somewhat resembles DIAYN [1] but with some grounding. Would it make sense to compare to a similar baseline?
- (Table 1) Why report (only) the median over 5 seeds? Papers usually report means. Plus, an indication of variance would be nice.
- It's not clear what the basketball dataset is. Where does it come from? Does it come with a simulator? If not, how do you evaluate style consistency or run step 11-12 of Alg. 2? Do you have a train/valid/test split to choose hyperparameters? (the appendix only suggests a train/test split)
- Why not cite MuJoCo? [2]
- For Figure 2 & 3, I suggest lowering the transparency/alpha value of the trajectories, as there is a lot of overlap.
- Table 1 is somewhat confusing. In (4) and the paragraph thereafter, you define \mathcal{L}^{style} as an error rate, i.e. when \lambda(\tau) \neq y, but in Table 1 you seem to report instead accuracy? (i.e. when \lambda(\tau) = y) If so, then you are reporting percentages, so \times 10^2 rather than 10^-2.
- You never report how well C_\psi is doing, and it's not clear to me why C_\psi is needed at all. When optimizing (8) are you directly treating (8)'s inner terms as negative rewards which is differentiated wrt \pi's parameters? If so, you are doing a form of DDPG, but there is a problem: after (4) you mention that L^style is not differentiable, meaning C_\psi doesn't provide gradient information to \pi. I assume that you acutally optimize (8) with L^label? Whether that's what you're doing or not, it should be clarified. If you are truly using L^style, then it's not clear why C_\psi is needed, as there is no differentiability anyways, and simply using \lambda directly will provide more signal.
- I'm somewhat perplexed by the values of Table 4. A log-likelihood of -190 represents a probability of 10^-83 (a _negative_ LL of -190 is, on the other hand, impossible by virtue of logs and probabilities, but I assume it is a "typo"). What is this the probability of? Entire trajectories? If so it would make more sense to report the _average_ log-likelihood, i.e. per timestep, because at this point in the paper, readers have no sense of how long trajectories are, and thus what these likelihoods represent. (for example, a NLL of 190 could be an average likelihood of 0.5 for 275 steps, or of 0.1 for 83 steps, or of .8 for 850 steps, which are all very different results! According to the appendix Table 5, the basketball trajectories are 25 steps long, which would mean that the imitation objective is not respected at all, exp(-190/25)=.0005, which would mean that pi(a|s) is .0005 on average.)
- Again Table 4, if the "-" is indeed a typo, then CTVAE-style is actually performing better than the baselines, especially for Cheetah (normally one wants negative log-likelihood to be as close to 0 as possible). Same for Table 10, if the NLL column is truly actually log-likelihood, then the "style+" objective really degrades imitation quality rather than improves it.
[1] Diversity is All You Need: Learning Skills without a Reward Function, Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine
[2] MuJoCo: A physics engine for model-based control, Emanuel Todorov, Tom Erez Yuval Tassa