Generating Long-term Trajectories Using Deep Hierarchical Networks
We study the problem of modeling spatiotemporal trajectories over long time horizons using expert demonstrations. For instance, in sports, agents often choose action sequences with long-term goals in mind, such as achieving a certain strategic position. Conventional policy learning approaches, such as those based on Markov decision processes, generally fail at learning cohesive long-term behavior in such high-dimensional state spaces, and are only effective when fairly myopic decisionmaking yields the desired behavior. The key difficulty is that conventional models are "single-scale" and only learn a single state-action policy. We instead propose a hierarchical policy class that automatically reasons about both long-term and shortterm goals, which we instantiate as a hierarchical neural network. We showcase our approach in a case study on learning to imitate demonstrated basketball trajectories, and show that it generates significantly more realistic trajectories compared to non-hierarchical baselines as judged by professional sports analysts.
The paper describes a specific instantiation of a deep memory network with an attention mechanism which was designed for hierarchical, long-term, action planing in complicated problems. The network is trained via behavioral cloning of expert trajectories and evaluated on a novel task: predicting basketball player behavior from captured real play-data. The different choices of the model architecture are studied in detail and thoroughly evaluated.
The paper is well written in general (aside from the minor stylistic issues mentioned below) and addresses an important problem: That of learning hierarchical behavioral policies from data. The chosen benchmark is interesting and is a welcome change from the standard benchmarks that are often used for control tasks/imitation learning.  The experiments presented in the paper are thorough and different model choices are studied in separation, allowing the authors to attribute the performance improvement of their model to concrete decisions made in the modeling. This is a big plus. The model formulation seems reasonable and the connection between hierarchical policies and draws from attention mechanisms for language modeling (a very reasonable idea in my mind). Aside from these clearly positive points the paper unfortunately also has a few issues. I will name them and then detail my criticism in turn. Overall, in my eyes, the paper could still well be accepted if the issues with the slightly confusing notation and the formulation of the loss function/training details are addressed (and the bad quality figures should obviously be replaced).   Related work details and citations: - You fail to cite related work from imitation learning/inverse reinforcement learning (IRL)/apprenticeship learning that could be used to solve the presented problem. In fact the presented algorithm can be interpreted as one instance of imitation learning to which the community refers to with the term "behavioral cloning". - In the related work you interchangeably use policy and planner, which at least to someone from the RL community seems somewhat odd. It also seems that calling DQN a "shallow planner" is a bit unfortunate terminology :).   Notation issues/Loss functions used: - In Equation (2) and (3) the auxiliary transfer function is is not conditioned on s_t but from Figure 3 and your network definition further below it seems that, obviously g is predicted from s_t and thus m_g should be conditional on it. Do you omit this simply because you assume it is an independent random event ? - Am I correct in assuming that the actual posterior derived in Equation (3) (and for the computation of which you discretize the state and action space) is never used during training ? It seems each part of the model is trained in a supervised manner and then fine tuning only occurs on the micro action level without computing the integral. Or does the probability density to which you refer in Equation (7) actually correspond to Equation (3) ? This is unclear from the context. - It also seems that suddenly between Equation (3) and Equation (7) the dependency on the history disappears. I presume this is just an error, but it is sloppy notation wise nonetheless. - My MAIN ISSUE with the loss function is captured by the footnote  you have on page 4. Although it is not completely obvious what the loss in Equation (7) is (see comment above) it indeed seems that it is symmetrical in the output of the micro and macro networks. Given that these operate on the same input, and there is no supervision on the macro-goals during fine-tuning, it seems that the objective function then has a trivial local minimum in which one of the networks just outputs a uniform distribution (unless you actually correctly integrate as in Equation (3) in which case I am not 100% sure about this). If this is the case then the only thing preventing the model from converging to this solution is the initialization and the fact that you choose small learning rates (as you write). I am not a big fan of the tactic to rely on the hope that the used, poor, optimizer will not find an obvious flaw in an objective function.  Experimental details: - I assume the correct thing to do during evaluation would be to sample from the posterior defined by Equation (3), but this is never mentioned.   If this is not what you do, what do you do ? Do you simply use the maximum action from the deterministic network output ? - While the task you evaluate on is interesting and fits the presented model well it would have been nice to see a comparison also on one more standard task in which one would expect hierarchical behavior to help, to facilitate comparison to existing work. This is less of a concern if the used data will be made available with the publication of the method.  Regarding the visualizations: - While it indeed seems that your model does not reproduce exact test trajectories it would have been interesting to also depict the closest trajectory in the training data (perhaps aligned with time-warping) to the one produced for the depicted test cases (hinting that the model does not merely learn to remember the training data). - You also could have included more visualizations for the baseline methods as a comparison (at least in the supplementary material).  Regarding the Human preference study: - You write that "During extrapolation, the other nine players do not animate.". This seems odd to me. First of all, I assume that you mean that the players are not animated in the visualization while they are still simulated for the network input (as the network should only be capable of dealing with such examples). Second, why is this a sensible idea ? Would this not heavily restrict the capabilities of the experts to tell apart a real, played out, strategy from a generated one ? I somehow doubt that a play can be extrapolated reliably by the experts just based on a few initial frames. I would rather have expected that you simply highlight the target player.  Regarding the evaluation of the macro-planner: - From the visualizations in Figure 6b it seems that the agent only rarely visits the actual macro goals. This is surprising to me. Could this be a result of the fine-tuning procedure (in which no macro labels are provided) and hint at the fact that the model indeed does not nicely separate macro and micro actions (as I would suspect it will not given the objective function used, see discussion above) ?  Regarding the Benchmark analysis: - The number of different model combinations studied here is nice and it allows for a concrete attribution of performance improvements to different modeling choices. - The evaluation presented in Table 2 has one main problem: You evaluate classification accuracy of the different studied models. However, given that you discretize the two-dimensional continuous action space into a large 17x17 categorical distribution I am not surprised by the generally low accuracies. While I understand that this accuracy is what you optimize during training it is definitely conceivable that a trained model might predict slightly different actions in each step which have almost the same effect as the ground truth actions but result in 0 % classification accuracy. As such, a different (perhaps distance based) measure would have been good to have in this table! - Tying into the question regarding deterministic/stochastic action choices above, it would have also been nice to have an evaluation of stochastic action selection here (which I assume is not what is presented in the table, although this is also not mentioned). - It also would have been better if you had splitted the test-data into several folds and reported mean and variances (or perhaps better medians with quantiles) in the table such that it would be possible to judge the statistical significance of the differences.  Minor issues: - The figures in the table are of terrible quality, making the model graphs hard to read. Please convert such graphics to vector graphics or at least include high resolution versions. I don't quite understand how you did not realize this before submission, did something go wrong in the submission process here ? - In-text citations are consistently in the wrong citation style. Please read the author guidelines and correctly use natbib \citet/\citep citations! - There are a few articles missing throughout the text and some other minor text mistakes. Here are the instances I found:   l.  22 "timescale low-level actions" -> "timescale of"   l.  33 "For instance, in Figure 1" here you are talking about the green "player: but that is not mentioned in the text   l.  68 "previous work which focus on" -> "which focuses on"   l.  96 "policy should simultaneously optimizes behavior" -> "optimize"   l. 176 "we used an simultaneously predicted the next 4" -> "and"   l. 184 "but instead learns the final output from a feature" -> "which instead"   l. 255 "may applicable" -> "may be applicable"