Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose "Imitative Models" to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.
Summary:
- key problem: expert-like probabilistic online motion planning to reach arbitrary goals without reward shaping thanks to off-line learning from expert demonstrations;
- contributions: 1) an imitative planning procedure via gradient-based log-likelihood maximization leveraging "imitative models" q(future states | features), 2) multiple proposals to define flexible goals in this probabilistic framework, 3) a complete implementation for end-to-end navigation in CARLA, 4) an extensive experimental evaluation showcasing the performance, flexibility, interpretability, and robustness of the proposed approach w.r.t. the previous state of the art and several Imitation Learning (IL) and Model-Based Reinforcement Learning (MBRL) baselines.
Recommendation: weak accept (leaning towards strong accept)
Key reason 1: principled probabilistic framework bringing the best of both IL and MBRL worlds.
- this planning as inference method is very succinctly and elegantly described in the paper with enough details in appendix (+ code) to suggest a high chance of reproducibility;
- the flexibility of defining different interpretable goals (6 different types explored in the paper) highlights the versatility of the approach;
- the additional benefits in terms of plan reliability estimation (Appendix E) are significant;
- the paper showcases how powerful and useful a good "imitative model" can be, therefore, reinforcing the interest of the research community in the important topic of off-line learning from large datasets of demonstrations (without requiring costly on-line data collection).
Key reason 2: thorough experimental evaluation with convincing results.
- the experimental protocol used is the standard one on CARLA and the results are state of the art;
- the comparison with related works, including recent ones, is thorough and well explained;
- the additional claims regarding robustness are substantiated by multiple experiments.
Suggested improvements:
- Not needing reward engineering is a major claim of this approach, but it seems that constructing goal likelihoods could be seen as a form of reward engineering, no? Table 3 indeed reports significant performance differences (absolute and relative) depending on how the goals are specified, especially in dynamic environments. As the experiments aim at maximizing the same performance metrics, is there a preferred goal type that works well across all experiments? If not, how is "goal definition" different than "reward engineering"?
- Could the authors please include a variance analysis (using different seeds) in Tables 3 and 4? Previous papers have reported high variance in similar settings (cf., Codevilla et al 2019), and this is a common issue in IL/RL.
- How important is knowing \lambda (traffic light state) perfectly in practice? Can the robustness to noise in \lambda be experimentally assessed? I would also clarify in section 4 and Table 2 that other methods do not use \lambda (the traffic light state), which is a signal very strongly correlated with the "ran red light" metric.
- More generally, what is the robustness of this approach to uncertainty / noise in \phi? Although it is typically available (as the authors mentioned) it is never perfect in practice. Can this be handled in a principled probabilistic way as an extension of the current formulation?
- The current model does not factor the influence of the agent on its environment (\phi := \phi_{t=0}). Is this framework limited to open loop planning, or does this open interesting future research directions towards closing the loop? It seems to be a key open problem to at least discuss in Section 5.
Additional Feedback:
- Figure 5 is confusing, not sure it adds much value to the paper;
- typos in Appendix ("pesudocode", "baselines that predicts", "search search").
 ## Update post rebuttal
Thanks to the authors' excellent replies and my initial inclination towards strong accept, I am happy to bump my score to 8. The authors did an excellent job, their rebuttal is on point, not avoiding hard questions, running additional requested experiments (incl. in a clever way for the most computationally expensive ones), and showing clear insights in future steps. Great job!