Imitation learning provides an appealing framework for autonomous control: in many tasks, demonstrations of preferred behavior can be readily obtained from human experts, removing the need for costly and potentially dangerous online data collection in the real world. However, policies learned with imitation learning have limited flexibility to accommodate varied goals at test time. Model-based reinforcement learning (MBRL) offers considerably more flexibility, since a predictive model learned from data can be used to achieve various goals at test time. However, MBRL suffers from two shortcomings. First, the model does not help to choose desired or safe outcomes -its dynamics estimate only what is possible, not what is preferred. Second, MBRL typically requires additional online data collection to ensure that the model is accurate in those situations that are actually encountered when attempting to achieve test time goals. Collecting this data with a partially trained model can be dangerous and time-consuming. In this paper, we aim to combine the benefits of imitation learning and MBRL, and propose imitative models: probabilistic predictive models able to plan expert-like trajectories to achieve arbitrary goals. We find this method substantially outperforms both direct imitation and MBRL in a simulated autonomous driving task, and can be learned efficiently from a fixed set of expert demonstrations without additional online data collection. We also show our model can flexibly incorporate user-supplied costs at test-time, can plan to sequences of goals, and can even perform well with imprecise goals, including goals on the wrong side of the road.
# Summary
This submission proposes a method to combine the benefits of model-based RL and Imitation Learning (IL) for navigation tasks. The key idea is to i) learn a prior over trajectory distributions from a fixed dataset of demonstrations, and ii) use this learned dynamical model for path planning via probabilistic inference. Reaching target waypoints is done by maximizing the trajectory likelihood conditioned on the planning goal. The prior is learned using R2P2 on LIDAR features and past positions. Experiments using the CARLA driving simulator show that this method can outperform standard control, IL, and model-based RL baselines, while flexibly incorporating test-time goals and costs thanks to its probabilistic formulation.
# Strengths
The method is an elegant way to get the best of both worlds in RL and IL, leveraging the recent R2P2 work to estimate a powerful sequential model used for planning via probabilistic inference. The flexibility of the method in considering test-time cost maps and user-defined goals (e.g. to avoid potholes) is appealing, especially since it does not require on-policy data collection.
The proposed planning-as-inference method can in theory handle the multi-modality present in human demonstrations by using a probabilistic model of the observed behaviors as prior over undirected expert trajectories.
The approach seem to outperform both model-based and imitation learning baselines on a simplified version of the CARLA benchmark, including on interesting fine-grained metrics (e.g., comfort based).
# Weaknesses
The main weakness of this submission lies in its experimental evaluation, especially the absence of any dynamic objects in the tested environment ("static world CARLA", section 1). It is unclear how this approach would generalize beyond just staying on the road. How would it handle traffic lights, pedestrians, other drivers, weather variations, and more complex driving tasks than waypoint following by traversing mostly free space? How does the prior generalize to more complex behaviors (e.g, by using more contextual information \phi)? How robust is the method to noise in the demonstrations, i.e. non-expert or suboptimal behavior? It seems that estimating the generative prior on human behavior might suffer from the same issues as behavior cloning, e.g., the sample inefficiency due to the combinatorial explosion of causal factors explaining complex human behaviors. It might be in fact even harder to estimate that generative model than use a direct discriminative approach (e.g., a modular pipeline), at the cost of reduced flexibility at test time of course. The currently reported sample efficiency (7000 training samples) and near perfect success rate seem to suggest that this (non-standard) version of the CARLA benchmark is too simple (no weather variations, no dynamic obstacles). Comparison to the state of the art (beyond the baselines implemented here) on the original CARLA benchmark seems needed (especially in the "Nav. dynamic" task).
The method is only described very succinctly in section 2. I do not believe there are enough details (especially around the learning algorithm, hyper-parameters, and other important technical elements) for reproducibility at this stage. Section 2.1 is also quite dense for people not familiar with the R2P2 paper. As the main contribution of the paper is to leverage that model for planning and control, it would be great to maybe discuss a bit deeper. Finally, the input modalities are not clear, especially for the baselines: the proposed method is using LIDAR and localization whereas the IL baseline seems to use vision (while the others just use the trajectory). This makes the fairness of the comparison really unclear (LIDAR is a much stronger signal for just staying on the road).
Minor remarks:
- Why use a proportional controller as a baseline instead of the standard PID one?
- Section 2.3 seems like it's missing the extension of equation 2 to the multi-goal case?
- Typos in section 3 ("trail-and-error"), section 4 ("autonmous", "knowledge to")
# Recommendation
Although the theoretical benefits of the method are well-motivated and clear (off-policy learning, probabilistic model, flexibility at test time), the experimental evaluation (custom simple CARLA test, unclear comparison to baselines) and lack of details impeding reproducibility seems to suggest that this submission needs a bit more work. First, adding more details as suggested above and clarifying the experimental protocol seem like a must, but can be easily addressed by an update to the text. Second, it would be ideal to evaluate the approach on the standard CARLA benchmark in order to compare fairly to the prior art. This is much more involved.
I personally like the approach, so although I think it is marginally below the acceptance threshold in its current form, I reserve my judgement for the time being and look forward to the authors' reply.
# Update
The submission has been drastically rewritten (the diff is massive) and I think it is in much better shape, answering some of my concerns around reproducibility and generalization. Furthermore, it reinforces the strengths of the approach (esp. around its flexibility).
I am willing to recommend acceptance, but I have some further questions (hence I have only updated my score to a 6 for now). They are mostly related to the comparison with IL (important to validate the claim in the paper that the proposed approach is quantitatively better than both existing IL and RL methods). See discussion below for details.