Current end-to-end deep learning driving models have two problems: (1) Poor generalization ability of unobserved driving environment when diversity of training driving dataset is limited (2) Lack of accident explanation ability when driving models don't work as expected. To tackle these two problems, rooted on the believe that knowledge of associated easy task is benificial for addressing difficult task, we proposed a new driving model which is composed of perception module for see and think and driving module for behave, and trained it with multi-task perception-related basic knowledge and driving knowledge stepwisely. Specifically segmentation map and depth map (pixel level understanding of images) were considered as what & where and how far knowledge for tackling easier drivingrelated perception problems before generating final control commands for difficult driving task. The results of experiments demonstrated the effectiveness of multitask perception knowledge for better generalization and accident explanation ability. With our method the average sucess rate of finishing most difficult navigation tasks in untrained city of CoRL test surpassed current benchmark method for 15 percent in trained weather and 20 percent in untrained weathers.
# Summary
This submission proposes a multi-task convolutional neural network architecture for end-to-end driving (going from an RGB image to controls) evaluated using the CARLA open source simulator. The architecture consists of an encoder and three decoders on top: two for perception (depth prediction and semantic segmentation), and one for driving controls prediction. The network is trained in a two-step supervised fashion: first training the encoder and perception decoders (using depth and semantic segmentation ground truth), second freezing the encoder and training the driving module (imitation learning on demonstrations). The network is evaluated on the standard CARLA benchmark showing better generalization performance in new driving conditions (town and weather) compared to the CARLA baselines (modular pipeline, imitation learning, RL). Qualitative results also show that failure modes are easier to interpret by looking at predicted depth maps and semantic segmentation results.
# Strengths
Simplicity of the approach: the overall architecture described above is simple (cf. Figure 1), combining the benefits of the modular and end-to-end approaches into a feed-forward CNN. The aforementioned two-stage learning algorithm is also explained clearly. Predicted depth maps and semantic segmentation results are indeed more interpretable than attention maps (as traditionally used in end-to-end driving).
Evaluation of the driving policy: the evaluation is done with actual navigation tasks using the CARLA (CoRL'18) benchmark, instead of just off-line behavior cloning accuracy (often used in end-to-end driving papers, easier to overfit to, not guaranteed to transfer to actual driving).
Simple ablative analysis: Table 2 quantifies the generalization performance benefits of pretraining and freezing the encoder on perception tasks (esp. going from 16% to 62% of completed episodes in the new town and weather dynamic navigation scenario).
# Weaknesses
## Writing
I have to start with the most obvious one. The paper is littered with typos and grammatical errors (way too many to list). For instance, the usage of "the" and "a" is almost non-existent. Overall, the paper is really hard to read and needs a thorough pass of proof-reading and editing. Also, please remove the acknowledgments section: I think it is borderline breaking the double-blind submission policy (I don't know these persons, but if I did that would be a breach of ICLR submission policy). Furthermore, I think its contents are not very professional for a submission at a top international academic venue, but that is just my opinion. 
## Novelty
This is the main weakness for me. The architecture is very close to at least the following works:
- Xu, H., Gao, Y., Yu, F. and Darrell, T., End-to-end learning of driving models from large-scale video datasets (CVPR'17): this reference is missing from the paper, whereas it is very closely related, as it also shows the benefit of a segmentation decoder on top of a shared encoder for end-to-end driving (calling it privileged training);
- Codevilla et al's Conditional Imitation Learning (ICRA'18): the only novelty in the current submission w.r.t. CIL is the addition of the depth and segmentation decoders;
- Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V., Driving Policy Transfer via Modularity and Abstraction (CoRL'18): the architecture also uses a shared perception module and segmentation (although in a mediated way instead of auxiliary task) to show better generalization performance (including from sim to real).
Additional missing related works include:
- Kim, J. and Canny, J.F., Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention (ICCV'17): uses post-hoc attention interpretation of "black box" end-to-end networks;
- Sauer, A., Savinov, N. and Geiger, A., Conditional Affordance Learning for Driving in Urban Environments (CoRL'18): also uses a perception module in the middle of the CIL network showing better generalization performance in CARLA (although a bit lower than the results in the current submission).
- Pomerleau, D.A., Alvinn: An autonomous land vehicle in a neural network (NIPS'89): the landmark paper for end-to-end driving with neural networks!
## Insights / significance
In light of the aforementioned prior art, I believe the claims are correct but already reported in other publications in the community (cf. references above). In particular, the proposed approach uses a lot more strongly labeled data (depth and semantic segmentation supervision in a dataset of 40,000 images) than the competing approaches mentioned above. For instance, the modular pipeline in the original CARLA paper uses only 2,500 labeled images, and I am sure its performance would be vastly improved with 40,000 images, but this is not evaluated, hence the comparison in Table 1 being unfair in my opinion. This matters because the encoder in the proposed method is frozen after training on the perception tasks, and the main point of the experiments is to convince that it results in a great (fixed) intermediate representation, which is in line with the aforementioned works doing mediated perception for driving.
The fine-tuning experiments are also confirming what is know in the litterature, namely that simple fine-tuning can lead to catastrophic forgetting (Table 3).
Finally, the qualitative evaluation of failure cases (5.3) leads to a trivial conclusion: a modular approach is indeed more interpretable than an end-to-end one. This is actually by design and the main advocated benefit of modular approaches: failure in the downstream perception module yields failure in the upstream driving module that builds on top of it. As the perception module is, by design, outputting a human interpretable representation (e.g., a semantic segmentation map), then this leads to better interpretation overall.
## Reproducibility
There are not enough details in section 3.1 about the deep net architecture to enable re-implementation ("structure similar to SegNet", no detailed description of the number of layers, non-linearities, number of channels, etc).
Will the authors release the perception training dataset collected in CARLA described in Section 4.2?
# Recommendation
Although the results of the proposed multi-task network on the CARLA driving benchmark are good, it is probably due to using almost two orders of magnitude more labeled data for semantic segmentation and depth prediction than prior works (which is only practical because the experiments are done in simulation). Prior work has confirmed that combining perception tasks like semantic segmentation with end-to-end driving networks yield better performance, including using a strongly related approach (Xu et al). In addition to the lack of novelty or new insights, the writing needs serious attention.
For these reasons, I believe this paper is not suitable for publication at ICLR.