System, methods, and other embodiments described herein relate to improving training of sub-modules for autonomously controlling a vehicle. In one embodiment, a method includes generating projected controls for autonomously controlling the vehicle through a driving scene by analyzing sensor data about the driving scene using an end-to-end (E2E) model. The E2E model is based, at least in part, on at least one of the sub-modules. The method includes training the E2E model according to the projected controls and labels for the sensor data that indicate expected controls for driving the vehicle through the driving scene. The method includes transferring electronic data of the E2E model into the at least one of the sub-modules associated with the E2E model to initialize the at least one of the sub-modules to improve operation of the at least one sub-modules at sub-tasks for autonomously controlling the vehicle.
System and method for leveraging end-to-end driving models for improving driving task modules