Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models
<|reference_start|>Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models: We propose the use of latent space generative world models to address the covariate shift problem in autonomous driving. A world model is a neural network capable of predicting an agent's next state given past states and actions. By leveraging a world model during training, the driving policy effectively mitigates covariate shift without requiring an excessive amount of training data. During end-to-end training, our policy learns how to recover from errors by aligning with states observed in human demonstrations, so that at runtime it can recover from perturbations outside the training distribution. Additionally, we introduce a novel transformer-based perception encoder that employs multi-view cross-attention and a learned scene query. We present qualitative and quantitative results, demonstrating significant improvements upon prior state of the art in closed-loop testing in the CARLA simulator, as well as showing the ability to handle perturbations in both CARLA and NVIDIA's DRIVE Sim.<|reference_end|>
@article{popov2024mitigating,
    title={Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles
  Using Latent Space Generative World Models},
    author={Alexander Popov, Alperen Degirmenci, David Wehr, Shashank Hegde, Ryan
  Oldja, Alexey Kamenev, Bertrand Douillard, David Nist'er, Urs Muller, Ruchi
  Bhargava, Stan Birchfield, Nikolai Smolyanskiy},
    journal={arXiv preprint arXiv:2409.16663},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.16663},
    primaryClass={cs.RO cs.CV cs.LG cs.SY eess.SY}
}