title: NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and
    Benchmarking
summary: Benchmarking vision-based driving policies is challenging. On one hand,
  open-loop evaluation with real data is easy, but these results do not reflect
  closed-loop performance. On the other, closed-loop evaluation is possible in
  simulation, but is hard to scale due to its significant computational demands.
  Further, the simulators available today exhibit a large domain gap to real
  data. This has resulted in an inability to draw clear conclusions from the
  rapidly growing body of research on end-to-end autonomous driving. In this
  paper, we present NAVSIM, a middle ground between these evaluation paradigms,
  where we use large datasets in combination with a non-reactive simulator to
  enable large-scale real-world benchmarking. Specifically, we gather
  simulation-based metrics, such as progress and time to collision, by unrolling
  bird's eye view abstractions of the test scenes for a short simulation horizon.
  Our simulation is non-reactive, i.e., the evaluated policy and environment do
  not influence each other. As we demonstrate empirically, this decoupling allows
  open-loop metric computation while being better aligned with closed-loop
  evaluations than traditional displacement errors. NAVSIM enabled a new
  competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
  in several new insights. On a large set of challenging scenarios, we observe
  that simple methods with moderate compute requirements such as TransFuser can
  match recent large-scale end-to-end driving architectures such as UniAD. Our
  modular framework can potentially be extended with new datasets, data curation
  strategies, and metrics, and will be continually maintained to host future
  challenges. Our code is available at
  https://github.com/autonomousvision/navsim.
paper_authors._id: 66790d52ec54ee1558b6ecf2
paper_authors.name: Daniel Dauner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:03:11.000Z
paper_authors._id: 633f377dc11d723b1809092c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633f377dc11d723b1809092c/Vvq6Zslw5DDYQmNTwJWfV.png
paper_authors.fullname: Daniel Dauner
paper_authors.type: user
paper_authors.user: DanielDauner
paper_authors._id: 66790d52ec54ee1558b6ecf3
paper_authors.name: Marcel Hallgarten
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:56:19.360Z
paper_authors._id: 65290083147d9119a64171ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65290083147d9119a64171ab/k8q6tKMOZr-sIDyUjdtyp.png
paper_authors.fullname: Marcel
paper_authors.type: user
paper_authors.user: mh0797
paper_authors._id: 66790d52ec54ee1558b6ecf4
paper_authors.name: Tianyu Li
paper_authors._id: 66790d52ec54ee1558b6ecf5
paper_authors.name: Xinshuo Weng
paper_authors._id: 66790d52ec54ee1558b6ecf6
paper_authors.name: Zhiyu Huang
paper_authors._id: 66790d52ec54ee1558b6ecf7
paper_authors.name: Zetong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:04:09.245Z
paper_authors._id: 64312d291e22a07ceb87f29e
paper_authors.avatarUrl: /avatars/a0caa037d30ed52a177d94a622544727.svg
paper_authors.fullname: Zetong Yang
paper_authors.type: user
paper_authors.user: tomztyang
paper_authors._id: 66790d52ec54ee1558b6ecf8
paper_authors.name: Hongyang Li
paper_authors._id: 66790d52ec54ee1558b6ecf9
paper_authors.name: Igor Gilitschenski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:05:11.039Z
paper_authors._id: 64893f0b9ab28735f1a1ec0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
paper_authors.fullname: Igor Gilitschenski
paper_authors.type: user
paper_authors.user: igilitschenski
paper_authors._id: 66790d52ec54ee1558b6ecfa
paper_authors.name: Boris Ivanovic
paper_authors._id: 66790d52ec54ee1558b6ecfb
paper_authors.name: Marco Pavone
paper_authors._id: 66790d52ec54ee1558b6ecfc
paper_authors.name: Andreas Geiger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-24T14:05:33.132Z
paper_authors._id: 620cae049086f3c07f01e3d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644998139538-noauth.jpeg
paper_authors.fullname: Andreas Geiger
paper_authors.type: user
paper_authors.user: andreas-geiger
paper_authors._id: 66790d52ec54ee1558b6ecfd
paper_authors.name: Kashyap Chitta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:27:17.429Z
paper_authors._id: 6489d7aada6bb0138f52027b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6489d7aada6bb0138f52027b/py4YEgVs8Y-QWzCmpa6Xx.jpeg
paper_authors.fullname: Kashyap Chitta
paper_authors.type: user
paper_authors.user: kashyap7x
paper_title: NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and
    Benchmarking
paper_summary: Benchmarking vision-based driving policies is challenging. On one hand,
  open-loop evaluation with real data is easy, but these results do not reflect
  closed-loop performance. On the other, closed-loop evaluation is possible in
  simulation, but is hard to scale due to its significant computational demands.
  Further, the simulators available today exhibit a large domain gap to real
  data. This has resulted in an inability to draw clear conclusions from the
  rapidly growing body of research on end-to-end autonomous driving. In this
  paper, we present NAVSIM, a middle ground between these evaluation paradigms,
  where we use large datasets in combination with a non-reactive simulator to
  enable large-scale real-world benchmarking. Specifically, we gather
  simulation-based metrics, such as progress and time to collision, by unrolling
  bird's eye view abstractions of the test scenes for a short simulation horizon.
  Our simulation is non-reactive, i.e., the evaluated policy and environment do
  not influence each other. As we demonstrate empirically, this decoupling allows
  open-loop metric computation while being better aligned with closed-loop
  evaluations than traditional displacement errors. NAVSIM enabled a new
  competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
  in several new insights. On a large set of challenging scenarios, we observe
  that simple methods with moderate compute requirements such as TransFuser can
  match recent large-scale end-to-end driving architectures such as UniAD. Our
  modular framework can potentially be extended with new datasets, data curation
  strategies, and metrics, and will be continually maintained to host future
  challenges. Our code is available at
  https://github.com/autonomousvision/navsim.
paper_ai_summary: NAVSIM combines large datasets with a non-reactive simulator to benchmark vision-based driving policies, enabling scalable and insightful evaluation that bridges open-loop and closed-loop paradigms.
paper_ai_keywords: open-loop evaluation closed-loop evaluation non-reactive simulator 鸟's eye view simulation horizon TransFuser UniAD benchmarking evaluation framework dataset curation metrics