title: HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for
    Autonomous Driving
summary: In the past few decades, autonomous driving algorithms have made significant
  progress in perception, planning, and control. However, evaluating individual
  components does not fully reflect the performance of entire systems,
  highlighting the need for more holistic assessment methods. This motivates the
  development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator
  for evaluating autonomous driving algorithms. We achieve this by lifting
  captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving
  the rendering quality for closed-loop scenarios, and building the closed-loop
  environment. In terms of rendering, We tackle challenges of novel view
  synthesis in closed-loop scenarios, including viewpoint extrapolation and
  360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further
  enables the full closed simulation loop, dynamically updating the ego and actor
  states and observations based on control commands. Moreover, HUGSIM offers a
  comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo,
  nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair
  and realistic evaluation platform for existing autonomous driving algorithms.
  HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks
  the potential for fine-tuning autonomous driving algorithms in a photorealistic
  closed-loop setting.
paper_authors._id: 674e7f8d5a971a957da459b3
paper_authors.name: Hongyu Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:15:10.817Z
paper_authors._id: 664d9d1eafe6e8c3a986ef67
paper_authors.avatarUrl: /avatars/88535076724e97f412ecd362a38904db.svg
paper_authors.fullname: hyzhou
paper_authors.type: user
paper_authors.user: hyzhou404
paper_authors._id: 674e7f8d5a971a957da459b4
paper_authors.name: Longzhong Lin
paper_authors._id: 674e7f8d5a971a957da459b5
paper_authors.name: Jiabao Wang
paper_authors._id: 674e7f8d5a971a957da459b6
paper_authors.name: Yichong Lu
paper_authors._id: 674e7f8d5a971a957da459b7
paper_authors.name: Dongfeng Bai
paper_authors._id: 674e7f8d5a971a957da459b8
paper_authors.name: Bingbing Liu
paper_authors._id: 674e7f8d5a971a957da459b9
paper_authors.name: Yue Wang
paper_authors._id: 674e7f8d5a971a957da459ba
paper_authors.name: Andreas Geiger
paper_authors._id: 674e7f8d5a971a957da459bb
paper_authors.name: Yiyi Liao
paper_title: HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for
    Autonomous Driving
paper_summary: In the past few decades, autonomous driving algorithms have made significant
  progress in perception, planning, and control. However, evaluating individual
  components does not fully reflect the performance of entire systems,
  highlighting the need for more holistic assessment methods. This motivates the
  development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator
  for evaluating autonomous driving algorithms. We achieve this by lifting
  captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving
  the rendering quality for closed-loop scenarios, and building the closed-loop
  environment. In terms of rendering, We tackle challenges of novel view
  synthesis in closed-loop scenarios, including viewpoint extrapolation and
  360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further
  enables the full closed simulation loop, dynamically updating the ego and actor
  states and observations based on control commands. Moreover, HUGSIM offers a
  comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo,
  nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair
  and realistic evaluation platform for existing autonomous driving algorithms.
  HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks
  the potential for fine-tuning autonomous driving algorithms in a photorealistic
  closed-loop setting.
paper_ai_keywords: 3D Gaussian Splatting novel view synthesis closed-loop simulation ego and actor states KITTI-360 Waymo nuScenes PandaSet