title: Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
    Environment from a Single Video
summary: Creating high-quality and interactive virtual environments, such as games and
  simulators, often involves complex and costly manual modeling processes. In
  this paper, we present Video2Game, a novel approach that automatically converts
  videos of real-world scenes into realistic and interactive game environments.
  At the heart of our system are three core components:(i) a neural radiance
  fields (NeRF) module that effectively captures the geometry and visual
  appearance of the scene; (ii) a mesh module that distills the knowledge from
  NeRF for faster rendering; and (iii) a physics module that models the
  interactions and physical dynamics among the objects. By following the
  carefully designed pipeline, one can construct an interactable and actionable
  digital replica of the real world. We benchmark our system on both indoor and
  large-scale outdoor scenes. We show that we can not only produce
  highly-realistic renderings in real-time, but also build interactive games on
  top.
paper_authors._id: 661df62686e7bb7bede31a9a
paper_authors.name: Hongchi Xia
paper_authors._id: 661df62686e7bb7bede31a9b
paper_authors.name: Zhi-Hao Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:40:59.523Z
paper_authors._id: 632a340c5f2ff1958c014191
paper_authors.avatarUrl: /avatars/d51ccb2e2b3e9756dea3d06b6e62f438.svg
paper_authors.fullname: Chih-Hao Lin
paper_authors.type: user
paper_authors.user: chih-hao-lin
paper_authors._id: 661df62686e7bb7bede31a9c
paper_authors.name: Wei-Chiu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:41:06.244Z
paper_authors._id: 624fb0c8841b0a7f1b583e72
paper_authors.avatarUrl: /avatars/a695a528817f57a98c946056d6f9b2f9.svg
paper_authors.fullname: Wei-Chiu Ma
paper_authors.type: user
paper_authors.user: weichium
paper_authors._id: 661df62686e7bb7bede31a9d
paper_authors.name: Shenlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T09:41:13.167Z
paper_authors._id: 62fbe0aaa80632fbd47b86f5
paper_authors.avatarUrl: /avatars/8175e3ac325745714f17dbe000ea955f.svg
paper_authors.fullname: Shenlong Wang
paper_authors.type: user
paper_authors.user: shenlong
paper_title: Video2Game: Real-time, Interactive, Realistic and Browser-Compatible
    Environment from a Single Video
paper_summary: Creating high-quality and interactive virtual environments, such as games and
  simulators, often involves complex and costly manual modeling processes. In
  this paper, we present Video2Game, a novel approach that automatically converts
  videos of real-world scenes into realistic and interactive game environments.
  At the heart of our system are three core components:(i) a neural radiance
  fields (NeRF) module that effectively captures the geometry and visual
  appearance of the scene; (ii) a mesh module that distills the knowledge from
  NeRF for faster rendering; and (iii) a physics module that models the
  interactions and physical dynamics among the objects. By following the
  carefully designed pipeline, one can construct an interactable and actionable
  digital replica of the real world. We benchmark our system on both indoor and
  large-scale outdoor scenes. We show that we can not only produce
  highly-realistic renderings in real-time, but also build interactive games on
  top.
paper_ai_summary: Video2Game converts real-world scenes into realistic interactive environments by combining neural radiance fields, mesh distillation, and physics modeling.
paper_ai_keywords: neural radiance fields NeRF mesh distillation physics modeling interactive environments