title: Fast View Synthesis of Casual Videos
summary: Novel view synthesis from an in-the-wild video is difficult due to challenges
  like scene dynamics and lack of parallax. While existing methods have shown
  promising results with implicit neural radiance fields, they are slow to train
  and render. This paper revisits explicit video representations to synthesize
  high-quality novel views from a monocular video efficiently. We treat static
  and dynamic video content separately. Specifically, we build a global static
  scene model using an extended plane-based scene representation to synthesize
  temporally coherent novel video. Our plane-based scene representation is
  augmented with spherical harmonics and displacement maps to capture
  view-dependent effects and model non-planar complex surface geometry. We opt to
  represent the dynamic content as per-frame point clouds for efficiency. While
  such representations are inconsistency-prone, minor temporal inconsistencies
  are perceptually masked due to motion. We develop a method to quickly estimate
  such a hybrid video representation and render novel views in real time. Our
  experiments show that our method can render high-quality novel views from an
  in-the-wild video with comparable quality to state-of-the-art methods while
  being 100x faster in training and enabling real-time rendering.
paper_authors._id: 656ea110896aeb11689a8008
paper_authors.name: Yao-Chih Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:18:19.822Z
paper_authors._id: 6323f24dc2d8cfcd80998ee2
paper_authors.avatarUrl: /avatars/8888965137382b6f18d806b3e1333ff5.svg
paper_authors.fullname: Yao-Chih Lee
paper_authors.type: user
paper_authors.user: yclee
paper_authors._id: 656ea110896aeb11689a8009
paper_authors.name: Zhoutong Zhang
paper_authors._id: 656ea110896aeb11689a800a
paper_authors.name: Kevin Blackburn-Matzen
paper_authors._id: 656ea110896aeb11689a800b
paper_authors.name: Simon Niklaus
paper_authors._id: 656ea110896aeb11689a800c
paper_authors.name: Jianming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:19:05.762Z
paper_authors._id: 632d52047bad38a82f75daee
paper_authors.avatarUrl: /avatars/56cc9b5dea95408a83cb80ee8c5565bf.svg
paper_authors.fullname: Jianming Zhang
paper_authors.type: user
paper_authors.user: jimmie33
paper_authors._id: 656ea110896aeb11689a800d
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:19:16.142Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_authors._id: 656ea110896aeb11689a800e
paper_authors.name: Feng Liu
paper_title: Fast View Synthesis of Casual Videos
paper_summary: Novel view synthesis from an in-the-wild video is difficult due to challenges
  like scene dynamics and lack of parallax. While existing methods have shown
  promising results with implicit neural radiance fields, they are slow to train
  and render. This paper revisits explicit video representations to synthesize
  high-quality novel views from a monocular video efficiently. We treat static
  and dynamic video content separately. Specifically, we build a global static
  scene model using an extended plane-based scene representation to synthesize
  temporally coherent novel video. Our plane-based scene representation is
  augmented with spherical harmonics and displacement maps to capture
  view-dependent effects and model non-planar complex surface geometry. We opt to
  represent the dynamic content as per-frame point clouds for efficiency. While
  such representations are inconsistency-prone, minor temporal inconsistencies
  are perceptually masked due to motion. We develop a method to quickly estimate
  such a hybrid video representation and render novel views in real time. Our
  experiments show that our method can render high-quality novel views from an
  in-the-wild video with comparable quality to state-of-the-art methods while
  being 100x faster in training and enabling real-time rendering.
paper_ai_keywords: implicit neural radiance fields explicit video representations global static scene model extended plane-based scene representation temporally coherent novel video spherical harmonics displacement maps non-planar complex surface geometry per-frame point clouds temporal inconsistencies perceptually masked hybrid video representation real-time rendering