Fast View Synthesis of Casual Videos
Novel view synthesis from an in-the-wild video is difficult due to challenges
like scene dynamics and lack of parallax. While existing methods have shown
promising results with implicit neural radiance fields, they are slow to train
and render. This paper revisits explicit video representations to synthesize
high-quality novel views from a monocular video efficiently. We treat static
and dynamic video content separately. Specifically, we build a global static
scene model using an extended plane-based scene representation to synthesize
temporally coherent novel video. Our plane-based scene representation is
augmented with spherical harmonics and displacement maps to capture
view-dependent effects and model non-planar complex surface geometry. We opt to
represent the dynamic content as per-frame point clouds for efficiency. While
such representations are inconsistency-prone, minor temporal inconsistencies
are perceptually masked due to motion. We develop a method to quickly estimate
such a hybrid video representation and render novel views in real time. Our
experiments show that our method can render high-quality novel views from an
in-the-wild video with comparable quality to state-of-the-art methods while
being 100x faster in training and enabling real-time rendering.
656ea110896aeb11689a8008
Yao-Chih Lee
admin_assigned
2023-12-05T09:18:19.822Z
6323f24dc2d8cfcd80998ee2
/avatars/8888965137382b6f18d806b3e1333ff5.svg
Yao-Chih Lee
user
yclee
656ea110896aeb11689a8009
Zhoutong Zhang
656ea110896aeb11689a800a
Kevin Blackburn-Matzen
656ea110896aeb11689a800b
Simon Niklaus
656ea110896aeb11689a800c
Jianming Zhang
admin_assigned
2023-12-05T09:19:05.762Z
632d52047bad38a82f75daee
/avatars/56cc9b5dea95408a83cb80ee8c5565bf.svg
Jianming Zhang
user
jimmie33
656ea110896aeb11689a800d
Jia-Bin Huang
admin_assigned
2023-12-05T09:19:16.142Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
656ea110896aeb11689a800e
Feng Liu
Fast View Synthesis of Casual Videos
Novel view synthesis from an in-the-wild video is difficult due to challenges
like scene dynamics and lack of parallax. While existing methods have shown
promising results with implicit neural radiance fields, they are slow to train
and render. This paper revisits explicit video representations to synthesize
high-quality novel views from a monocular video efficiently. We treat static
and dynamic video content separately. Specifically, we build a global static
scene model using an extended plane-based scene representation to synthesize
temporally coherent novel video. Our plane-based scene representation is
augmented with spherical harmonics and displacement maps to capture
view-dependent effects and model non-planar complex surface geometry. We opt to
represent the dynamic content as per-frame point clouds for efficiency. While
such representations are inconsistency-prone, minor temporal inconsistencies
are perceptually masked due to motion. We develop a method to quickly estimate
such a hybrid video representation and render novel views in real time. Our
experiments show that our method can render high-quality novel views from an
in-the-wild video with comparable quality to state-of-the-art methods while
being 100x faster in training and enabling real-time rendering.
A novel method for efficient real-time novel view synthesis from monocular in-the-wild videos using a hybrid plane-based and point cloud representation achieves high quality with greatly improved training and rendering speed.
implicit neural radiance fields explicit video representations novel view synthesis plane-based scene representation spherical harmonics displacement maps point clouds real-time rendering