VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams
Neural Radiance Fields (NeRFs) excel in photorealistically rendering static
scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous
devices remains challenging, due to data storage and computational constraints.
In this paper, we introduce VideoRF, the first approach to enable real-time
streaming and rendering of dynamic radiance fields on mobile platforms. At the
core is a serialized 2D feature image stream representing the 4D radiance field
all in one. We introduce a tailored training scheme directly applied to this 2D
domain to impose the temporal and spatial redundancy of the feature image
stream. By leveraging the redundancy, we show that the feature image stream can
be efficiently compressed by 2D video codecs, which allows us to exploit video
hardware accelerators to achieve real-time decoding. On the other hand, based
on the feature image stream, we propose a novel rendering pipeline for VideoRF,
which has specialized space mappings to query radiance properties efficiently.
Paired with a deferred shading model, VideoRF has the capability of real-time
rendering on mobile devices thanks to its efficiency. We have developed a
real-time interactive player that enables online streaming and rendering of
dynamic scenes, offering a seamless and immersive free-viewpoint experience
across a range of devices, from desktops to mobile phones.
656e9da57e3e612b355b0995
Liao Wang
admin_assigned
2023-12-05T09:32:07.385Z
64180cb3db24526c7c9b011e
/avatars/bcbfe1e18a2c6402d5db14520279b04b.svg
wliao
user
liaowang11
656e9da57e3e612b355b0996
Kaixin Yao
656e9da57e3e612b355b0997
Chengcheng Guo
admin_assigned
2023-12-05T09:32:31.703Z
64e4a9e6b78bc92221b2909e
/avatars/b57b879d43fafa457cbad77dc4d387b1.svg
fy
user
chengchengguo123
656e9da57e3e612b355b0998
Zhirui Zhang
656e9da57e3e612b355b0999
Qiang Hu
656e9da57e3e612b355b099a
Jingyi Yu
admin_assigned
2023-12-05T09:33:14.258Z
63ffcec6e7767a89533b8f80
/avatars/b9e8673943741c2e7b80b15a061bb951.svg
JINGYI Yu
user
Daluuu
656e9da57e3e612b355b099b
Lan Xu
656e9da57e3e612b355b099c
Minye Wu
VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams
Neural Radiance Fields (NeRFs) excel in photorealistically rendering static
scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous
devices remains challenging, due to data storage and computational constraints.
In this paper, we introduce VideoRF, the first approach to enable real-time
streaming and rendering of dynamic radiance fields on mobile platforms. At the
core is a serialized 2D feature image stream representing the 4D radiance field
all in one. We introduce a tailored training scheme directly applied to this 2D
domain to impose the temporal and spatial redundancy of the feature image
stream. By leveraging the redundancy, we show that the feature image stream can
be efficiently compressed by 2D video codecs, which allows us to exploit video
hardware accelerators to achieve real-time decoding. On the other hand, based
on the feature image stream, we propose a novel rendering pipeline for VideoRF,
which has specialized space mappings to query radiance properties efficiently.
Paired with a deferred shading model, VideoRF has the capability of real-time
rendering on mobile devices thanks to its efficiency. We have developed a
real-time interactive player that enables online streaming and rendering of
dynamic scenes, offering a seamless and immersive free-viewpoint experience
across a range of devices, from desktops to mobile phones.
VideoRF enables real-time rendering of dynamic scenes on mobile devices using serialized 2D feature images and efficient video codecs.
Neural Radiance Fields VideoRF 4D radiance field serialized 2D feature images temporal redundancy spatial redundancy 2D video codecs deferred shading model real-time rendering real-time interactive player free-viewpoint experience