Dynamic Mesh-Aware Radiance Fields
Embedding polygonal mesh assets within photorealistic Neural Radience Fields
(NeRF) volumes, such that they can be rendered and their dynamics simulated in
a physically consistent manner with the NeRF, is under-explored from the system
perspective of integrating NeRF into the traditional graphics pipeline. This
paper designs a two-way coupling between mesh and NeRF during rendering and
simulation. We first review the light transport equations for both mesh and
NeRF, then distill them into an efficient algorithm for updating radiance and
throughput along a cast ray with an arbitrary number of bounces. To resolve the
discrepancy between the linear color space that the path tracer assumes and the
sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
(HDR) images. We also present a strategy to estimate light sources and cast
shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
formulation can be efficiently integrated with a high-performance physics
simulator that supports cloth, rigid and soft bodies. The full rendering and
simulation system can be run on a GPU at interactive rates. We show that a
hybrid system approach outperforms alternatives in visual realism for mesh
insertion, because it allows realistic light transport from volumetric NeRF
media onto surfaces, which affects the appearance of reflective/refractive
surfaces and illumination of diffuse surfaces informed by the dynamic scene.
64fff396f355a888f92fa3cc
Yi-Ling Qiao
claimed_verified
2023-09-12T15:51:43.746Z
65007c1cea5699b59a9be9ba
/avatars/ce56da4f9263f2f5517a394470303823.svg
Yiling Qiao
user
yilingq
64fff396f355a888f92fa3cd
Alexander Gao
claimed_verified
2023-09-12T12:22:14.059Z
650133c252aec889f4cbd3c1
/avatars/c807a722b47679b725d49944de6e648c.svg
Alexander Gao
user
alexandergao
64fff396f355a888f92fa3ce
Yiran Xu
admin_assigned
2023-09-12T12:22:24.363Z
630cc8203dc31beba6ec000e
/avatars/c99f819593d76e9c3b05fa9b01d033f0.svg
Yiran Xu
user
twizwei
64fff396f355a888f92fa3cf
Yue Feng
claimed_verified
2023-12-18T08:48:44.472Z
64625b1c2a83863b97bdeb49
/avatars/0dc29f85eabc09cb5793c94afd9aac42.svg
Yue Feng
user
yuefeng21
64fff396f355a888f92fa3d0
Jia-Bin Huang
admin_assigned
2023-09-12T12:22:54.776Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
64fff396f355a888f92fa3d1
Ming C. Lin
Dynamic Mesh-Aware Radiance Fields
Embedding polygonal mesh assets within photorealistic Neural Radience Fields
(NeRF) volumes, such that they can be rendered and their dynamics simulated in
a physically consistent manner with the NeRF, is under-explored from the system
perspective of integrating NeRF into the traditional graphics pipeline. This
paper designs a two-way coupling between mesh and NeRF during rendering and
simulation. We first review the light transport equations for both mesh and
NeRF, then distill them into an efficient algorithm for updating radiance and
throughput along a cast ray with an arbitrary number of bounces. To resolve the
discrepancy between the linear color space that the path tracer assumes and the
sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
(HDR) images. We also present a strategy to estimate light sources and cast
shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
formulation can be efficiently integrated with a high-performance physics
simulator that supports cloth, rigid and soft bodies. The full rendering and
simulation system can be run on a GPU at interactive rates. We show that a
hybrid system approach outperforms alternatives in visual realism for mesh
insertion, because it allows realistic light transport from volumetric NeRF
media onto surfaces, which affects the appearance of reflective/refractive
surfaces and illumination of diffuse surfaces informed by the dynamic scene.
Rendering and simulating mesh assets within NeRF volumes integrates mesh and NeRF light transport in an efficient, GPU-accelerated system for enhanced visual realism.
Neural Radience Fields NeRF polygonal mesh light transport equations path tracer radiance throughput High Dynamic Range HDR images light sources shadows hybrid surface-volumetric physics simulator cloth rigid bodies soft bodies interactive rendering