title: Dynamic Mesh-Aware Radiance Fields
summary: Embedding polygonal mesh assets within photorealistic Neural Radience Fields
  (NeRF) volumes, such that they can be rendered and their dynamics simulated in
  a physically consistent manner with the NeRF, is under-explored from the system
  perspective of integrating NeRF into the traditional graphics pipeline. This
  paper designs a two-way coupling between mesh and NeRF during rendering and
  simulation. We first review the light transport equations for both mesh and
  NeRF, then distill them into an efficient algorithm for updating radiance and
  throughput along a cast ray with an arbitrary number of bounces. To resolve the
  discrepancy between the linear color space that the path tracer assumes and the
  sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
  (HDR) images. We also present a strategy to estimate light sources and cast
  shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
  formulation can be efficiently integrated with a high-performance physics
  simulator that supports cloth, rigid and soft bodies. The full rendering and
  simulation system can be run on a GPU at interactive rates. We show that a
  hybrid system approach outperforms alternatives in visual realism for mesh
  insertion, because it allows realistic light transport from volumetric NeRF
  media onto surfaces, which affects the appearance of reflective/refractive
  surfaces and illumination of diffuse surfaces informed by the dynamic scene.
paper_authors._id: 64fff396f355a888f92fa3cc
paper_authors.name: Yi-Ling Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-12T15:51:43.746Z
paper_authors._id: 65007c1cea5699b59a9be9ba
paper_authors.avatarUrl: /avatars/ce56da4f9263f2f5517a394470303823.svg
paper_authors.fullname: Yiling Qiao
paper_authors.type: user
paper_authors.user: yilingq
paper_authors._id: 64fff396f355a888f92fa3cd
paper_authors.name: Alexander Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-12T12:22:14.059Z
paper_authors._id: 650133c252aec889f4cbd3c1
paper_authors.avatarUrl: /avatars/c807a722b47679b725d49944de6e648c.svg
paper_authors.fullname: Alexander Gao
paper_authors.type: user
paper_authors.user: alexandergao
paper_authors._id: 64fff396f355a888f92fa3ce
paper_authors.name: Yiran Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:22:24.363Z
paper_authors._id: 630cc8203dc31beba6ec000e
paper_authors.avatarUrl: /avatars/c99f819593d76e9c3b05fa9b01d033f0.svg
paper_authors.fullname: Yiran Xu
paper_authors.type: user
paper_authors.user: twizwei
paper_authors._id: 64fff396f355a888f92fa3cf
paper_authors.name: Yue Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T08:48:44.472Z
paper_authors._id: 64625b1c2a83863b97bdeb49
paper_authors.avatarUrl: /avatars/0dc29f85eabc09cb5793c94afd9aac42.svg
paper_authors.fullname: Yue Feng
paper_authors.type: user
paper_authors.user: yuefeng21
paper_authors._id: 64fff396f355a888f92fa3d0
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:22:54.776Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_authors._id: 64fff396f355a888f92fa3d1
paper_authors.name: Ming C. Lin
paper_title: Dynamic Mesh-Aware Radiance Fields
paper_summary: Embedding polygonal mesh assets within photorealistic Neural Radience Fields
  (NeRF) volumes, such that they can be rendered and their dynamics simulated in
  a physically consistent manner with the NeRF, is under-explored from the system
  perspective of integrating NeRF into the traditional graphics pipeline. This
  paper designs a two-way coupling between mesh and NeRF during rendering and
  simulation. We first review the light transport equations for both mesh and
  NeRF, then distill them into an efficient algorithm for updating radiance and
  throughput along a cast ray with an arbitrary number of bounces. To resolve the
  discrepancy between the linear color space that the path tracer assumes and the
  sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
  (HDR) images. We also present a strategy to estimate light sources and cast
  shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
  formulation can be efficiently integrated with a high-performance physics
  simulator that supports cloth, rigid and soft bodies. The full rendering and
  simulation system can be run on a GPU at interactive rates. We show that a
  hybrid system approach outperforms alternatives in visual realism for mesh
  insertion, because it allows realistic light transport from volumetric NeRF
  media onto surfaces, which affects the appearance of reflective/refractive
  surfaces and illumination of diffuse surfaces informed by the dynamic scene.
paper_ai_summary: Rendering and simulating mesh assets within NeRF volumes integrates mesh and NeRF light transport in an efficient, GPU-accelerated system for enhanced visual realism.
paper_ai_keywords: Neural Radience Fields NeRF polygonal mesh light transport equations path tracer radiance throughput High Dynamic Range HDR images light sources shadows hybrid surface-volumetric physics simulator cloth rigid bodies soft bodies interactive rendering