title: CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
summary: We propose CARFF: Conditional Auto-encoded Radiance Field for 3D Scene
  Forecasting, a method for predicting future 3D scenes given past observations,
  such as 2D ego-centric images. Our method maps an image to a distribution over
  plausible 3D latent scene configurations using a probabilistic encoder, and
  predicts the evolution of the hypothesized scenes through time. Our latent
  scene representation conditions a global Neural Radiance Field (NeRF) to
  represent a 3D scene model, which enables explainable predictions and
  straightforward downstream applications. This approach extends beyond previous
  neural rendering work by considering complex scenarios of uncertainty in
  environmental states and dynamics. We employ a two-stage training of
  Pose-Conditional-VAE and NeRF to learn 3D representations. Additionally, we
  auto-regressively predict latent scene representations as a partially
  observable Markov decision process, utilizing a mixture density network. We
  demonstrate the utility of our method in realistic scenarios using the CARLA
  driving simulator, where CARFF can be used to enable efficient trajectory and
  contingency planning in complex multi-agent autonomous driving scenarios
  involving visual occlusions.
paper_authors._id: 65bb17317677ff48f5ca5199
paper_authors.name: Jiezhi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:01:21.948Z
paper_authors._id: 65bc1ad237491e7adc964694
paper_authors.avatarUrl: /avatars/2700fc6ebf7a4b23bc031386b18571e3.svg
paper_authors.fullname: Jiezhi "Stephen" Yang
paper_authors.type: user
paper_authors.user: stephenjyang
paper_authors._id: 65bb17317677ff48f5ca519a
paper_authors.name: Khushi Desai
paper_authors._id: 65bb17317677ff48f5ca519b
paper_authors.name: Charles Packer
paper_authors._id: 65bb17317677ff48f5ca519c
paper_authors.name: Harshil Bhatia
paper_authors._id: 65bb17317677ff48f5ca519d
paper_authors.name: Nicholas Rhinehart
paper_authors._id: 65bb17317677ff48f5ca519e
paper_authors.name: Rowan McAllister
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-01T11:47:10.706Z
paper_authors._id: 633640f18a858c0785bf9ab0
paper_authors.avatarUrl: /avatars/43f6a8090aae1705d7ed3a45b81ce2bc.svg
paper_authors.fullname: Rowan McAllister
paper_authors.type: user
paper_authors.user: RowanMcAllister
paper_authors._id: 65bb17317677ff48f5ca519f
paper_authors.name: Joseph Gonzalez
paper_title: CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
paper_summary: We propose CARFF: Conditional Auto-encoded Radiance Field for 3D Scene
  Forecasting, a method for predicting future 3D scenes given past observations,
  such as 2D ego-centric images. Our method maps an image to a distribution over
  plausible 3D latent scene configurations using a probabilistic encoder, and
  predicts the evolution of the hypothesized scenes through time. Our latent
  scene representation conditions a global Neural Radiance Field (NeRF) to
  represent a 3D scene model, which enables explainable predictions and
  straightforward downstream applications. This approach extends beyond previous
  neural rendering work by considering complex scenarios of uncertainty in
  environmental states and dynamics. We employ a two-stage training of
  Pose-Conditional-VAE and NeRF to learn 3D representations. Additionally, we
  auto-regressively predict latent scene representations as a partially
  observable Markov decision process, utilizing a mixture density network. We
  demonstrate the utility of our method in realistic scenarios using the CARLA
  driving simulator, where CARFF can be used to enable efficient trajectory and
  contingency planning in complex multi-agent autonomous driving scenarios
  involving visual occlusions.
paper_ai_keywords: Conditional Auto-encoded Radiance Field Probabilistic encoder Neural Radiance Field (NeRF) 3D latent scene configurations Pose-Conditional-VAE Mixture density network Partially observable Markov decision process