CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
We propose CARFF: Conditional Auto-encoded Radiance Field for 3D Scene
Forecasting, a method for predicting future 3D scenes given past observations,
such as 2D ego-centric images. Our method maps an image to a distribution over
plausible 3D latent scene configurations using a probabilistic encoder, and
predicts the evolution of the hypothesized scenes through time. Our latent
scene representation conditions a global Neural Radiance Field (NeRF) to
represent a 3D scene model, which enables explainable predictions and
straightforward downstream applications. This approach extends beyond previous
neural rendering work by considering complex scenarios of uncertainty in
environmental states and dynamics. We employ a two-stage training of
Pose-Conditional-VAE and NeRF to learn 3D representations. Additionally, we
auto-regressively predict latent scene representations as a partially
observable Markov decision process, utilizing a mixture density network. We
demonstrate the utility of our method in realistic scenarios using the CARLA
driving simulator, where CARFF can be used to enable efficient trajectory and
contingency planning in complex multi-agent autonomous driving scenarios
involving visual occlusions.
65bb17317677ff48f5ca5199
Jiezhi Yang
claimed_verified
2024-02-02T09:01:21.948Z
65bc1ad237491e7adc964694
/avatars/2700fc6ebf7a4b23bc031386b18571e3.svg
Jiezhi "Stephen" Yang
user
stephenjyang
65bb17317677ff48f5ca519a
Khushi Desai
65bb17317677ff48f5ca519b
Charles Packer
65bb17317677ff48f5ca519c
Harshil Bhatia
65bb17317677ff48f5ca519d
Nicholas Rhinehart
65bb17317677ff48f5ca519e
Rowan McAllister
admin_assigned
2024-02-01T11:47:10.706Z
633640f18a858c0785bf9ab0
/avatars/43f6a8090aae1705d7ed3a45b81ce2bc.svg
Rowan McAllister
user
RowanMcAllister
65bb17317677ff48f5ca519f
Joseph Gonzalez
CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
We propose CARFF: Conditional Auto-encoded Radiance Field for 3D Scene
Forecasting, a method for predicting future 3D scenes given past observations,
such as 2D ego-centric images. Our method maps an image to a distribution over
plausible 3D latent scene configurations using a probabilistic encoder, and
predicts the evolution of the hypothesized scenes through time. Our latent
scene representation conditions a global Neural Radiance Field (NeRF) to
represent a 3D scene model, which enables explainable predictions and
straightforward downstream applications. This approach extends beyond previous
neural rendering work by considering complex scenarios of uncertainty in
environmental states and dynamics. We employ a two-stage training of
Pose-Conditional-VAE and NeRF to learn 3D representations. Additionally, we
auto-regressively predict latent scene representations as a partially
observable Markov decision process, utilizing a mixture density network. We
demonstrate the utility of our method in realistic scenarios using the CARLA
driving simulator, where CARFF can be used to enable efficient trajectory and
contingency planning in complex multi-agent autonomous driving scenarios
involving visual occlusions.
A method for predicting future 3D scenes using a conditional auto-encoded radiance field, mapping images to plausible 3D latent configurations and leveraging probabilistic and auto-regressive predictions for autonomous driving applications.
Conditional Auto-encoded Radiance Field CARFF Neural Radiance Field NeRF Pose-Conditional-VAE mixture density network 3D scene forecasting 3D latent representations partially observable Markov decision process CARLA driving simulator trajectory planning contingency planning