3D Scene Understanding Through Local Random Access Sequence Modeling
3D scene understanding from single images is a pivotal problem in computer
vision with numerous downstream applications in graphics, augmented reality,
and robotics. While diffusion-based modeling approaches have shown promise,
they often struggle to maintain object and scene consistency, especially in
complex real-world scenarios. To address these limitations, we propose an
autoregressive generative approach called Local Random Access Sequence (LRAS)
modeling, which uses local patch quantization and randomly ordered sequence
generation. By utilizing optical flow as an intermediate representation for 3D
scene editing, our experiments demonstrate that LRAS achieves state-of-the-art
novel view synthesis and 3D object manipulation capabilities. Furthermore, we
show that our framework naturally extends to self-supervised depth estimation
through a simple modification of the sequence design. By achieving strong
performance on multiple 3D scene understanding tasks, LRAS provides a unified
and effective framework for building the next generation of 3D vision models.
67f5b6a8cc0c176ad85b8f34
Wanhee Lee
67f5b6a8cc0c176ad85b8f35
Klemen Kotar
claimed_verified
2025-04-09T07:35:13.488Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
67f5b6a8cc0c176ad85b8f36
Rahul Mysore Venkatesh
67f5b6a8cc0c176ad85b8f37
Jared Watrous
claimed_verified
2025-04-10T06:45:53.826Z
67f6cabfbd46ca8c483f7efc
/avatars/e859f4a9479ba0d54a09799998de690d.svg
Jared Watrous
user
TheTrueJard
67f5b6a8cc0c176ad85b8f38
Honglin Chen
67f5b6a8cc0c176ad85b8f39
Khai Loong Aw
claimed_verified
2025-04-10T06:45:57.132Z
631ad4b2fac58c9c8167bafc
https://cdn-avatars.huggingface.co/v1/production/uploads/631ad4b2fac58c9c8167bafc/jum1yrhVWkl-biBdKitF3.png
Khai Loong Aw
user
awwkl
67f5b6a8cc0c176ad85b8f3a
Daniel L. K. Yamins
claimed_verified
2025-04-09T09:48:21.449Z
67f625c5f7a398c2a285d1b8
/avatars/1e4b708d68caee4a2d9ae6fe6e1c7b91.svg
Dan Yamins
user
dyamins
3D Scene Understanding Through Local Random Access Sequence Modeling
3D scene understanding from single images is a pivotal problem in computer
vision with numerous downstream applications in graphics, augmented reality,
and robotics. While diffusion-based modeling approaches have shown promise,
they often struggle to maintain object and scene consistency, especially in
complex real-world scenarios. To address these limitations, we propose an
autoregressive generative approach called Local Random Access Sequence (LRAS)
modeling, which uses local patch quantization and randomly ordered sequence
generation. By utilizing optical flow as an intermediate representation for 3D
scene editing, our experiments demonstrate that LRAS achieves state-of-the-art
novel view synthesis and 3D object manipulation capabilities. Furthermore, we
show that our framework naturally extends to self-supervised depth estimation
through a simple modification of the sequence design. By achieving strong
performance on multiple 3D scene understanding tasks, LRAS provides a unified
and effective framework for building the next generation of 3D vision models.
LRAS, an autoregressive generative approach using local patch quantization and random sequence generation, achieves state-of-the-art performance in 3D scene understanding tasks, including novel view synthesis and depth estimation.
autoregressive generative approach Local Random Access Sequence LRAS local patch quantization randomly ordered sequence generation optical flow 3D scene editing novel view synthesis 3D object manipulation self-supervised depth estimation