title: 3D Scene Understanding Through Local Random Access Sequence Modeling
summary: 3D scene understanding from single images is a pivotal problem in computer
  vision with numerous downstream applications in graphics, augmented reality,
  and robotics. While diffusion-based modeling approaches have shown promise,
  they often struggle to maintain object and scene consistency, especially in
  complex real-world scenarios. To address these limitations, we propose an
  autoregressive generative approach called Local Random Access Sequence (LRAS)
  modeling, which uses local patch quantization and randomly ordered sequence
  generation. By utilizing optical flow as an intermediate representation for 3D
  scene editing, our experiments demonstrate that LRAS achieves state-of-the-art
  novel view synthesis and 3D object manipulation capabilities. Furthermore, we
  show that our framework naturally extends to self-supervised depth estimation
  through a simple modification of the sequence design. By achieving strong
  performance on multiple 3D scene understanding tasks, LRAS provides a unified
  and effective framework for building the next generation of 3D vision models.
paper_authors._id: 67f5b6a8cc0c176ad85b8f34
paper_authors.name: Wanhee Lee
paper_authors._id: 67f5b6a8cc0c176ad85b8f35
paper_authors.name: Klemen Kotar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T07:35:13.488Z
paper_authors._id: 646d3cae4220471ca0c6cb13
paper_authors.avatarUrl: /avatars/30e47db03a23d49d946cc5248b28959a.svg
paper_authors.fullname: Klemen Kotar
paper_authors.type: user
paper_authors.user: klemenk
paper_authors._id: 67f5b6a8cc0c176ad85b8f36
paper_authors.name: Rahul Mysore Venkatesh
paper_authors._id: 67f5b6a8cc0c176ad85b8f37
paper_authors.name: Jared Watrous
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:45:53.826Z
paper_authors._id: 67f6cabfbd46ca8c483f7efc
paper_authors.avatarUrl: /avatars/e859f4a9479ba0d54a09799998de690d.svg
paper_authors.fullname: Jared Watrous
paper_authors.type: user
paper_authors.user: TheTrueJard
paper_authors._id: 67f5b6a8cc0c176ad85b8f38
paper_authors.name: Honglin Chen
paper_authors._id: 67f5b6a8cc0c176ad85b8f39
paper_authors.name: Khai Loong Aw
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:45:57.132Z
paper_authors._id: 631ad4b2fac58c9c8167bafc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631ad4b2fac58c9c8167bafc/jum1yrhVWkl-biBdKitF3.png
paper_authors.fullname: Khai Loong Aw
paper_authors.type: user
paper_authors.user: awwkl
paper_authors._id: 67f5b6a8cc0c176ad85b8f3a
paper_authors.name: Daniel L. K. Yamins
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:21.449Z
paper_authors._id: 67f625c5f7a398c2a285d1b8
paper_authors.avatarUrl: /avatars/1e4b708d68caee4a2d9ae6fe6e1c7b91.svg
paper_authors.fullname: Dan Yamins
paper_authors.type: user
paper_authors.user: dyamins
paper_title: 3D Scene Understanding Through Local Random Access Sequence Modeling
paper_summary: 3D scene understanding from single images is a pivotal problem in computer
  vision with numerous downstream applications in graphics, augmented reality,
  and robotics. While diffusion-based modeling approaches have shown promise,
  they often struggle to maintain object and scene consistency, especially in
  complex real-world scenarios. To address these limitations, we propose an
  autoregressive generative approach called Local Random Access Sequence (LRAS)
  modeling, which uses local patch quantization and randomly ordered sequence
  generation. By utilizing optical flow as an intermediate representation for 3D
  scene editing, our experiments demonstrate that LRAS achieves state-of-the-art
  novel view synthesis and 3D object manipulation capabilities. Furthermore, we
  show that our framework naturally extends to self-supervised depth estimation
  through a simple modification of the sequence design. By achieving strong
  performance on multiple 3D scene understanding tasks, LRAS provides a unified
  and effective framework for building the next generation of 3D vision models.
paper_ai_summary: LRAS, an autoregressive generative approach using local patch quantization and random sequence generation, achieves state-of-the-art performance in 3D scene understanding tasks, including novel view synthesis and depth estimation.
paper_ai_keywords: autoregressive generative approach Local Random Access Sequence LRAS local patch quantization randomly ordered sequence generation optical flow 3D scene editing novel view synthesis 3D object manipulation self-supervised depth estimation