title: SceneScript: Reconstructing Scenes With An Autoregressive Structured
    Language Model
summary: We introduce SceneScript, a method that directly produces full scene models
  as a sequence of structured language commands using an autoregressive,
  token-based approach. Our proposed scene representation is inspired by recent
  successes in transformers & LLMs, and departs from more traditional methods
  which commonly describe scenes as meshes, voxel grids, point clouds or radiance
  fields. Our method infers the set of structured language commands directly from
  encoded visual data using a scene language encoder-decoder architecture. To
  train SceneScript, we generate and release a large-scale synthetic dataset
  called Aria Synthetic Environments consisting of 100k high-quality in-door
  scenes, with photorealistic and ground-truth annotated renders of egocentric
  scene walkthroughs. Our method gives state-of-the art results in architectural
  layout estimation, and competitive results in 3D object detection. Lastly, we
  explore an advantage for SceneScript, which is the ability to readily adapt to
  new commands via simple additions to the structured language, which we
  illustrate for tasks such as coarse 3D object part reconstruction.
paper_authors._id: 65fb9f77769c433dc27c3f65
paper_authors.name: Armen Avetisyan
paper_authors._id: 65fb9f77769c433dc27c3f66
paper_authors.name: Christopher Xie
paper_authors._id: 65fb9f77769c433dc27c3f67
paper_authors.name: Henry Howard-Jenkins
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:13:04.363Z
paper_authors._id: 65d4991111930466954340ae
paper_authors.avatarUrl: /avatars/ffc07be80323aa4b7dadf8849b70ffde.svg
paper_authors.fullname: Henry Howard-Jenkins
paper_authors.type: user
paper_authors.user: henryhj
paper_authors._id: 65fb9f77769c433dc27c3f68
paper_authors.name: Tsun-Yi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:24:36.204Z
paper_authors._id: 64cbc59b0bf3949c69143b4f
paper_authors.avatarUrl: /avatars/5aa9abe4fda94e65452c67b28060d169.svg
paper_authors.fullname: Tsun-Yi Yang
paper_authors.type: user
paper_authors.user: shamangary
paper_authors._id: 65fb9f77769c433dc27c3f69
paper_authors.name: Samir Aroudj
paper_authors._id: 65fb9f77769c433dc27c3f6a
paper_authors.name: Suvam Patra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:24:51.501Z
paper_authors._id: 64a881b2a59a9777069f4df4
paper_authors.avatarUrl: /avatars/9d79cf64a10e4da3bc708c8f567d8349.svg
paper_authors.fullname: Suvam Patra
paper_authors.type: user
paper_authors.user: Suvam
paper_authors._id: 65fb9f77769c433dc27c3f6b
paper_authors.name: Fuyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:24:59.402Z
paper_authors._id: 63bb22d9df5897db7f04e4fa
paper_authors.avatarUrl: /avatars/1b33600d0e9be0d53fbb7e44a89c4710.svg
paper_authors.fullname: Fuyang Zhang
paper_authors.type: user
paper_authors.user: zfy
paper_authors._id: 65fb9f77769c433dc27c3f6c
paper_authors.name: Duncan Frost
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T10:58:26.831Z
paper_authors._id: 65fc12c6fc9132a2dfe9e8b3
paper_authors.avatarUrl: /avatars/3d04d79721f4ec6b0b090e0bbbf85f70.svg
paper_authors.fullname: Duncan Frost
paper_authors.type: user
paper_authors.user: frosd01
paper_authors._id: 65fb9f77769c433dc27c3f6d
paper_authors.name: Luke Holland
paper_authors._id: 65fb9f77769c433dc27c3f6e
paper_authors.name: Campbell Orme
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T13:24:55.813Z
paper_authors._id: 65fc1b9c85d98567da3138ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fc1b9c85d98567da3138ef/f0F_7Lc70GZ5u_4i2n3H7.jpeg
paper_authors.fullname: Campbell Orme
paper_authors.type: user
paper_authors.user: ormbot
paper_authors._id: 65fb9f77769c433dc27c3f6f
paper_authors.name: Jakob Engel
paper_authors._id: 65fb9f77769c433dc27c3f70
paper_authors.name: Edward Miller
paper_authors._id: 65fb9f77769c433dc27c3f71
paper_authors.name: Richard Newcombe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T09:26:19.855Z
paper_authors._id: 633ea8814b469e3151bb38f2
paper_authors.avatarUrl: /avatars/245f849cde665f1d7dfec15dead6b880.svg
paper_authors.fullname: Richard Newcombe
paper_authors.type: user
paper_authors.user: MONGIE30
paper_authors._id: 65fb9f77769c433dc27c3f72
paper_authors.name: Vasileios Balntas
paper_title: SceneScript: Reconstructing Scenes With An Autoregressive Structured
    Language Model
paper_summary: We introduce SceneScript, a method that directly produces full scene models
  as a sequence of structured language commands using an autoregressive,
  token-based approach. Our proposed scene representation is inspired by recent
  successes in transformers & LLMs, and departs from more traditional methods
  which commonly describe scenes as meshes, voxel grids, point clouds or radiance
  fields. Our method infers the set of structured language commands directly from
  encoded visual data using a scene language encoder-decoder architecture. To
  train SceneScript, we generate and release a large-scale synthetic dataset
  called Aria Synthetic Environments consisting of 100k high-quality in-door
  scenes, with photorealistic and ground-truth annotated renders of egocentric
  scene walkthroughs. Our method gives state-of-the art results in architectural
  layout estimation, and competitive results in 3D object detection. Lastly, we
  explore an advantage for SceneScript, which is the ability to readily adapt to
  new commands via simple additions to the structured language, which we
  illustrate for tasks such as coarse 3D object part reconstruction.
paper_ai_summary: SceneScript generates full scene models using structured language commands via an encoder-decoder architecture, achieving top performance in architectural layout estimation and competitive results in 3D object detection.
paper_ai_keywords: SceneScript autoregressive token-based approach transformers LLMs scene language encoder-decoder architecture Aria Synthetic Environments architectural layout estimation 3D object detection