SceneScript: Reconstructing Scenes With An Autoregressive Structured
  Language Model
We introduce SceneScript, a method that directly produces full scene models
as a sequence of structured language commands using an autoregressive,
token-based approach. Our proposed scene representation is inspired by recent
successes in transformers & LLMs, and departs from more traditional methods
which commonly describe scenes as meshes, voxel grids, point clouds or radiance
fields. Our method infers the set of structured language commands directly from
encoded visual data using a scene language encoder-decoder architecture. To
train SceneScript, we generate and release a large-scale synthetic dataset
called Aria Synthetic Environments consisting of 100k high-quality in-door
scenes, with photorealistic and ground-truth annotated renders of egocentric
scene walkthroughs. Our method gives state-of-the art results in architectural
layout estimation, and competitive results in 3D object detection. Lastly, we
explore an advantage for SceneScript, which is the ability to readily adapt to
new commands via simple additions to the structured language, which we
illustrate for tasks such as coarse 3D object part reconstruction.
65fb9f77769c433dc27c3f65
Armen Avetisyan
65fb9f77769c433dc27c3f66
Christopher Xie
65fb9f77769c433dc27c3f67
Henry Howard-Jenkins
admin_assigned
2024-03-21T09:13:04.363Z
65d4991111930466954340ae
/avatars/ffc07be80323aa4b7dadf8849b70ffde.svg
Henry Howard-Jenkins
user
henryhj
65fb9f77769c433dc27c3f68
Tsun-Yi Yang
admin_assigned
2024-03-21T09:24:36.204Z
64cbc59b0bf3949c69143b4f
/avatars/5aa9abe4fda94e65452c67b28060d169.svg
Tsun-Yi Yang
user
shamangary
65fb9f77769c433dc27c3f69
Samir Aroudj
65fb9f77769c433dc27c3f6a
Suvam Patra
admin_assigned
2024-03-21T09:24:51.501Z
64a881b2a59a9777069f4df4
/avatars/9d79cf64a10e4da3bc708c8f567d8349.svg
Suvam Patra
user
Suvam
65fb9f77769c433dc27c3f6b
Fuyang Zhang
admin_assigned
2024-03-21T09:24:59.402Z
63bb22d9df5897db7f04e4fa
/avatars/1b33600d0e9be0d53fbb7e44a89c4710.svg
Fuyang Zhang
user
zfy
65fb9f77769c433dc27c3f6c
Duncan Frost
claimed_verified
2024-03-21T10:58:26.831Z
65fc12c6fc9132a2dfe9e8b3
/avatars/3d04d79721f4ec6b0b090e0bbbf85f70.svg
Duncan Frost
user
frosd01
65fb9f77769c433dc27c3f6d
Luke Holland
65fb9f77769c433dc27c3f6e
Campbell Orme
claimed_verified
2024-03-22T13:24:55.813Z
65fc1b9c85d98567da3138ef
https://cdn-avatars.huggingface.co/v1/production/uploads/65fc1b9c85d98567da3138ef/f0F_7Lc70GZ5u_4i2n3H7.jpeg
Campbell Orme
user
ormbot
65fb9f77769c433dc27c3f6f
Jakob Engel
65fb9f77769c433dc27c3f70
Edward Miller
65fb9f77769c433dc27c3f71
Richard Newcombe
admin_assigned
2024-03-21T09:26:19.855Z
633ea8814b469e3151bb38f2
/avatars/245f849cde665f1d7dfec15dead6b880.svg
Richard Newcombe
user
MONGIE30
65fb9f77769c433dc27c3f72
Vasileios Balntas
SceneScript: Reconstructing Scenes With An Autoregressive Structured
  Language Model
We introduce SceneScript, a method that directly produces full scene models
as a sequence of structured language commands using an autoregressive,
token-based approach. Our proposed scene representation is inspired by recent
successes in transformers & LLMs, and departs from more traditional methods
which commonly describe scenes as meshes, voxel grids, point clouds or radiance
fields. Our method infers the set of structured language commands directly from
encoded visual data using a scene language encoder-decoder architecture. To
train SceneScript, we generate and release a large-scale synthetic dataset
called Aria Synthetic Environments consisting of 100k high-quality in-door
scenes, with photorealistic and ground-truth annotated renders of egocentric
scene walkthroughs. Our method gives state-of-the art results in architectural
layout estimation, and competitive results in 3D object detection. Lastly, we
explore an advantage for SceneScript, which is the ability to readily adapt to
new commands via simple additions to the structured language, which we
illustrate for tasks such as coarse 3D object part reconstruction.
SceneScript generates full scene models using structured language commands via an encoder-decoder architecture, achieving top performance in architectural layout estimation and competitive results in 3D object detection.
SceneScript autoregressive token-based approach transformers LLMs scene language encoder-decoder architecture Aria Synthetic Environments architectural layout estimation 3D object detection