title: Generating Physically Stable and Buildable LEGO Designs from Text
summary: We introduce LegoGPT, the first approach for generating physically stable
  LEGO brick models from text prompts. To achieve this, we construct a
  large-scale, physically stable dataset of LEGO designs, along with their
  associated captions, and train an autoregressive large language model to
  predict the next brick to add via next-token prediction. To improve the
  stability of the resulting designs, we employ an efficient validity check and
  physics-aware rollback during autoregressive inference, which prunes infeasible
  token predictions using physics laws and assembly constraints. Our experiments
  show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
  designs that align closely with the input text prompts. We also develop a
  text-based LEGO texturing method to generate colored and textured designs. We
  show that our designs can be assembled manually by humans and automatically by
  robotic arms. We also release our new dataset, StableText2Lego, containing over
  47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
  captions, along with our code and models at the project website:
  https://avalovelace1.github.io/LegoGPT/.
paper_authors._id: 681db3a8a9286b53a51dc77b
paper_authors.name: Ava Pun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:06:33.422Z
paper_authors._id: 672403d5f328a3e6638331ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TXr0SKWI-z-6FvUXTNWXT.jpeg
paper_authors.fullname: Ava Pun
paper_authors.type: user
paper_authors.user: AvaLovelace
paper_authors._id: 681db3a8a9286b53a51dc77c
paper_authors.name: Kangle Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:06:39.736Z
paper_authors._id: 645d34ecce72244df7b29317
paper_authors.avatarUrl: /avatars/1248933d9f89a15e67086325a8322d5e.svg
paper_authors.fullname: Kangle Deng
paper_authors.type: user
paper_authors.user: kangled
paper_authors._id: 681db3a8a9286b53a51dc77d
paper_authors.name: Ruixuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:06:45.442Z
paper_authors._id: 658b307b75ddc76f9dc747ca
paper_authors.avatarUrl: /avatars/fc5393dc0bb33a8c0fea3a6f79640386.svg
paper_authors.fullname: Ruixuan Liu
paper_authors.type: user
paper_authors.user: RLCMU
paper_authors._id: 681db3a8a9286b53a51dc77e
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:06:52.637Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_authors._id: 681db3a8a9286b53a51dc77f
paper_authors.name: Changliu Liu
paper_authors._id: 681db3a8a9286b53a51dc780
paper_authors.name: Jun-Yan Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:07:07.095Z
paper_authors._id: 63a0acc32fabbbb899952a2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
paper_authors.fullname: Jun-Yan Zhu
paper_authors.type: user
paper_authors.user: junyanz
paper_title: Generating Physically Stable and Buildable LEGO Designs from Text
paper_summary: We introduce LegoGPT, the first approach for generating physically stable
  LEGO brick models from text prompts. To achieve this, we construct a
  large-scale, physically stable dataset of LEGO designs, along with their
  associated captions, and train an autoregressive large language model to
  predict the next brick to add via next-token prediction. To improve the
  stability of the resulting designs, we employ an efficient validity check and
  physics-aware rollback during autoregressive inference, which prunes infeasible
  token predictions using physics laws and assembly constraints. Our experiments
  show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
  designs that align closely with the input text prompts. We also develop a
  text-based LEGO texturing method to generate colored and textured designs. We
  show that our designs can be assembled manually by humans and automatically by
  robotic arms. We also release our new dataset, StableText2Lego, containing over
  47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
  captions, along with our code and models at the project website:
  https://avalovelace1.github.io/LegoGPT/.
paper_ai_summary: LegoGPT generates physically stable LEGO models from text prompts using a large language model with physics-aware constraints and supports manual and automated assembly.
paper_ai_keywords: autoregressive large language model next-token prediction physics-aware rollback validity check text-based LEGO texturing StableText2Lego dataset