Generating Physically Stable and Buildable LEGO Designs from Text
We introduce LegoGPT, the first approach for generating physically stable
LEGO brick models from text prompts. To achieve this, we construct a
large-scale, physically stable dataset of LEGO designs, along with their
associated captions, and train an autoregressive large language model to
predict the next brick to add via next-token prediction. To improve the
stability of the resulting designs, we employ an efficient validity check and
physics-aware rollback during autoregressive inference, which prunes infeasible
token predictions using physics laws and assembly constraints. Our experiments
show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
designs that align closely with the input text prompts. We also develop a
text-based LEGO texturing method to generate colored and textured designs. We
show that our designs can be assembled manually by humans and automatically by
robotic arms. We also release our new dataset, StableText2Lego, containing over
47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
captions, along with our code and models at the project website:
https://avalovelace1.github.io/LegoGPT/.
681db3a8a9286b53a51dc77b
Ava Pun
admin_assigned
2025-05-09T09:06:33.422Z
672403d5f328a3e6638331ee
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/TXr0SKWI-z-6FvUXTNWXT.jpeg
Ava Pun
user
AvaLovelace
681db3a8a9286b53a51dc77c
Kangle Deng
admin_assigned
2025-05-09T09:06:39.736Z
645d34ecce72244df7b29317
/avatars/1248933d9f89a15e67086325a8322d5e.svg
Kangle Deng
user
kangled
681db3a8a9286b53a51dc77d
Ruixuan Liu
admin_assigned
2025-05-09T09:06:45.442Z
658b307b75ddc76f9dc747ca
/avatars/fc5393dc0bb33a8c0fea3a6f79640386.svg
Ruixuan Liu
user
RLCMU
681db3a8a9286b53a51dc77e
Deva Ramanan
admin_assigned
2025-05-09T09:06:52.637Z
6337151b0267ebcf02640eb6
/avatars/14a723cafc5587043bdfb19304fc202d.svg
Deva Ramanan
user
devakramanan
681db3a8a9286b53a51dc77f
Changliu Liu
681db3a8a9286b53a51dc780
Jun-Yan Zhu
admin_assigned
2025-05-09T09:07:07.095Z
63a0acc32fabbbb899952a2b
https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
Jun-Yan Zhu
user
junyanz
Generating Physically Stable and Buildable LEGO Designs from Text
We introduce LegoGPT, the first approach for generating physically stable
LEGO brick models from text prompts. To achieve this, we construct a
large-scale, physically stable dataset of LEGO designs, along with their
associated captions, and train an autoregressive large language model to
predict the next brick to add via next-token prediction. To improve the
stability of the resulting designs, we employ an efficient validity check and
physics-aware rollback during autoregressive inference, which prunes infeasible
token predictions using physics laws and assembly constraints. Our experiments
show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO
designs that align closely with the input text prompts. We also develop a
text-based LEGO texturing method to generate colored and textured designs. We
show that our designs can be assembled manually by humans and automatically by
robotic arms. We also release our new dataset, StableText2Lego, containing over
47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed
captions, along with our code and models at the project website:
https://avalovelace1.github.io/LegoGPT/.
LegoGPT generates physically stable LEGO models from text prompts using a large language model with physics-aware constraints and supports manual and automated assembly.
autoregressive large language model next-token prediction physics-aware rollback validity check text-based LEGO texturing StableText2Lego dataset