title: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
summary: This work explores expanding the capabilities of large language models (LLMs)
  pretrained on text to generate 3D meshes within a unified model. This offers
  key advantages of (1) leveraging spatial knowledge already embedded in LLMs,
  derived from textual sources like 3D tutorials, and (2) enabling conversational
  3D generation and mesh understanding. A primary challenge is effectively
  tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly.
  To address this, we introduce LLaMA-Mesh, a novel approach that represents the
  vertex coordinates and face definitions of 3D meshes as plain text, allowing
  direct integration with LLMs without expanding the vocabulary. We construct a
  supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate
  3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs
  as required, and (3) understand and interpret 3D meshes. Our work is the first
  to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge
  for 3D mesh generation in a text-based format, effectively unifying the 3D and
  text modalities. LLaMA-Mesh achieves mesh generation quality on par with models
  trained from scratch while maintaining strong text generation performance.
paper_authors._id: 6736d0f867d01d58999f710b
paper_authors.name: Zhengyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:12:25.832Z
paper_authors._id: 634e15aec1ce28f1de91c470
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634e15aec1ce28f1de91c470/MdtJKgE_fbHydhqESr-DH.png
paper_authors.fullname: Zhengyi Wang
paper_authors.type: user
paper_authors.user: Zhengyi
paper_authors._id: 6736d0f867d01d58999f710c
paper_authors.name: Jonathan Lorraine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T13:49:53.477Z
paper_authors._id: 631b7370bf1351ed2bd0abdc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631b7370bf1351ed2bd0abdc/p0ZRMjgp5mt3sT5OhdHsp.png
paper_authors.fullname: Jonathan Lorraine
paper_authors.type: user
paper_authors.user: lorraine2
paper_authors._id: 6736d0f867d01d58999f710d
paper_authors.name: Yikai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T17:25:42.497Z
paper_authors._id: 64206b35e40f66bcd1e3c985
paper_authors.avatarUrl: /avatars/a8f830d1c47702f642045399e59c9fe8.svg
paper_authors.fullname: Yikai Wang
paper_authors.type: user
paper_authors.user: yikaiw
paper_authors._id: 6736d0f867d01d58999f710e
paper_authors.name: Hang Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T13:54:00.109Z
paper_authors._id: 64212f75d039a891a915560a
paper_authors.avatarUrl: /avatars/0fa50ce554d4db4d64f83be7c11773e9.svg
paper_authors.fullname: Hang Su
paper_authors.type: user
paper_authors.user: hang401
paper_authors._id: 6736d0f867d01d58999f710f
paper_authors.name: Jun Zhu
paper_authors._id: 6736d0f867d01d58999f7110
paper_authors.name: Sanja Fidler
paper_authors._id: 6736d0f867d01d58999f7111
paper_authors.name: Xiaohui Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-16T19:46:59.604Z
paper_authors._id: 63363cb8a8048332fdc268cd
paper_authors.avatarUrl: /avatars/a9bf6ce3394a8b76eade1a7517b37fa1.svg
paper_authors.fullname: xiaohui zeng
paper_authors.type: user
paper_authors.user: xiaohui2022
paper_title: LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
paper_summary: This work explores expanding the capabilities of large language models (LLMs)
  pretrained on text to generate 3D meshes within a unified model. This offers
  key advantages of (1) leveraging spatial knowledge already embedded in LLMs,
  derived from textual sources like 3D tutorials, and (2) enabling conversational
  3D generation and mesh understanding. A primary challenge is effectively
  tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly.
  To address this, we introduce LLaMA-Mesh, a novel approach that represents the
  vertex coordinates and face definitions of 3D meshes as plain text, allowing
  direct integration with LLMs without expanding the vocabulary. We construct a
  supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate
  3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs
  as required, and (3) understand and interpret 3D meshes. Our work is the first
  to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge
  for 3D mesh generation in a text-based format, effectively unifying the 3D and
  text modalities. LLaMA-Mesh achieves mesh generation quality on par with models
  trained from scratch while maintaining strong text generation performance.
paper_ai_summary: The work demonstrates the capability of LLMs to generate 3D meshes from text by introducing a novel approach to tokenize 3D mesh data, allowing the unification of 3D and text modalities without expanding the model's vocabulary.
paper_ai_keywords: large language models LLMs 3D meshes spatial knowledge textual sources vertex coordinates face definitions LLaMA-Mesh supervised fine-tuning SFT mesh generation interleaved outputs mesh understanding text-based format