title: MagicClay: Sculpting Meshes With Generative Neural Fields
summary: The recent developments in neural fields have brought phenomenal capabilities
  to the field of shape generation, but they lack crucial properties, such as
  incremental control - a fundamental requirement for artistic work. Triangular
  meshes, on the other hand, are the representation of choice for most geometry
  related tasks, offering efficiency and intuitive control, but do not lend
  themselves to neural optimization. To support downstream tasks, previous art
  typically proposes a two-step approach, where first a shape is generated using
  neural fields, and then a mesh is extracted for further processing. Instead, in
  this paper we introduce a hybrid approach that maintains both a mesh and a
  Signed Distance Field (SDF) representations consistently. Using this
  representation, we introduce MagicClay - an artist friendly tool for sculpting
  regions of a mesh according to textual prompts while keeping other regions
  untouched. Our framework carefully and efficiently balances consistency between
  the representations and regularizations in every step of the shape
  optimization; Relying on the mesh representation, we show how to render the SDF
  at higher resolutions and faster. In addition, we employ recent work in
  differentiable mesh reconstruction to adaptively allocate triangles in the mesh
  where required, as indicated by the SDF. Using an implemented prototype, we
  demonstrate superior generated geometry compared to the state-of-the-art, and
  novel consistent control, allowing sequential prompt-based edits to the same
  mesh for the first time.
paper_authors._id: 65e7faa2ab088a3c36c2b5ec
paper_authors.name: Amir Barda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T10:02:08.589Z
paper_authors._id: 65970b381b663b1e536e76c0
paper_authors.avatarUrl: /avatars/095dfd84599c74d7600228d8104faac9.svg
paper_authors.fullname: Amir Barda
paper_authors.type: user
paper_authors.user: Amir001
paper_authors._id: 65e7faa2ab088a3c36c2b5ed
paper_authors.name: Vladimir G. Kim
paper_authors._id: 65e7faa2ab088a3c36c2b5ee
paper_authors.name: Noam Aigerman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T10:05:05.606Z
paper_authors._id: 637ed06c1dbae09191065d0c
paper_authors.avatarUrl: /avatars/93bb8b51ca27c89b3ddce1be08f641b3.svg
paper_authors.fullname: noam
paper_authors.type: user
paper_authors.user: noamaig
paper_authors._id: 65e7faa2ab088a3c36c2b5ef
paper_authors.name: Amit H. Bermano
paper_authors._id: 65e7faa2ab088a3c36c2b5f0
paper_authors.name: Thibault Groueix
paper_title: MagicClay: Sculpting Meshes With Generative Neural Fields
paper_summary: The recent developments in neural fields have brought phenomenal capabilities
  to the field of shape generation, but they lack crucial properties, such as
  incremental control - a fundamental requirement for artistic work. Triangular
  meshes, on the other hand, are the representation of choice for most geometry
  related tasks, offering efficiency and intuitive control, but do not lend
  themselves to neural optimization. To support downstream tasks, previous art
  typically proposes a two-step approach, where first a shape is generated using
  neural fields, and then a mesh is extracted for further processing. Instead, in
  this paper we introduce a hybrid approach that maintains both a mesh and a
  Signed Distance Field (SDF) representations consistently. Using this
  representation, we introduce MagicClay - an artist friendly tool for sculpting
  regions of a mesh according to textual prompts while keeping other regions
  untouched. Our framework carefully and efficiently balances consistency between
  the representations and regularizations in every step of the shape
  optimization; Relying on the mesh representation, we show how to render the SDF
  at higher resolutions and faster. In addition, we employ recent work in
  differentiable mesh reconstruction to adaptively allocate triangles in the mesh
  where required, as indicated by the SDF. Using an implemented prototype, we
  demonstrate superior generated geometry compared to the state-of-the-art, and
  novel consistent control, allowing sequential prompt-based edits to the same
  mesh for the first time.
paper_ai_keywords: neural fields shape generation incremental control triangular meshes Signed Distance Field (SDF) MagicClay mesh sculpting textual prompts shape optimization differentiable mesh reconstruction