title: GALA: Generating Animatable Layered Assets from a Single Scan
summary: We present GALA, a framework that takes as input a single-layer clothed 3D
  human mesh and decomposes it into complete multi-layered 3D assets. The outputs
  can then be combined with other assets to create novel clothed human avatars
  with any pose. Existing reconstruction approaches often treat clothed humans as
  a single-layer of geometry and overlook the inherent compositionality of humans
  with hairstyles, clothing, and accessories, thereby limiting the utility of the
  meshes for downstream applications. Decomposing a single-layer mesh into
  separate layers is a challenging task because it requires the synthesis of
  plausible geometry and texture for the severely occluded regions. Moreover,
  even with successful decomposition, meshes are not normalized in terms of poses
  and body shapes, failing coherent composition with novel identities and poses.
  To address these challenges, we propose to leverage the general knowledge of a
  pretrained 2D diffusion model as geometry and appearance prior for humans and
  other assets. We first separate the input mesh using the 3D surface
  segmentation extracted from multi-view 2D segmentations. Then we synthesize the
  missing geometry of different layers in both posed and canonical spaces using a
  novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
  inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
  texture to obtain the complete appearance including the initially occluded
  regions. Through a series of decomposition steps, we obtain multiple layers of
  3D assets in a shared canonical space normalized in terms of poses and human
  shapes, hence supporting effortless composition to novel identities and
  reanimation with novel poses. Our experiments demonstrate the effectiveness of
  our approach for decomposition, canonicalization, and composition tasks
  compared to existing solutions.
paper_authors._id: 65b08e9fa3085d018b4ba217
paper_authors.name: Taeksoo Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:19:04.821Z
paper_authors._id: 64b1673635c2e9909c266407
paper_authors.avatarUrl: /avatars/2a0fdca5983dcaf1f2b09eac56cf586b.svg
paper_authors.fullname: Taeksoo Kim
paper_authors.type: user
paper_authors.user: Taeksoo
paper_authors._id: 65b08e9fa3085d018b4ba218
paper_authors.name: Byungjun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T10:27:26.817Z
paper_authors._id: 644f822623d7eb05ca691d95
paper_authors.avatarUrl: /avatars/b9d3f39155ee96a79b8c873e4bab57e6.svg
paper_authors.fullname: Byungjun Kim
paper_authors.type: user
paper_authors.user: byungjun-kim
paper_authors._id: 65b08e9fa3085d018b4ba219
paper_authors.name: Shunsuke Saito
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-24T10:19:37.621Z
paper_authors._id: 630688edd37ce67e0e50e548
paper_authors.avatarUrl: /avatars/1681e465d7649f67f94e1b69d236cb1e.svg
paper_authors.fullname: Shunsuke Saito
paper_authors.type: user
paper_authors.user: psyth
paper_authors._id: 65b08e9fa3085d018b4ba21a
paper_authors.name: Hanbyul Joo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T12:18:44.367Z
paper_authors._id: 65b0ebe1165e69e0e6dc6ddd
paper_authors.avatarUrl: /avatars/d2e95b2040c761200df1293db2949da7.svg
paper_authors.fullname: Hanbyul Joo
paper_authors.type: user
paper_authors.user: jhugestar
paper_title: GALA: Generating Animatable Layered Assets from a Single Scan
paper_summary: We present GALA, a framework that takes as input a single-layer clothed 3D
  human mesh and decomposes it into complete multi-layered 3D assets. The outputs
  can then be combined with other assets to create novel clothed human avatars
  with any pose. Existing reconstruction approaches often treat clothed humans as
  a single-layer of geometry and overlook the inherent compositionality of humans
  with hairstyles, clothing, and accessories, thereby limiting the utility of the
  meshes for downstream applications. Decomposing a single-layer mesh into
  separate layers is a challenging task because it requires the synthesis of
  plausible geometry and texture for the severely occluded regions. Moreover,
  even with successful decomposition, meshes are not normalized in terms of poses
  and body shapes, failing coherent composition with novel identities and poses.
  To address these challenges, we propose to leverage the general knowledge of a
  pretrained 2D diffusion model as geometry and appearance prior for humans and
  other assets. We first separate the input mesh using the 3D surface
  segmentation extracted from multi-view 2D segmentations. Then we synthesize the
  missing geometry of different layers in both posed and canonical spaces using a
  novel pose-guided Score Distillation Sampling (SDS) loss. Once we complete
  inpainting high-fidelity 3D geometry, we also apply the same SDS loss to its
  texture to obtain the complete appearance including the initially occluded
  regions. Through a series of decomposition steps, we obtain multiple layers of
  3D assets in a shared canonical space normalized in terms of poses and human
  shapes, hence supporting effortless composition to novel identities and
  reanimation with novel poses. Our experiments demonstrate the effectiveness of
  our approach for decomposition, canonicalization, and composition tasks
  compared to existing solutions.
paper_ai_summary: GALA uses a pretrained 2D diffusion model and pose-guided Score Distillation Sampling to decompose single-layer 3D human meshes into multi-layered assets for use in novel avatars.
paper_ai_keywords: GALA 2D diffusion model 3D surface segmentation multi-view 2D segmentations poses body shapes Score Distillation Sampling SDS loss high-fidelity 3D geometry canonical space decomposition canonicalization composition tasks