title: UniPose: A Unified Multimodal Framework for Human Pose Comprehension,
    Generation and Editing
summary: Human pose plays a crucial role in the digital age. While recent works have
  achieved impressive progress in understanding and generating human poses, they
  often support only a single modality of control signals and operate in
  isolation, limiting their application in real-world scenarios. This paper
  presents UniPose, a framework employing Large Language Models (LLMs) to
  comprehend, generate, and edit human poses across various modalities, including
  images, text, and 3D SMPL poses. Specifically, we apply a pose tokenizer to
  convert 3D poses into discrete pose tokens, enabling seamless integration into
  the LLM within a unified vocabulary. To further enhance the fine-grained pose
  perception capabilities, we facilitate UniPose with a mixture of visual
  encoders, among them a pose-specific visual encoder. Benefiting from a unified
  learning strategy, UniPose effectively transfers knowledge across different
  pose-relevant tasks, adapts to unseen tasks, and exhibits extended
  capabilities. This work serves as the first attempt at building a
  general-purpose framework for pose comprehension, generation, and editing.
  Extensive experiments highlight UniPose's competitive and even superior
  performance across various pose-relevant tasks.
paper_authors._id: 674713a34295e90fc523ab3d
paper_authors.name: Yiheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:19:41.408Z
paper_authors._id: 66afcc9b3cbe4ea9a4b57df6
paper_authors.avatarUrl: /avatars/add1c3cdfbb9c1013820761057a6a643.svg
paper_authors.fullname: Li YiHeng
paper_authors.type: user
paper_authors.user: L-yiheng
paper_authors._id: 674713a34295e90fc523ab3e
paper_authors.name: Ruibing Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T11:37:17.826Z
paper_authors._id: 64f1e1badd92641727444364
paper_authors.avatarUrl: /avatars/a2e9331b456694ff5f955281416e33d9.svg
paper_authors.fullname: ruibing hou
paper_authors.type: user
paper_authors.user: flow2023
paper_authors._id: 674713a34295e90fc523ab3f
paper_authors.name: Hong Chang
paper_authors._id: 674713a34295e90fc523ab40
paper_authors.name: Shiguang Shan
paper_authors._id: 674713a34295e90fc523ab41
paper_authors.name: Xilin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T11:37:38.181Z
paper_authors._id: 670f0d33c9504aac1179cd3c
paper_authors.avatarUrl: /avatars/a593778043e9dca533cfb89e03ce57f9.svg
paper_authors.fullname: xilin chen
paper_authors.type: user
paper_authors.user: Heylon
paper_title: UniPose: A Unified Multimodal Framework for Human Pose Comprehension,
    Generation and Editing
paper_summary: Human pose plays a crucial role in the digital age. While recent works have
  achieved impressive progress in understanding and generating human poses, they
  often support only a single modality of control signals and operate in
  isolation, limiting their application in real-world scenarios. This paper
  presents UniPose, a framework employing Large Language Models (LLMs) to
  comprehend, generate, and edit human poses across various modalities, including
  images, text, and 3D SMPL poses. Specifically, we apply a pose tokenizer to
  convert 3D poses into discrete pose tokens, enabling seamless integration into
  the LLM within a unified vocabulary. To further enhance the fine-grained pose
  perception capabilities, we facilitate UniPose with a mixture of visual
  encoders, among them a pose-specific visual encoder. Benefiting from a unified
  learning strategy, UniPose effectively transfers knowledge across different
  pose-relevant tasks, adapts to unseen tasks, and exhibits extended
  capabilities. This work serves as the first attempt at building a
  general-purpose framework for pose comprehension, generation, and editing.
  Extensive experiments highlight UniPose's competitive and even superior
  performance across various pose-relevant tasks.
paper_ai_keywords: Large Language Models (LLMs) pose tokenizer 3D SMPL poses discrete pose tokens unified vocabulary pose-specific visual encoder unified learning strategy