title: MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation
summary: Human image animation has gained increasing attention and developed rapidly
  due to its broad applications in digital humans. However, existing methods rely
  largely on 2D-rendered pose images for motion guidance, which limits
  generalization and discards essential 3D information for open-world animation.
  To tackle this problem, we propose MTVCrafter (Motion Tokenization Video
  Crafter), the first framework that directly models raw 3D motion sequences
  (i.e., 4D motion) for human image animation. Specifically, we introduce 4DMoT
  (4D motion tokenizer) to quantize 3D motion sequences into 4D motion tokens.
  Compared to 2D-rendered pose images, 4D motion tokens offer more robust
  spatio-temporal cues and avoid strict pixel-level alignment between pose image
  and character, enabling more flexible and disentangled control. Then, we
  introduce MV-DiT (Motion-aware Video DiT). By designing unique motion attention
  with 4D positional encodings, MV-DiT can effectively leverage motion tokens as
  4D compact yet expressive context for human image animation in the complex 3D
  world. Hence, it marks a significant step forward in this field and opens a new
  direction for pose-guided human video generation. Experiments show that our
  MTVCrafter achieves state-of-the-art results with an FID-VID of 6.98,
  surpassing the second-best by 65%. Powered by robust motion tokens, MTVCrafter
  also generalizes well to diverse open-world characters (single/multiple,
  full/half-body) across various styles and scenarios. Our video demos and code
  are on: https://github.com/DINGYANB/MTVCrafter.
paper_authors._id: 682bfefa73f0db9ddd6c73f7
paper_authors.name: Yanbo Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:19.748Z
paper_authors._id: 65c09224a9c1b20e69a61569
paper_authors.avatarUrl: /avatars/78c73be711f2c7a889acb088507ca0aa.svg
paper_authors.fullname: YANBO DING
paper_authors.type: user
paper_authors.user: yanboding
paper_authors._id: 682bfefa73f0db9ddd6c73f8
paper_authors.name: Xirui Hu
paper_authors._id: 682bfefa73f0db9ddd6c73f9
paper_authors.name: Zhizhi Guo
paper_authors._id: 682bfefa73f0db9ddd6c73fa
paper_authors.name: Yali Wang
paper_title: MTVCrafter: 4D Motion Tokenization for Open-World Human Image Animation
paper_summary: Human image animation has gained increasing attention and developed rapidly
  due to its broad applications in digital humans. However, existing methods rely
  largely on 2D-rendered pose images for motion guidance, which limits
  generalization and discards essential 3D information for open-world animation.
  To tackle this problem, we propose MTVCrafter (Motion Tokenization Video
  Crafter), the first framework that directly models raw 3D motion sequences
  (i.e., 4D motion) for human image animation. Specifically, we introduce 4DMoT
  (4D motion tokenizer) to quantize 3D motion sequences into 4D motion tokens.
  Compared to 2D-rendered pose images, 4D motion tokens offer more robust
  spatio-temporal cues and avoid strict pixel-level alignment between pose image
  and character, enabling more flexible and disentangled control. Then, we
  introduce MV-DiT (Motion-aware Video DiT). By designing unique motion attention
  with 4D positional encodings, MV-DiT can effectively leverage motion tokens as
  4D compact yet expressive context for human image animation in the complex 3D
  world. Hence, it marks a significant step forward in this field and opens a new
  direction for pose-guided human video generation. Experiments show that our
  MTVCrafter achieves state-of-the-art results with an FID-VID of 6.98,
  surpassing the second-best by 65%. Powered by robust motion tokens, MTVCrafter
  also generalizes well to diverse open-world characters (single/multiple,
  full/half-body) across various styles and scenarios. Our video demos and code
  are on: https://github.com/DINGYANB/MTVCrafter.
paper_ai_summary: MTVCrafter, a framework using 4D motion tokens and motion-aware Multi-view DiT, significantly improves human image animation by modeling raw 3D motion sequences.
paper_ai_keywords: 4D motion tokenizer 4D motion tokens MV-DiT motion attention 4D positional encodings FID-VID open-world animation human video generation