title: HumanVid: Demystifying Training Data for Camera-controllable Human Image
    Animation
summary: Human image animation involves generating videos from a character photo,
  allowing user control and unlocking potential for video and movie production.
  While recent approaches yield impressive results using high-quality training
  data, the inaccessibility of these datasets hampers fair and transparent
  benchmarking. Moreover, these approaches prioritize 2D human motion and
  overlook the significance of camera motions in videos, leading to limited
  control and unstable video generation.To demystify the training data, we
  present HumanVid, the first large-scale high-quality dataset tailored for human
  image animation, which combines crafted real-world and synthetic data. For the
  real-world data, we compile a vast collection of copyright-free real-world
  videos from the internet. Through a carefully designed rule-based filtering
  strategy, we ensure the inclusion of high-quality videos, resulting in a
  collection of 20K human-centric videos in 1080P resolution. Human and camera
  motion annotation is accomplished using a 2D pose estimator and a SLAM-based
  method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets
  to augment existing available 3D assets. Notably, we introduce a rule-based
  camera trajectory generation method, enabling the synthetic pipeline to
  incorporate diverse and precise camera motion annotation, which can rarely be
  found in real-world data. To verify the effectiveness of HumanVid, we establish
  a baseline model named CamAnimate, short for Camera-controllable Human
  Animation, that considers both human and camera motions as conditions. Through
  extensive experimentation, we demonstrate that such simple baseline training on
  our HumanVid achieves state-of-the-art performance in controlling both human
  pose and camera motions, setting a new benchmark. Code and data will be
  publicly available at https://github.com/zhenzhiwang/HumanVid/.
paper_authors._id: 66a1e2857e9bf88779ea40ee
paper_authors.name: Zhenzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T09:08:07.604Z
paper_authors._id: 6519346a186bc3b6997c1aaf
paper_authors.avatarUrl: /avatars/8981bd278962da50f9bbfb92c2abe2bf.svg
paper_authors.fullname: Zhenzhi Wang
paper_authors.type: user
paper_authors.user: zhenzhiwang
paper_authors._id: 66a1e2857e9bf88779ea40ef
paper_authors.name: Yixuan Li
paper_authors._id: 66a1e2857e9bf88779ea40f0
paper_authors.name: Yanhong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T10:04:39.166Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 66a1e2857e9bf88779ea40f1
paper_authors.name: Youqing Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T10:04:48.288Z
paper_authors._id: 6600f734997ede4f9b1bb33e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6600f734997ede4f9b1bb33e/vvBsz6c1WUzAl9ifGSl_m.jpeg
paper_authors.fullname: Fang
paper_authors.type: user
paper_authors.user: Youqing
paper_authors._id: 66a1e2857e9bf88779ea40f2
paper_authors.name: Yuwei Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:23:58.366Z
paper_authors._id: 6371f83d5ffa922d638ef486
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
paper_authors.fullname: Yuwei Guo
paper_authors.type: user
paper_authors.user: guoyww
paper_authors._id: 66a1e2857e9bf88779ea40f3
paper_authors.name: Wenran Liu
paper_authors._id: 66a1e2857e9bf88779ea40f4
paper_authors.name: Jing Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:40:34.873Z
paper_authors._id: 65367c40061949598892dbdc
paper_authors.avatarUrl: /avatars/4baf27263841471cbd5f629a8b99424d.svg
paper_authors.fullname: Jing Tan
paper_authors.type: user
paper_authors.user: jingtan
paper_authors._id: 66a1e2857e9bf88779ea40f5
paper_authors.name: Kai Chen
paper_authors._id: 66a1e2857e9bf88779ea40f6
paper_authors.name: Tianfan Xue
paper_authors._id: 66a1e2857e9bf88779ea40f7
paper_authors.name: Bo Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T10:21:07.145Z
paper_authors._id: 665e2bd0fbaa279db369b488
paper_authors.avatarUrl: /avatars/8cbc18d17d2813617976dbe709235799.svg
paper_authors.fullname: Bo Dai
paper_authors.type: user
paper_authors.user: doubling
paper_authors._id: 66a1e2857e9bf88779ea40f8
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T10:20:46.159Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: HumanVid: Demystifying Training Data for Camera-controllable Human Image
    Animation
paper_summary: Human image animation involves generating videos from a character photo,
  allowing user control and unlocking potential for video and movie production.
  While recent approaches yield impressive results using high-quality training
  data, the inaccessibility of these datasets hampers fair and transparent
  benchmarking. Moreover, these approaches prioritize 2D human motion and
  overlook the significance of camera motions in videos, leading to limited
  control and unstable video generation.To demystify the training data, we
  present HumanVid, the first large-scale high-quality dataset tailored for human
  image animation, which combines crafted real-world and synthetic data. For the
  real-world data, we compile a vast collection of copyright-free real-world
  videos from the internet. Through a carefully designed rule-based filtering
  strategy, we ensure the inclusion of high-quality videos, resulting in a
  collection of 20K human-centric videos in 1080P resolution. Human and camera
  motion annotation is accomplished using a 2D pose estimator and a SLAM-based
  method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets
  to augment existing available 3D assets. Notably, we introduce a rule-based
  camera trajectory generation method, enabling the synthetic pipeline to
  incorporate diverse and precise camera motion annotation, which can rarely be
  found in real-world data. To verify the effectiveness of HumanVid, we establish
  a baseline model named CamAnimate, short for Camera-controllable Human
  Animation, that considers both human and camera motions as conditions. Through
  extensive experimentation, we demonstrate that such simple baseline training on
  our HumanVid achieves state-of-the-art performance in controlling both human
  pose and camera motions, setting a new benchmark. Code and data will be
  publicly available at https://github.com/zhenzhiwang/HumanVid/.
paper_ai_summary: A new dataset, HumanVid, for human image animation includes high-quality real-world and synthetic video data with human and camera motion annotations, achieving state-of-the-art results in controlling both human pose and camera motions.
paper_ai_keywords: 2D pose estimator SLAM-based method camera trajectory generation human image animation HumanVid CamAnimate