HumanVid: Demystifying Training Data for Camera-controllable Human Image
  Animation
Human image animation involves generating videos from a character photo,
allowing user control and unlocking potential for video and movie production.
While recent approaches yield impressive results using high-quality training
data, the inaccessibility of these datasets hampers fair and transparent
benchmarking. Moreover, these approaches prioritize 2D human motion and
overlook the significance of camera motions in videos, leading to limited
control and unstable video generation.To demystify the training data, we
present HumanVid, the first large-scale high-quality dataset tailored for human
image animation, which combines crafted real-world and synthetic data. For the
real-world data, we compile a vast collection of copyright-free real-world
videos from the internet. Through a carefully designed rule-based filtering
strategy, we ensure the inclusion of high-quality videos, resulting in a
collection of 20K human-centric videos in 1080P resolution. Human and camera
motion annotation is accomplished using a 2D pose estimator and a SLAM-based
method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets
to augment existing available 3D assets. Notably, we introduce a rule-based
camera trajectory generation method, enabling the synthetic pipeline to
incorporate diverse and precise camera motion annotation, which can rarely be
found in real-world data. To verify the effectiveness of HumanVid, we establish
a baseline model named CamAnimate, short for Camera-controllable Human
Animation, that considers both human and camera motions as conditions. Through
extensive experimentation, we demonstrate that such simple baseline training on
our HumanVid achieves state-of-the-art performance in controlling both human
pose and camera motions, setting a new benchmark. Code and data will be
publicly available at https://github.com/zhenzhiwang/HumanVid/.
66a1e2857e9bf88779ea40ee
Zhenzhi Wang
claimed_verified
2024-07-25T09:08:07.604Z
6519346a186bc3b6997c1aaf
/avatars/8981bd278962da50f9bbfb92c2abe2bf.svg
Zhenzhi Wang
user
zhenzhiwang
66a1e2857e9bf88779ea40ef
Yixuan Li
66a1e2857e9bf88779ea40f0
Yanhong Zeng
admin_assigned
2024-07-25T10:04:39.166Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
66a1e2857e9bf88779ea40f1
Youqing Fang
admin_assigned
2024-07-25T10:04:48.288Z
6600f734997ede4f9b1bb33e
https://cdn-avatars.huggingface.co/v1/production/uploads/6600f734997ede4f9b1bb33e/vvBsz6c1WUzAl9ifGSl_m.jpeg
Fang
user
Youqing
66a1e2857e9bf88779ea40f2
Yuwei Guo
claimed_verified
2025-03-16T21:23:58.366Z
6371f83d5ffa922d638ef486
https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
Yuwei Guo
user
guoyww
66a1e2857e9bf88779ea40f3
Wenran Liu
66a1e2857e9bf88779ea40f4
Jing Tan
claimed_verified
2024-07-26T07:40:34.873Z
65367c40061949598892dbdc
/avatars/4baf27263841471cbd5f629a8b99424d.svg
Jing Tan
user
jingtan
66a1e2857e9bf88779ea40f5
Kai Chen
66a1e2857e9bf88779ea40f6
Tianfan Xue
66a1e2857e9bf88779ea40f7
Bo Dai
admin_assigned
2024-07-25T10:21:07.145Z
665e2bd0fbaa279db369b488
/avatars/8cbc18d17d2813617976dbe709235799.svg
Bo Dai
user
doubling
66a1e2857e9bf88779ea40f8
Dahua Lin
admin_assigned
2024-07-25T10:20:46.159Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
HumanVid: Demystifying Training Data for Camera-controllable Human Image
  Animation
Human image animation involves generating videos from a character photo,
allowing user control and unlocking potential for video and movie production.
While recent approaches yield impressive results using high-quality training
data, the inaccessibility of these datasets hampers fair and transparent
benchmarking. Moreover, these approaches prioritize 2D human motion and
overlook the significance of camera motions in videos, leading to limited
control and unstable video generation.To demystify the training data, we
present HumanVid, the first large-scale high-quality dataset tailored for human
image animation, which combines crafted real-world and synthetic data. For the
real-world data, we compile a vast collection of copyright-free real-world
videos from the internet. Through a carefully designed rule-based filtering
strategy, we ensure the inclusion of high-quality videos, resulting in a
collection of 20K human-centric videos in 1080P resolution. Human and camera
motion annotation is accomplished using a 2D pose estimator and a SLAM-based
method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets
to augment existing available 3D assets. Notably, we introduce a rule-based
camera trajectory generation method, enabling the synthetic pipeline to
incorporate diverse and precise camera motion annotation, which can rarely be
found in real-world data. To verify the effectiveness of HumanVid, we establish
a baseline model named CamAnimate, short for Camera-controllable Human
Animation, that considers both human and camera motions as conditions. Through
extensive experimentation, we demonstrate that such simple baseline training on
our HumanVid achieves state-of-the-art performance in controlling both human
pose and camera motions, setting a new benchmark. Code and data will be
publicly available at https://github.com/zhenzhiwang/HumanVid/.
A new dataset, HumanVid, for human image animation includes high-quality real-world and synthetic video data with human and camera motion annotations, achieving state-of-the-art results in controlling both human pose and camera motions.
2D pose estimator SLAM-based method camera trajectory generation human image animation HumanVid CamAnimate