AniClipart: Clipart Animation with Text-to-Video Priors
Clipart, a pre-made graphic art form, offers a convenient and efficient way
of illustrating visual content. Traditional workflows to convert static clipart
images into motion sequences are laborious and time-consuming, involving
numerous intricate steps like rigging, key animation and in-betweening. Recent
advancements in text-to-video generation hold great potential in resolving this
problem. Nevertheless, direct application of text-to-video generation models
often struggles to retain the visual identity of clipart images or generate
cartoon-style motions, resulting in unsatisfactory animation outcomes. In this
paper, we introduce AniClipart, a system that transforms static clipart images
into high-quality motion sequences guided by text-to-video priors. To generate
cartoon-style and smooth motion, we first define B\'{e}zier curves over
keypoints of the clipart image as a form of motion regularization. We then
align the motion trajectories of the keypoints with the provided text prompt by
optimizing the Video Score Distillation Sampling (VSDS) loss, which encodes
adequate knowledge of natural motion within a pretrained text-to-video
diffusion model. With a differentiable As-Rigid-As-Possible shape deformation
algorithm, our method can be end-to-end optimized while maintaining deformation
rigidity. Experimental results show that the proposed AniClipart consistently
outperforms existing image-to-video generation models, in terms of text-video
alignment, visual identity preservation, and motion consistency. Furthermore,
we showcase the versatility of AniClipart by adapting it to generate a broader
array of animation formats, such as layered animation, which allows topological
changes.
6621c2ea118a0a5cefaec4ee
Ronghuan Wu
6621c2ea118a0a5cefaec4ef
Wanchao Su
extracted_pending
2024-07-07T13:20:27.525Z
668a960da1837d8b34f39dec
/avatars/a0751920f7a625dce6337400d05d65d6.svg
Wanchao
user
WancSSSSS
6621c2ea118a0a5cefaec4f0
Kede Ma
6621c2ea118a0a5cefaec4f1
Jing Liao
admin_assigned
2024-04-19T08:40:18.462Z
65e77726767bfc7d109c45bf
/avatars/24e68c86e06055ea1209598ba49ce8b9.svg
Jing Liao
user
CeciliaJL
AniClipart: Clipart Animation with Text-to-Video Priors
Clipart, a pre-made graphic art form, offers a convenient and efficient way
of illustrating visual content. Traditional workflows to convert static clipart
images into motion sequences are laborious and time-consuming, involving
numerous intricate steps like rigging, key animation and in-betweening. Recent
advancements in text-to-video generation hold great potential in resolving this
problem. Nevertheless, direct application of text-to-video generation models
often struggles to retain the visual identity of clipart images or generate
cartoon-style motions, resulting in unsatisfactory animation outcomes. In this
paper, we introduce AniClipart, a system that transforms static clipart images
into high-quality motion sequences guided by text-to-video priors. To generate
cartoon-style and smooth motion, we first define B\'{e}zier curves over
keypoints of the clipart image as a form of motion regularization. We then
align the motion trajectories of the keypoints with the provided text prompt by
optimizing the Video Score Distillation Sampling (VSDS) loss, which encodes
adequate knowledge of natural motion within a pretrained text-to-video
diffusion model. With a differentiable As-Rigid-As-Possible shape deformation
algorithm, our method can be end-to-end optimized while maintaining deformation
rigidity. Experimental results show that the proposed AniClipart consistently
outperforms existing image-to-video generation models, in terms of text-video
alignment, visual identity preservation, and motion consistency. Furthermore,
we showcase the versatility of AniClipart by adapting it to generate a broader
array of animation formats, such as layered animation, which allows topological
changes.
AniClipart system transforms static clipart images into high-quality motion sequences using Bézier curves, text-to-video priors, and Video Score Distillation Sampling loss for coherent and identity-preserved animations.
Bézier curves motion regularization keypoints Video Score Distillation Sampling (VSDS) loss text-to-video diffusion model As-Rigid-As-Possible shape deformation layered animation topological changes