title: AniClipart: Clipart Animation with Text-to-Video Priors
summary: Clipart, a pre-made graphic art form, offers a convenient and efficient way
  of illustrating visual content. Traditional workflows to convert static clipart
  images into motion sequences are laborious and time-consuming, involving
  numerous intricate steps like rigging, key animation and in-betweening. Recent
  advancements in text-to-video generation hold great potential in resolving this
  problem. Nevertheless, direct application of text-to-video generation models
  often struggles to retain the visual identity of clipart images or generate
  cartoon-style motions, resulting in unsatisfactory animation outcomes. In this
  paper, we introduce AniClipart, a system that transforms static clipart images
  into high-quality motion sequences guided by text-to-video priors. To generate
  cartoon-style and smooth motion, we first define B\'{e}zier curves over
  keypoints of the clipart image as a form of motion regularization. We then
  align the motion trajectories of the keypoints with the provided text prompt by
  optimizing the Video Score Distillation Sampling (VSDS) loss, which encodes
  adequate knowledge of natural motion within a pretrained text-to-video
  diffusion model. With a differentiable As-Rigid-As-Possible shape deformation
  algorithm, our method can be end-to-end optimized while maintaining deformation
  rigidity. Experimental results show that the proposed AniClipart consistently
  outperforms existing image-to-video generation models, in terms of text-video
  alignment, visual identity preservation, and motion consistency. Furthermore,
  we showcase the versatility of AniClipart by adapting it to generate a broader
  array of animation formats, such as layered animation, which allows topological
  changes.
paper_authors._id: 6621c2ea118a0a5cefaec4ee
paper_authors.name: Ronghuan Wu
paper_authors._id: 6621c2ea118a0a5cefaec4ef
paper_authors.name: Wanchao Su
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-07T13:20:27.525Z
paper_authors._id: 668a960da1837d8b34f39dec
paper_authors.avatarUrl: /avatars/a0751920f7a625dce6337400d05d65d6.svg
paper_authors.fullname: Wanchao
paper_authors.type: user
paper_authors.user: WancSSSSS
paper_authors._id: 6621c2ea118a0a5cefaec4f0
paper_authors.name: Kede Ma
paper_authors._id: 6621c2ea118a0a5cefaec4f1
paper_authors.name: Jing Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:40:18.462Z
paper_authors._id: 65e77726767bfc7d109c45bf
paper_authors.avatarUrl: /avatars/24e68c86e06055ea1209598ba49ce8b9.svg
paper_authors.fullname: Jing Liao
paper_authors.type: user
paper_authors.user: CeciliaJL
paper_title: AniClipart: Clipart Animation with Text-to-Video Priors
paper_summary: Clipart, a pre-made graphic art form, offers a convenient and efficient way
  of illustrating visual content. Traditional workflows to convert static clipart
  images into motion sequences are laborious and time-consuming, involving
  numerous intricate steps like rigging, key animation and in-betweening. Recent
  advancements in text-to-video generation hold great potential in resolving this
  problem. Nevertheless, direct application of text-to-video generation models
  often struggles to retain the visual identity of clipart images or generate
  cartoon-style motions, resulting in unsatisfactory animation outcomes. In this
  paper, we introduce AniClipart, a system that transforms static clipart images
  into high-quality motion sequences guided by text-to-video priors. To generate
  cartoon-style and smooth motion, we first define B\'{e}zier curves over
  keypoints of the clipart image as a form of motion regularization. We then
  align the motion trajectories of the keypoints with the provided text prompt by
  optimizing the Video Score Distillation Sampling (VSDS) loss, which encodes
  adequate knowledge of natural motion within a pretrained text-to-video
  diffusion model. With a differentiable As-Rigid-As-Possible shape deformation
  algorithm, our method can be end-to-end optimized while maintaining deformation
  rigidity. Experimental results show that the proposed AniClipart consistently
  outperforms existing image-to-video generation models, in terms of text-video
  alignment, visual identity preservation, and motion consistency. Furthermore,
  we showcase the versatility of AniClipart by adapting it to generate a broader
  array of animation formats, such as layered animation, which allows topological
  changes.
paper_ai_summary: AniClipart system transforms static clipart images into high-quality motion sequences using Bézier curves, text-to-video priors, and Video Score Distillation Sampling loss for coherent and identity-preserved animations.
paper_ai_keywords: Bézier curves motion regularization keypoints Video Score Distillation Sampling (VSDS) loss text-to-video diffusion model As-Rigid-As-Possible shape deformation layered animation topological changes