title: Animated Stickers: Bringing Stickers to Life with Video Diffusion
summary: We introduce animated stickers, a video diffusion model which generates an
  animation conditioned on a text prompt and static sticker image. Our model is
  built on top of the state-of-the-art Emu text-to-image model, with the addition
  of temporal layers to model motion. Due to the domain gap, i.e. differences in
  visual and motion style, a model which performed well on generating natural
  videos can no longer generate vivid videos when applied to stickers. To bridge
  this gap, we employ a two-stage finetuning pipeline: first with weakly
  in-domain data, followed by human-in-the-loop (HITL) strategy which we term
  ensemble-of-teachers. It distills the best qualities of multiple teachers into
  a smaller student model. We show that this strategy allows us to specifically
  target improvements to motion quality while maintaining the style from the
  static image. With inference optimizations, our model is able to generate an
  eight-frame video with high-quality, interesting, and relevant motion in under
  one second.
paper_authors._id: 65c991bafe3148b9187d576c
paper_authors.name: David Yan
paper_authors._id: 65c991bafe3148b9187d576d
paper_authors.name: Winnie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:07:18.270Z
paper_authors._id: 64564b304a7ffb7d5a47f515
paper_authors.avatarUrl: /avatars/b848f02609d9ee8ab7df350ccb274c23.svg
paper_authors.fullname: Winnie Zhang
paper_authors.type: user
paper_authors.user: WinnieJackJack
paper_authors._id: 65c991bafe3148b9187d576e
paper_authors.name: Luxin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:07:25.007Z
paper_authors._id: 65a4fa7d2548c41ad9d9b710
paper_authors.avatarUrl: /avatars/3cace2d2f11f7194d8eca4b95b0b57cc.svg
paper_authors.fullname: Luxin Zhang
paper_authors.type: user
paper_authors.user: Luczzz
paper_authors._id: 65c991bafe3148b9187d576f
paper_authors.name: Anmol Kalia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:07:32.582Z
paper_authors._id: 6573c2a8769f3ee9bdf98a74
paper_authors.avatarUrl: /avatars/df40b6153ec7368561049746a9c8adc2.svg
paper_authors.fullname: Anmol Kalia
paper_authors.type: user
paper_authors.user: anmolkalia
paper_authors._id: 65c991bafe3148b9187d5770
paper_authors.name: Dingkang Wang
paper_authors._id: 65c991bafe3148b9187d5771
paper_authors.name: Ankit Ramchandani
paper_authors._id: 65c991bafe3148b9187d5772
paper_authors.name: Miao Liu
paper_authors._id: 65c991bafe3148b9187d5773
paper_authors.name: Albert Pumarola
paper_authors._id: 65c991bafe3148b9187d5774
paper_authors.name: Edgar Schoenfeld
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:08:30.615Z
paper_authors._id: 6571c662d5c6a6d3b0bdae88
paper_authors.avatarUrl: /avatars/8080bcdf8b331f62383b724050189660.svg
paper_authors.fullname: Edgar Schoenfeld
paper_authors.type: user
paper_authors.user: edgarschoenfeld
paper_authors._id: 65c991bafe3148b9187d5775
paper_authors.name: Elliot Blanchard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:08:36.850Z
paper_authors._id: 6563cf4a718b9db5215d07b4
paper_authors.avatarUrl: /avatars/20e5be0bd3fcd888959a0af0f0b462e4.svg
paper_authors.fullname: Elliot Blanchard
paper_authors.type: user
paper_authors.user: elliotblanchard
paper_authors._id: 65c991bafe3148b9187d5776
paper_authors.name: Krishna Narni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:08:42.582Z
paper_authors._id: 63441505a56e97f0495056a6
paper_authors.avatarUrl: /avatars/1497958399aeff861ed0b2fbb2a9b7b7.svg
paper_authors.fullname: Krishna Narni
paper_authors.type: user
paper_authors.user: krishnanarni
paper_authors._id: 65c991bafe3148b9187d5777
paper_authors.name: Yaqiao Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:08:52.709Z
paper_authors._id: 63d2a5a6c20657d24347000c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674749322268-noauth.jpeg
paper_authors.fullname: Luo
paper_authors.type: user
paper_authors.user: Yaqiao
paper_authors._id: 65c991bafe3148b9187d5778
paper_authors.name: Lawrence Chen
paper_authors._id: 65c991bafe3148b9187d5779
paper_authors.name: Guan Pang
paper_authors._id: 65c991bafe3148b9187d577a
paper_authors.name: Ali Thabet
paper_authors._id: 65c991bafe3148b9187d577b
paper_authors.name: Peter Vajda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:11:37.356Z
paper_authors._id: 636f4b35af79dd193603342b
paper_authors.avatarUrl: /avatars/577328195738d92ac41a3d0ff3108d7c.svg
paper_authors.fullname: Peter Vajda
paper_authors.type: user
paper_authors.user: vaphab
paper_authors._id: 65c991bafe3148b9187d577c
paper_authors.name: Amy Bearman
paper_authors._id: 65c991bafe3148b9187d577d
paper_authors.name: Licheng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T17:12:02.180Z
paper_authors._id: 6372b1f9bd81fae2b3a712e8
paper_authors.avatarUrl: /avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
paper_authors.fullname: Licheng Yu
paper_authors.type: user
paper_authors.user: lichengyu
paper_title: Animated Stickers: Bringing Stickers to Life with Video Diffusion
paper_summary: We introduce animated stickers, a video diffusion model which generates an
  animation conditioned on a text prompt and static sticker image. Our model is
  built on top of the state-of-the-art Emu text-to-image model, with the addition
  of temporal layers to model motion. Due to the domain gap, i.e. differences in
  visual and motion style, a model which performed well on generating natural
  videos can no longer generate vivid videos when applied to stickers. To bridge
  this gap, we employ a two-stage finetuning pipeline: first with weakly
  in-domain data, followed by human-in-the-loop (HITL) strategy which we term
  ensemble-of-teachers. It distills the best qualities of multiple teachers into
  a smaller student model. We show that this strategy allows us to specifically
  target improvements to motion quality while maintaining the style from the
  static image. With inference optimizations, our model is able to generate an
  eight-frame video with high-quality, interesting, and relevant motion in under
  one second.
paper_ai_keywords: video diffusion model text prompt static sticker image Emu text-to-image model temporal layers domain gap visual and application style in-domain data human-in-the-loop (HITL) strategy ensemble-of-teachers motion quality inference optimizations eight-frame video high-quality motion