title: Pix2Gif: Motion-Guided Diffusion for GIF Generation
summary: We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video)
  generation. We tackle this problem differently by formulating the task as an
  image translation problem steered by text and motion magnitude prompts, as
  shown in teaser fig. To ensure that the model adheres to motion guidance, we
  propose a new motion-guided warping module to spatially transform the features
  of the source image conditioned on the two types of prompts. Furthermore, we
  introduce a perceptual loss to ensure the transformed feature map remains
  within the same space as the target image, ensuring content consistency and
  coherence. In preparation for the model training, we meticulously curated data
  by extracting coherent image frames from the TGIF video-caption dataset, which
  provides rich information about the temporal changes of subjects. After
  pretraining, we apply our model in a zero-shot manner to a number of video
  datasets. Extensive qualitative and quantitative experiments demonstrate the
  effectiveness of our model -- it not only captures the semantic prompt from
  text but also the spatial ones from motion guidance. We train all our models
  using a single node of 16xV100 GPUs. Code, dataset and models are made public
  at: https://hiteshk03.github.io/Pix2Gif/.
paper_authors._id: 65eaa2e62bcb457b2fd61622
paper_authors.name: Hitesh Kandala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:52:17.246Z
paper_authors._id: 642c17428f90c557f741c188
paper_authors.avatarUrl: /avatars/3577c59b73f0b2904a63a976663b74fb.svg
paper_authors.fullname: Hitesh Kandala
paper_authors.type: user
paper_authors.user: v1an1
paper_authors._id: 65eaa2e62bcb457b2fd61623
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:52:33.583Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 65eaa2e62bcb457b2fd61624
paper_authors.name: Jianwei Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-08T05:32:24.143Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_title: Pix2Gif: Motion-Guided Diffusion for GIF Generation
paper_summary: We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video)
  generation. We tackle this problem differently by formulating the task as an
  image translation problem steered by text and motion magnitude prompts, as
  shown in teaser fig. To ensure that the model adheres to motion guidance, we
  propose a new motion-guided warping module to spatially transform the features
  of the source image conditioned on the two types of prompts. Furthermore, we
  introduce a perceptual loss to ensure the transformed feature map remains
  within the same space as the target image, ensuring content consistency and
  coherence. In preparation for the model training, we meticulously curated data
  by extracting coherent image frames from the TGIF video-caption dataset, which
  provides rich information about the temporal changes of subjects. After
  pretraining, we apply our model in a zero-shot manner to a number of video
  datasets. Extensive qualitative and quantitative experiments demonstrate the
  effectiveness of our model -- it not only captures the semantic prompt from
  text but also the spatial ones from motion guidance. We train all our models
  using a single node of 16xV100 GPUs. Code, dataset and models are made public
  at: https://hiteshk03.github.io/Pix2Gif/.
paper_ai_summary: Pix2Gif, a motion-guided diffusion model, generates image-to-GIF videos using text and motion prompts, ensuring coherence and consistency through a perceptual loss and motion-guided warping module.
paper_ai_keywords: motion-guided diffusion model image translation motion-guided warping module perceptual loss TGIF video-caption dataset zero-shot diffusion model