title: FusionFrames: Efficient Architectural Aspects for Text-to-Video
    Generation Pipeline
summary: Multimedia generation approaches occupy a prominent place in artificial
  intelligence research. Text-to-image models achieved high-quality results over
  the last few years. However, video synthesis methods recently started to
  develop. This paper presents a new two-stage latent diffusion text-to-video
  generation architecture based on the text-to-image diffusion model. The first
  stage concerns keyframes synthesis to figure the storyline of a video, while
  the second one is devoted to interpolation frames generation to make movements
  of the scene and objects smooth. We compare several temporal conditioning
  approaches for keyframes generation. The results show the advantage of using
  separate temporal blocks over temporal layers in terms of metrics reflecting
  video generation quality aspects and human preference. The design of our
  interpolation model significantly reduces computational costs compared to other
  masked frame interpolation approaches. Furthermore, we evaluate different
  configurations of MoVQ-based video decoding scheme to improve consistency and
  achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our
  pipeline with existing solutions and achieve top-2 scores overall and top-1
  among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page:
  https://ai-forever.github.io/kandinsky-video/
paper_authors._id: 655eedc22ff001a46a5104f7
paper_authors.name: Vladimir Arkhipkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:56:44.959Z
paper_authors._id: 655b81eb31c4978366a991cb
paper_authors.avatarUrl: /avatars/48987024c708c521224d66381bb08c43.svg
paper_authors.fullname: Arkhipkin Vladimir
paper_authors.type: user
paper_authors.user: oriBetelgeuse
paper_authors._id: 655eedc22ff001a46a5104f8
paper_authors.name: Zein Shaheen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:56:52.580Z
paper_authors._id: 64d8e8539a6a7ae9844e6fc0
paper_authors.avatarUrl: /avatars/f5b805ac1e123bcc2c3bbddf374a358f.svg
paper_authors.fullname: Zein Shaheen
paper_authors.type: user
paper_authors.user: zeinshaheen
paper_authors._id: 655eedc22ff001a46a5104f9
paper_authors.name: Viacheslav Vasilev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:56:58.592Z
paper_authors._id: 64e4c7764af6c29a0697f57b
paper_authors.avatarUrl: /avatars/efc4e9f9b105586fd090b22a1bc7dbb7.svg
paper_authors.fullname: Viacheslav Vasilev
paper_authors.type: user
paper_authors.user: vvasilev
paper_authors._id: 655eedc22ff001a46a5104fa
paper_authors.name: Elizaveta Dakhova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:57:05.463Z
paper_authors._id: 61c0b5de505ec68f54e4ece0
paper_authors.avatarUrl: /avatars/46567f9b880e127419db0702cf00a146.svg
paper_authors.fullname: Elizaveta Dakhova
paper_authors.type: user
paper_authors.user: lizadakhova
paper_authors._id: 655eedc22ff001a46a5104fb
paper_authors.name: Andrey Kuznetsov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:57:59.070Z
paper_authors._id: 643984dceb7c5616ef3f5d54
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
paper_authors.fullname: Andrey Kuznetsov
paper_authors.type: user
paper_authors.user: kuznetsoffandrey
paper_authors._id: 655eedc22ff001a46a5104fc
paper_authors.name: Denis Dimitrov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:33:04.763Z
paper_authors._id: 6669a678465d1d802181e456
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
paper_authors.fullname: Denis Dimitrov
paper_authors.type: user
paper_authors.user: dendimitrov
paper_title: FusionFrames: Efficient Architectural Aspects for Text-to-Video
    Generation Pipeline
paper_summary: Multimedia generation approaches occupy a prominent place in artificial
  intelligence research. Text-to-image models achieved high-quality results over
  the last few years. However, video synthesis methods recently started to
  develop. This paper presents a new two-stage latent diffusion text-to-video
  generation architecture based on the text-to-image diffusion model. The first
  stage concerns keyframes synthesis to figure the storyline of a video, while
  the second one is devoted to interpolation frames generation to make movements
  of the scene and objects smooth. We compare several temporal conditioning
  approaches for keyframes generation. The results show the advantage of using
  separate temporal blocks over temporal layers in terms of metrics reflecting
  video generation quality aspects and human preference. The design of our
  interpolation model significantly reduces computational costs compared to other
  masked frame interpolation approaches. Furthermore, we evaluate different
  configurations of MoVQ-based video decoding scheme to improve consistency and
  achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our
  pipeline with existing solutions and achieve top-2 scores overall and top-1
  among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page:
  https://ai-forever.github.io/kandinsky-video/
paper_ai_keywords: latent diffusion text-to-image diffusion model. keyframes synthesis. interpolation frames generation. temporal conditioning. classification performance. computational costs. MoVQ-based video decoding scheme. consistency. PSNR. SSIM. MSE. LPIPS scores. CLUPSIM. FVD.