FusionFrames: Efficient Architectural Aspects for Text-to-Video
  Generation Pipeline
Multimedia generation approaches occupy a prominent place in artificial
intelligence research. Text-to-image models achieved high-quality results over
the last few years. However, video synthesis methods recently started to
develop. This paper presents a new two-stage latent diffusion text-to-video
generation architecture based on the text-to-image diffusion model. The first
stage concerns keyframes synthesis to figure the storyline of a video, while
the second one is devoted to interpolation frames generation to make movements
of the scene and objects smooth. We compare several temporal conditioning
approaches for keyframes generation. The results show the advantage of using
separate temporal blocks over temporal layers in terms of metrics reflecting
video generation quality aspects and human preference. The design of our
interpolation model significantly reduces computational costs compared to other
masked frame interpolation approaches. Furthermore, we evaluate different
configurations of MoVQ-based video decoding scheme to improve consistency and
achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our
pipeline with existing solutions and achieve top-2 scores overall and top-1
among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page:
https://ai-forever.github.io/kandinsky-video/
655eedc22ff001a46a5104f7
Vladimir Arkhipkin
admin_assigned
2023-11-23T10:56:44.959Z
655b81eb31c4978366a991cb
/avatars/48987024c708c521224d66381bb08c43.svg
Arkhipkin Vladimir
user
oriBetelgeuse
655eedc22ff001a46a5104f8
Zein Shaheen
admin_assigned
2023-11-23T10:56:52.580Z
64d8e8539a6a7ae9844e6fc0
/avatars/f5b805ac1e123bcc2c3bbddf374a358f.svg
Zein Shaheen
user
zeinshaheen
655eedc22ff001a46a5104f9
Viacheslav Vasilev
admin_assigned
2023-11-23T10:56:58.592Z
64e4c7764af6c29a0697f57b
/avatars/efc4e9f9b105586fd090b22a1bc7dbb7.svg
Viacheslav Vasilev
user
vvasilev
655eedc22ff001a46a5104fa
Elizaveta Dakhova
admin_assigned
2023-11-23T10:57:05.463Z
61c0b5de505ec68f54e4ece0
/avatars/46567f9b880e127419db0702cf00a146.svg
Elizaveta Dakhova
user
lizadakhova
655eedc22ff001a46a5104fb
Andrey Kuznetsov
admin_assigned
2023-11-23T10:57:59.070Z
643984dceb7c5616ef3f5d54
https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
Andrey Kuznetsov
user
kuznetsoffandrey
655eedc22ff001a46a5104fc
Denis Dimitrov
claimed_verified
2024-06-13T09:33:04.763Z
6669a678465d1d802181e456
https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
Denis Dimitrov
user
dendimitrov
FusionFrames: Efficient Architectural Aspects for Text-to-Video
  Generation Pipeline
Multimedia generation approaches occupy a prominent place in artificial
intelligence research. Text-to-image models achieved high-quality results over
the last few years. However, video synthesis methods recently started to
develop. This paper presents a new two-stage latent diffusion text-to-video
generation architecture based on the text-to-image diffusion model. The first
stage concerns keyframes synthesis to figure the storyline of a video, while
the second one is devoted to interpolation frames generation to make movements
of the scene and objects smooth. We compare several temporal conditioning
approaches for keyframes generation. The results show the advantage of using
separate temporal blocks over temporal layers in terms of metrics reflecting
video generation quality aspects and human preference. The design of our
interpolation model significantly reduces computational costs compared to other
masked frame interpolation approaches. Furthermore, we evaluate different
configurations of MoVQ-based video decoding scheme to improve consistency and
achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our
pipeline with existing solutions and achieve top-2 scores overall and top-1
among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page:
https://ai-forever.github.io/kandinsky-video/
A new two-stage latent diffusion model generates videos from text, achieving high quality and efficiency in keyframe synthesis and interpolation.
latent diffusion model text-to-video keyframes synthesis interpolation frames temporal conditioning temporal blocks temporal layers masked frame interpolation MoVQ-based video decoding PSNR SSIM MSE LPIPS CLIPSIM FVD