title: Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video
    Generation
summary: Significant advancements have been achieved in the realm of large-scale
  pre-trained text-to-video Diffusion Models (VDMs). However, previous methods
  either rely solely on pixel-based VDMs, which come with high computational
  costs, or on latent-based VDMs, which often struggle with precise text-video
  alignment. In this paper, we are the first to propose a hybrid model, dubbed as
  Show-1, which marries pixel-based and latent-based VDMs for text-to-video
  generation. Our model first uses pixel-based VDMs to produce a low-resolution
  video of strong text-video correlation. After that, we propose a novel expert
  translation method that employs the latent-based VDMs to further upsample the
  low-resolution video to high resolution. Compared to latent VDMs, Show-1 can
  produce high-quality videos of precise text-video alignment; Compared to pixel
  VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G
  vs 72G). We also validate our model on standard video generation benchmarks.
  Our code and model weights are publicly available at
  https://github.com/showlab/Show-1.
paper_authors._id: 651507604a40f143596de2f2
paper_authors.name: David Junhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T08:57:39.392Z
paper_authors._id: 6468855763a564ba347e1efa
paper_authors.avatarUrl: /avatars/caa99c3318c1d6d21be44d2d32795a62.svg
paper_authors.fullname: David Junhao ZHANG
paper_authors.type: user
paper_authors.user: Junhao233
paper_authors._id: 651507604a40f143596de2f3
paper_authors.name: Jay Zhangjie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-28T08:20:16.342Z
paper_authors._id: 633aaf695df91da9cea92960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
paper_authors.fullname: Jay Wu
paper_authors.type: user
paper_authors.user: jayw
paper_authors._id: 651507604a40f143596de2f4
paper_authors.name: Jia-Wei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:04:35.169Z
paper_authors._id: 64230048a73327caad9d8241
paper_authors.avatarUrl: /avatars/ac115f1a21c743db7c925f0e18451145.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: JiaweiLIU
paper_authors._id: 651507604a40f143596de2f5
paper_authors.name: Rui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:25:07.548Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 651507604a40f143596de2f6
paper_authors.name: Lingmin Ran
paper_authors._id: 651507604a40f143596de2f7
paper_authors.name: Yuchao Gu
paper_authors._id: 651507604a40f143596de2f8
paper_authors.name: Difei Gao
paper_authors._id: 651507604a40f143596de2f9
paper_authors.name: Mike Zheng Shou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-28T09:01:32.912Z
paper_authors._id: 63a55320ce5763e06f78519c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
paper_authors.fullname: Mike Shou
paper_authors.type: user
paper_authors.user: mikeshou
paper_title: Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video
    Generation
paper_summary: Significant advancements have been achieved in the realm of large-scale
  pre-trained text-to-video Diffusion Models (VDMs). However, previous methods
  either rely solely on pixel-based VDMs, which come with high computational
  costs, or on latent-based VDMs, which often struggle with precise text-video
  alignment. In this paper, we are the first to propose a hybrid model, dubbed as
  Show-1, which marries pixel-based and latent-based VDMs for text-to-video
  generation. Our model first uses pixel-based VDMs to produce a low-resolution
  video of strong text-video correlation. After that, we propose a novel expert
  translation method that employs the latent-based VDMs to further upsample the
  low-resolution video to high resolution. Compared to latent VDMs, Show-1 can
  produce high-quality videos of precise text-video alignment; Compared to pixel
  VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G
  vs 72G). We also validate our model on standard video generation benchmarks.
  Our code and model weights are publicly available at
  https://github.com/showlab/Show-1.
paper_ai_summary: Show-1, a hybrid model combining pixel-based and latent-based VDMs, generates high-quality text-to-video with precise alignment and reduced computational cost.
paper_ai_keywords: Diffusion Models (VDMs) pixel-based VDMs latent-based VDMs text-video correlation expert translation method text-video alignment GPU memory usage