title: The Best of Both Worlds: Integrating Language Models and Diffusion
    Models for Video Generation
summary: Recent advancements in text-to-video (T2V) generation have been driven by two
  competing paradigms: autoregressive language models and diffusion models.
  However, each paradigm has intrinsic limitations: language models struggle with
  visual quality and error accumulation, while diffusion models lack semantic
  understanding and causal modeling. In this work, we propose LanDiff, a hybrid
  framework that synergizes the strengths of both paradigms through
  coarse-to-fine generation. Our architecture introduces three key innovations:
  (1) a semantic tokenizer that compresses 3D visual features into compact 1D
  discrete representations through efficient semantic compression, achieving a
  sim14,000times compression ratio; (2) a language model that generates
  semantic tokens with high-level semantic relationships; (3) a streaming
  diffusion model that refines coarse semantics into high-fidelity videos.
  Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the
  VBench T2V benchmark, surpassing the state-of-the-art open-source models
  Hunyuan Video (13B) and other commercial models such as Sora, Keling, and
  Hailuo. Furthermore, our model also achieves state-of-the-art performance in
  long video generation, surpassing other open-source models in this field. Our
  demo can be viewed at https://landiff.github.io/.
paper_authors._id: 67ca7b8a2a2c299d98944909
paper_authors.name: Aoxiong Yin
paper_authors._id: 67ca7b8a2a2c299d9894490a
paper_authors.name: Kai Shen
paper_authors._id: 67ca7b8a2a2c299d9894490b
paper_authors.name: Yichong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:41:57.733Z
paper_authors._id: 64a0347b528a9bbe59d6e08c
paper_authors.avatarUrl: /avatars/6dd0bad84d711d1048a0a4169e621773.svg
paper_authors.fullname: Yichong Leng
paper_authors.type: user
paper_authors.user: ustcscallion
paper_authors._id: 67ca7b8a2a2c299d9894490c
paper_authors.name: Xu Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:43:46.715Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 67ca7b8a2a2c299d9894490d
paper_authors.name: Xinyu Zhou
paper_authors._id: 67ca7b8a2a2c299d9894490e
paper_authors.name: Juncheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:42:06.818Z
paper_authors._id: 67bc247b593452cc18965cb1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/EA3kTYaaff0Hr7-dGiOOj.png
paper_authors.fullname: JUNCHENG LI
paper_authors.type: user
paper_authors.user: JunchengLi
paper_authors._id: 67ca7b8a2a2c299d9894490f
paper_authors.name: Siliang Tang
paper_title: The Best of Both Worlds: Integrating Language Models and Diffusion
    Models for Video Generation
paper_summary: Recent advancements in text-to-video (T2V) generation have been driven by two
  competing paradigms: autoregressive language models and diffusion models.
  However, each paradigm has intrinsic limitations: language models struggle with
  visual quality and error accumulation, while diffusion models lack semantic
  understanding and causal modeling. In this work, we propose LanDiff, a hybrid
  framework that synergizes the strengths of both paradigms through
  coarse-to-fine generation. Our architecture introduces three key innovations:
  (1) a semantic tokenizer that compresses 3D visual features into compact 1D
  discrete representations through efficient semantic compression, achieving a
  sim14,000times compression ratio; (2) a language model that generates
  semantic tokens with high-level semantic relationships; (3) a streaming
  diffusion model that refines coarse semantics into high-fidelity videos.
  Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the
  VBench T2V benchmark, surpassing the state-of-the-art open-source models
  Hunyuan Video (13B) and other commercial models such as Sora, Keling, and
  Hailuo. Furthermore, our model also achieves state-of-the-art performance in
  long video generation, surpassing other open-source models in this field. Our
  demo can be viewed at https://landiff.github.io/.
paper_ai_keywords: text-to-video (T2V) generation autoregressive language models diffusion models visual quality error accumulation semantic understanding causal modeling semantic tokenizer efficient semantic compression semantic tokens high-level semantic relationships streaming diffusion model coarse semantics high-fidelity videos VBench T2V benchmark Hunyuan Video long video generation