title: Mimir: Improving Video Diffusion Models for Precise Text Understanding
summary: Text serves as the key control signal in video generation due to its
  narrative nature. To render text descriptions into video clips, current video
  diffusion models borrow features from text encoders yet struggle with limited
  text comprehension. The recent success of large language models (LLMs)
  showcases the power of decoder-only transformers, which offers three clear
  benefits for text-to-video (T2V) generation, namely, precise text understanding
  resulting from the superior scalability, imagination beyond the input text
  enabled by next token prediction, and flexibility to prioritize user interests
  through instruction tuning. Nevertheless, the feature distribution gap emerging
  from the two different text modeling paradigms hinders the direct use of LLMs
  in established T2V models. This work addresses this challenge with Mimir, an
  end-to-end training framework featuring a carefully tailored token fuser to
  harmonize the outputs from text encoders and LLMs. Such a design allows the T2V
  model to fully leverage learned video priors while capitalizing on the
  text-related capability of LLMs. Extensive quantitative and qualitative results
  demonstrate the effectiveness of Mimir in generating high-quality videos with
  excellent text comprehension, especially when processing short captions and
  managing shifting motions. Project page:
  https://lucaria-academy.github.io/Mimir/
paper_authors._id: 67511c893da4637b2ec180c7
paper_authors.name: Shuai Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T10:22:00.716Z
paper_authors._id: 6697b62a27bf71ffe289492f
paper_authors.avatarUrl: /avatars/f48095f1e2ca69836294343301be1700.svg
paper_authors.fullname: Shuai Tan
paper_authors.type: user
paper_authors.user: Shuaishuai0219
paper_authors._id: 67511c893da4637b2ec180c8
paper_authors.name: Biao Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:18:13.242Z
paper_authors._id: 644fcbea4f7316588267dc80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644fcbea4f7316588267dc80/w8-2Gkaw9BN9VzppNXrTP.jpeg
paper_authors.fullname: Biao Gong
paper_authors.type: user
paper_authors.user: BiaoGong
paper_authors._id: 67511c893da4637b2ec180c9
paper_authors.name: Yutong Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:22:46.887Z
paper_authors._id: 64a54e468cfaa458bd6844bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
paper_authors.fullname: Yutong Feng
paper_authors.type: user
paper_authors.user: fengyutong
paper_authors._id: 67511c893da4637b2ec180ca
paper_authors.name: Kecheng Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:22:40.934Z
paper_authors._id: 64252045a4f3051f54dd1d53
paper_authors.avatarUrl: /avatars/0e423a3291091be3b4736a14da3ce495.svg
paper_authors.fullname: kecheng zheng
paper_authors.type: user
paper_authors.user: zkcys001
paper_authors._id: 67511c893da4637b2ec180cb
paper_authors.name: Dandan Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:22:30.343Z
paper_authors._id: 65dd699a89a2a760d15f7d35
paper_authors.avatarUrl: /avatars/e098b56c413d147d1f38cf33a4b0ecde.svg
paper_authors.fullname: Dandan Zheng
paper_authors.type: user
paper_authors.user: zhengdd0422
paper_authors._id: 67511c893da4637b2ec180cc
paper_authors.name: Shuwei Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T12:23:03.099Z
paper_authors._id: 65688c20178ee07f040bcae8
paper_authors.avatarUrl: /avatars/24d57ad55eb00af398ca3df6507af239.svg
paper_authors.fullname: SHUWEI SHI
paper_authors.type: user
paper_authors.user: ssw20
paper_authors._id: 67511c893da4637b2ec180cd
paper_authors.name: Yujun Shen
paper_authors._id: 67511c893da4637b2ec180ce
paper_authors.name: Jingdong Chen
paper_authors._id: 67511c893da4637b2ec180cf
paper_authors.name: Ming Yang
paper_title: Mimir: Improving Video Diffusion Models for Precise Text Understanding
paper_summary: Text serves as the key control signal in video generation due to its
  narrative nature. To render text descriptions into video clips, current video
  diffusion models borrow features from text encoders yet struggle with limited
  text comprehension. The recent success of large language models (LLMs)
  showcases the power of decoder-only transformers, which offers three clear
  benefits for text-to-video (T2V) generation, namely, precise text understanding
  resulting from the superior scalability, imagination beyond the input text
  enabled by next token prediction, and flexibility to prioritize user interests
  through instruction tuning. Nevertheless, the feature distribution gap emerging
  from the two different text modeling paradigms hinders the direct use of LLMs
  in established T2V models. This work addresses this challenge with Mimir, an
  end-to-end training framework featuring a carefully tailored token fuser to
  harmonize the outputs from text encoders and LLMs. Such a design allows the T2V
  model to fully leverage learned video priors while capitalizing on the
  text-related capability of LLMs. Extensive quantitative and qualitative results
  demonstrate the effectiveness of Mimir in generating high-quality videos with
  excellent text comprehension, especially when processing short captions and
  managing shifting motions. Project page:
  https://lucaria-academy.github.io/Mimir/
paper_ai_summary: Mimir, using token fuser, integrates video priors and LLM capabilities for improved text-to-video generation, with strong text comprehension and handling of short captions and motion.
paper_ai_keywords: video diffusion models text encoders large language models (LLMs) decoder-only transformers precise text understanding imagination instruction tuning token fuser text-to-video (T2V) feature distribution gap learned video priors