Mimir: Improving Video Diffusion Models for Precise Text Understanding
Text serves as the key control signal in video generation due to its
narrative nature. To render text descriptions into video clips, current video
diffusion models borrow features from text encoders yet struggle with limited
text comprehension. The recent success of large language models (LLMs)
showcases the power of decoder-only transformers, which offers three clear
benefits for text-to-video (T2V) generation, namely, precise text understanding
resulting from the superior scalability, imagination beyond the input text
enabled by next token prediction, and flexibility to prioritize user interests
through instruction tuning. Nevertheless, the feature distribution gap emerging
from the two different text modeling paradigms hinders the direct use of LLMs
in established T2V models. This work addresses this challenge with Mimir, an
end-to-end training framework featuring a carefully tailored token fuser to
harmonize the outputs from text encoders and LLMs. Such a design allows the T2V
model to fully leverage learned video priors while capitalizing on the
text-related capability of LLMs. Extensive quantitative and qualitative results
demonstrate the effectiveness of Mimir in generating high-quality videos with
excellent text comprehension, especially when processing short captions and
managing shifting motions. Project page:
https://lucaria-academy.github.io/Mimir/
67511c893da4637b2ec180c7
Shuai Tan
claimed_verified
2025-03-11T10:22:00.716Z
6697b62a27bf71ffe289492f
/avatars/f48095f1e2ca69836294343301be1700.svg
Shuai Tan
user
Shuaishuai0219
67511c893da4637b2ec180c8
Biao Gong
claimed_verified
2024-12-05T09:18:13.242Z
644fcbea4f7316588267dc80
https://cdn-avatars.huggingface.co/v1/production/uploads/644fcbea4f7316588267dc80/w8-2Gkaw9BN9VzppNXrTP.jpeg
Biao Gong
user
BiaoGong
67511c893da4637b2ec180c9
Yutong Feng
admin_assigned
2024-12-05T12:22:46.887Z
64a54e468cfaa458bd6844bf
https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
Yutong Feng
user
fengyutong
67511c893da4637b2ec180ca
Kecheng Zheng
admin_assigned
2024-12-05T12:22:40.934Z
64252045a4f3051f54dd1d53
/avatars/0e423a3291091be3b4736a14da3ce495.svg
kecheng zheng
user
zkcys001
67511c893da4637b2ec180cb
Dandan Zheng
admin_assigned
2024-12-05T12:22:30.343Z
65dd699a89a2a760d15f7d35
/avatars/e098b56c413d147d1f38cf33a4b0ecde.svg
Dandan Zheng
user
zhengdd0422
67511c893da4637b2ec180cc
Shuwei Shi
admin_assigned
2024-12-05T12:23:03.099Z
65688c20178ee07f040bcae8
/avatars/24d57ad55eb00af398ca3df6507af239.svg
SHUWEI SHI
user
ssw20
67511c893da4637b2ec180cd
Yujun Shen
67511c893da4637b2ec180ce
Jingdong Chen
67511c893da4637b2ec180cf
Ming Yang
Mimir: Improving Video Diffusion Models for Precise Text Understanding
Text serves as the key control signal in video generation due to its
narrative nature. To render text descriptions into video clips, current video
diffusion models borrow features from text encoders yet struggle with limited
text comprehension. The recent success of large language models (LLMs)
showcases the power of decoder-only transformers, which offers three clear
benefits for text-to-video (T2V) generation, namely, precise text understanding
resulting from the superior scalability, imagination beyond the input text
enabled by next token prediction, and flexibility to prioritize user interests
through instruction tuning. Nevertheless, the feature distribution gap emerging
from the two different text modeling paradigms hinders the direct use of LLMs
in established T2V models. This work addresses this challenge with Mimir, an
end-to-end training framework featuring a carefully tailored token fuser to
harmonize the outputs from text encoders and LLMs. Such a design allows the T2V
model to fully leverage learned video priors while capitalizing on the
text-related capability of LLMs. Extensive quantitative and qualitative results
demonstrate the effectiveness of Mimir in generating high-quality videos with
excellent text comprehension, especially when processing short captions and
managing shifting motions. Project page:
https://lucaria-academy.github.io/Mimir/
Mimir, using token fuser, integrates video priors and LLM capabilities for improved text-to-video generation, with strong text comprehension and handling of short captions and motion.
video diffusion models text encoders large language models (LLMs) decoder-only transformers precise text understanding imagination instruction tuning token fuser text-to-video (T2V) feature distribution gap learned video priors