NVLM: Open Frontier-Class Multimodal LLMs
We introduce NVLM 1.0, a family of frontier-class multimodal large language
models (LLMs) that achieve state-of-the-art results on vision-language tasks,
rivaling the leading proprietary models (e.g., GPT-4o) and open-access models
(e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved
text-only performance over its LLM backbone after multimodal training. In terms
of model design, we perform a comprehensive comparison between decoder-only
multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g.,
Flamingo). Based on the strengths and weaknesses of both approaches, we propose
a novel architecture that enhances both training efficiency and multimodal
reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for
tile-based dynamic high-resolution images, which significantly boosts
performance on multimodal reasoning and OCR-related tasks. Regarding training
data, we meticulously curate and provide detailed information on our multimodal
pretraining and supervised fine-tuning datasets. Our findings indicate that
dataset quality and task diversity are more important than scale, even during
the pretraining phase, across all architectures. Notably, we develop
production-grade multimodality for the NVLM-1.0 models, enabling them to excel
in vision-language tasks while maintaining and even improving text-only
performance compared to their LLM backbones. To achieve this, we craft and
integrate a high-quality text-only dataset into multimodal training, alongside
a substantial amount of multimodal math and reasoning data, leading to enhanced
math and coding capabilities across modalities. To advance research in the
field, we are releasing the model weights and will open-source the code for the
community: https://nvlm-project.github.io/.
66ea3bb26d83fd1e4fce2550
Wenliang Dai
claimed_verified
2024-09-19T07:07:26.360Z
61bcb994a4d85c3406ad2c23
/avatars/8ff55c0329d3c0986d54af08f8c0c7fa.svg
Wenliang
user
wdaiai
66ea3bb26d83fd1e4fce2551
Nayeon Lee
claimed_verified
2024-09-18T12:03:49.192Z
65e7a68a263a71d47fbe7942
/avatars/c727e702c7d56f1a7f31bd6bfa4d3047.svg
Nayeon LEE
user
nayeon7lee
66ea3bb26d83fd1e4fce2552
Boxin Wang
admin_assigned
2024-09-18T07:16:10.325Z
6493391101196c83393b3104
https://cdn-avatars.huggingface.co/v1/production/uploads/6493391101196c83393b3104/C3bl7zYr-91BVxweA2KeW.jpeg
Boxin Wang
user
boxin-wbx
66ea3bb26d83fd1e4fce2553
Zhuoling Yang
claimed_verified
2024-09-19T07:07:13.707Z
66d77106673a350b18f0eb14
/avatars/5da80142ba244160be6fab95e169db7f.svg
Zhuolin Yang
user
lucas110550
66ea3bb26d83fd1e4fce2554
Zihan Liu
admin_assigned
2024-09-18T07:15:55.780Z
65f33b1c9f7970ccc0234cbf
/avatars/99fbab303912e3674663251c04279907.svg
Zihan Liu
user
zihanliu
66ea3bb26d83fd1e4fce2555
Jon Barker
admin_assigned
2024-09-18T07:15:41.805Z
664f890e5a78417f01d350d3
/avatars/c3dc91b1f49fafd0bbbced5c0c89b3c5.svg
Jon Barker
user
jon-barker
66ea3bb26d83fd1e4fce2556
Tuomas Rintamaki
admin_assigned
2024-09-18T07:14:40.331Z
666247f1eaeff469dac00fe8
/avatars/128d5efd79cc005d2ede0c72657030f0.svg
Tuomas Rintamaki
user
tuomass
66ea3bb26d83fd1e4fce2557
Mohammad Shoeybi
admin_assigned
2024-09-18T07:14:33.926Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
66ea3bb26d83fd1e4fce2558
Bryan Catanzaro
admin_assigned
2024-09-18T07:14:27.685Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66ea3bb26d83fd1e4fce2559
Wei Ping
NVLM: Open Frontier-Class Multimodal LLMs
We introduce NVLM 1.0, a family of frontier-class multimodal large language
models (LLMs) that achieve state-of-the-art results on vision-language tasks,
rivaling the leading proprietary models (e.g., GPT-4o) and open-access models
(e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved
text-only performance over its LLM backbone after multimodal training. In terms
of model design, we perform a comprehensive comparison between decoder-only
multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g.,
Flamingo). Based on the strengths and weaknesses of both approaches, we propose
a novel architecture that enhances both training efficiency and multimodal
reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for
tile-based dynamic high-resolution images, which significantly boosts
performance on multimodal reasoning and OCR-related tasks. Regarding training
data, we meticulously curate and provide detailed information on our multimodal
pretraining and supervised fine-tuning datasets. Our findings indicate that
dataset quality and task diversity are more important than scale, even during
the pretraining phase, across all architectures. Notably, we develop
production-grade multimodality for the NVLM-1.0 models, enabling them to excel
in vision-language tasks while maintaining and even improving text-only
performance compared to their LLM backbones. To achieve this, we craft and
integrate a high-quality text-only dataset into multimodal training, alongside
a substantial amount of multimodal math and reasoning data, leading to enhanced
math and coding capabilities across modalities. To advance research in the
field, we are releasing the model weights and will open-source the code for the
community: https://nvlm-project.github.io/.
NVML 1.0, a family of multimodal large language models, achieves state-of-the-art results in vision-language tasks by combining text-only and multimodal training, utilizing a new architecture and dataset strategy.
multimodal large language models decoder-only multimodal LLMs cross-attention-based models 1-D tile-tagging design tile-based dynamic high-resolution images multimodal pretraining supervised fine-tuning datasets multimodality vision-language tasks text-only performance math and reasoning data