title: NVLM: Open Frontier-Class Multimodal LLMs
summary: We introduce NVLM 1.0, a family of frontier-class multimodal large language
  models (LLMs) that achieve state-of-the-art results on vision-language tasks,
  rivaling the leading proprietary models (e.g., GPT-4o) and open-access models
  (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved
  text-only performance over its LLM backbone after multimodal training. In terms
  of model design, we perform a comprehensive comparison between decoder-only
  multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g.,
  Flamingo). Based on the strengths and weaknesses of both approaches, we propose
  a novel architecture that enhances both training efficiency and multimodal
  reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for
  tile-based dynamic high-resolution images, which significantly boosts
  performance on multimodal reasoning and OCR-related tasks. Regarding training
  data, we meticulously curate and provide detailed information on our multimodal
  pretraining and supervised fine-tuning datasets. Our findings indicate that
  dataset quality and task diversity are more important than scale, even during
  the pretraining phase, across all architectures. Notably, we develop
  production-grade multimodality for the NVLM-1.0 models, enabling them to excel
  in vision-language tasks while maintaining and even improving text-only
  performance compared to their LLM backbones. To achieve this, we craft and
  integrate a high-quality text-only dataset into multimodal training, alongside
  a substantial amount of multimodal math and reasoning data, leading to enhanced
  math and coding capabilities across modalities. To advance research in the
  field, we are releasing the model weights and will open-source the code for the
  community: https://nvlm-project.github.io/.
paper_authors._id: 66ea3bb26d83fd1e4fce2550
paper_authors.name: Wenliang Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:07:26.360Z
paper_authors._id: 61bcb994a4d85c3406ad2c23
paper_authors.avatarUrl: /avatars/8ff55c0329d3c0986d54af08f8c0c7fa.svg
paper_authors.fullname: Wenliang
paper_authors.type: user
paper_authors.user: wdaiai
paper_authors._id: 66ea3bb26d83fd1e4fce2551
paper_authors.name: Nayeon Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-18T12:03:49.192Z
paper_authors._id: 65e7a68a263a71d47fbe7942
paper_authors.avatarUrl: /avatars/c727e702c7d56f1a7f31bd6bfa4d3047.svg
paper_authors.fullname: Nayeon LEE
paper_authors.type: user
paper_authors.user: nayeon7lee
paper_authors._id: 66ea3bb26d83fd1e4fce2552
paper_authors.name: Boxin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:16:10.325Z
paper_authors._id: 6493391101196c83393b3104
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6493391101196c83393b3104/C3bl7zYr-91BVxweA2KeW.jpeg
paper_authors.fullname: Boxin Wang
paper_authors.type: user
paper_authors.user: boxin-wbx
paper_authors._id: 66ea3bb26d83fd1e4fce2553
paper_authors.name: Zhuoling Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:07:13.707Z
paper_authors._id: 66d77106673a350b18f0eb14
paper_authors.avatarUrl: /avatars/5da80142ba244160be6fab95e169db7f.svg
paper_authors.fullname: Zhuolin Yang
paper_authors.type: user
paper_authors.user: lucas110550
paper_authors._id: 66ea3bb26d83fd1e4fce2554
paper_authors.name: Zihan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:15:55.780Z
paper_authors._id: 65f33b1c9f7970ccc0234cbf
paper_authors.avatarUrl: /avatars/99fbab303912e3674663251c04279907.svg
paper_authors.fullname: Zihan Liu
paper_authors.type: user
paper_authors.user: zihanliu
paper_authors._id: 66ea3bb26d83fd1e4fce2555
paper_authors.name: Jon Barker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:15:41.805Z
paper_authors._id: 664f890e5a78417f01d350d3
paper_authors.avatarUrl: /avatars/c3dc91b1f49fafd0bbbced5c0c89b3c5.svg
paper_authors.fullname: Jon Barker
paper_authors.type: user
paper_authors.user: jon-barker
paper_authors._id: 66ea3bb26d83fd1e4fce2556
paper_authors.name: Tuomas Rintamaki
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:14:40.331Z
paper_authors._id: 666247f1eaeff469dac00fe8
paper_authors.avatarUrl: /avatars/128d5efd79cc005d2ede0c72657030f0.svg
paper_authors.fullname: Tuomas Rintamaki
paper_authors.type: user
paper_authors.user: tuomass
paper_authors._id: 66ea3bb26d83fd1e4fce2557
paper_authors.name: Mohammad Shoeybi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:14:33.926Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 66ea3bb26d83fd1e4fce2558
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T07:14:27.685Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66ea3bb26d83fd1e4fce2559
paper_authors.name: Wei Ping
paper_title: NVLM: Open Frontier-Class Multimodal LLMs
paper_summary: We introduce NVLM 1.0, a family of frontier-class multimodal large language
  models (LLMs) that achieve state-of-the-art results on vision-language tasks,
  rivaling the leading proprietary models (e.g., GPT-4o) and open-access models
  (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved
  text-only performance over its LLM backbone after multimodal training. In terms
  of model design, we perform a comprehensive comparison between decoder-only
  multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g.,
  Flamingo). Based on the strengths and weaknesses of both approaches, we propose
  a novel architecture that enhances both training efficiency and multimodal
  reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for
  tile-based dynamic high-resolution images, which significantly boosts
  performance on multimodal reasoning and OCR-related tasks. Regarding training
  data, we meticulously curate and provide detailed information on our multimodal
  pretraining and supervised fine-tuning datasets. Our findings indicate that
  dataset quality and task diversity are more important than scale, even during
  the pretraining phase, across all architectures. Notably, we develop
  production-grade multimodality for the NVLM-1.0 models, enabling them to excel
  in vision-language tasks while maintaining and even improving text-only
  performance compared to their LLM backbones. To achieve this, we craft and
  integrate a high-quality text-only dataset into multimodal training, alongside
  a substantial amount of multimodal math and reasoning data, leading to enhanced
  math and coding capabilities across modalities. To advance research in the
  field, we are releasing the model weights and will open-source the code for the
  community: https://nvlm-project.github.io/.
paper_ai_keywords: multimodal large language models vision-language tasks state-of-the-art decoder-only multimodal LLMs cross-attention-based models 1-D tile-tagging design tile-based dynamic high-resolution images multimodal reasoning capabilities OCR-related tasks dataset quality task diversity production-grade multimodality text-only performance multimodal pretraining supervised fine-tuning datasets high-quality text-only dataset multimodal math and reasoning data math and coding capabilities