content.raw: Good folks at 
content.type: text
content.value: Good folks at 
content.raw: @nvidia
content.type: mention
content.user: nvidia
content.raw:  have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
content.type: text
content.value:  have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
content.type: new_line
content.type: new_line
content.raw: Here is how they did it:
content.type: text
content.value: Here is how they did it:
content.type: new_line
content.type: new_line
content.raw: 1. Model Architecture Design:
content.type: text
content.value: 1. Model Architecture Design:
content.type: new_line
content.raw: - Developed three model architectures:
content.type: text
content.value: - Developed three model architectures:
content.type: new_line
content.raw: a) NVLM-D: Decoder-only architecture
content.type: text
content.value: a) NVLM-D: Decoder-only architecture
content.type: new_line
content.raw: b) NVLM-X: Cross-attention-based architecture
content.type: text
content.value: b) NVLM-X: Cross-attention-based architecture
content.type: new_line
content.raw: c) NVLM-H: Novel hybrid architecture
content.type: text
content.value: c) NVLM-H: Novel hybrid architecture
content.type: new_line
content.type: new_line
content.raw: 2. Vision Encoder:
content.type: text
content.value: 2. Vision Encoder:
content.type: new_line
content.raw: - Used InternViT-6B-448px-V1-5 as the vision encoder
content.type: text
content.value: - Used InternViT-6B-448px-V1-5 as the vision encoder
content.type: new_line
content.raw: - Implemented dynamic high-resolution (DHR) input handling
content.type: text
content.value: - Implemented dynamic high-resolution (DHR) input handling
content.type: new_line
content.type: new_line
content.raw: 3. Language Model:
content.type: text
content.value: 3. Language Model:
content.type: new_line
content.raw: - Used Qwen2-72B-Instruct as the base LLM
content.type: text
content.value: - Used Qwen2-72B-Instruct as the base LLM
content.type: new_line
content.type: new_line
content.raw: 4. Training Data Curation:
content.type: text
content.value: 4. Training Data Curation:
content.type: new_line
content.raw: - Carefully curated high-quality pretraining and supervised fine-tuning datasets
content.type: text
content.value: - Carefully curated high-quality pretraining and supervised fine-tuning datasets
content.type: new_line
content.raw: - Included diverse task-oriented datasets for various capabilities
content.type: text
content.value: - Included diverse task-oriented datasets for various capabilities
content.type: new_line
content.type: new_line
content.raw: 5. Pretraining:
content.type: text
content.value: 5. Pretraining:
content.type: new_line
content.raw: - Froze LLM and vision encoder
content.type: text
content.value: - Froze LLM and vision encoder
content.type: new_line
content.raw: - Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
content.type: text
content.value: - Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
content.type: new_line
content.raw: - Used a large batch size of 2048
content.type: text
content.value: - Used a large batch size of 2048
content.type: new_line
content.type: new_line
content.raw: 6. Supervised Fine-Tuning (SFT):
content.type: text
content.value: 6. Supervised Fine-Tuning (SFT):
content.type: new_line
content.raw: - Unfroze LLM while keeping the vision encoder frozen
content.type: text
content.value: - Unfroze LLM while keeping the vision encoder frozen
content.type: new_line
content.raw: - Trained on multimodal SFT datasets and high-quality text-only SFT data
content.type: text
content.value: - Trained on multimodal SFT datasets and high-quality text-only SFT data
content.type: new_line
content.raw: - Implemented 1-D tile tagging for dynamic high-resolution inputs
content.type: text
content.value: - Implemented 1-D tile tagging for dynamic high-resolution inputs
content.type: new_line
content.type: new_line
content.raw: 7. Evaluation:
content.type: text
content.value: 7. Evaluation:
content.type: new_line
content.raw: - Evaluated on multiple vision-language benchmarks
content.type: text
content.value: - Evaluated on multiple vision-language benchmarks
content.type: new_line
content.raw: - Compared performance to leading proprietary and open-source models
content.type: text
content.value: - Compared performance to leading proprietary and open-source models
content.type: new_line
content.type: new_line
content.raw: 8. Optimization:
content.type: text
content.value: 8. Optimization:
content.type: new_line
content.raw: - Iterated on model designs and training approaches
content.type: text
content.value: - Iterated on model designs and training approaches
content.type: new_line
content.raw: - Used smaller 34B models for faster experimentation before scaling to 72B
content.type: text
content.value: - Used smaller 34B models for faster experimentation before scaling to 72B
content.type: new_line
content.type: new_line
content.raw: 9. Now comes the best part...Open-Sourcing:
content.type: text
content.value: 9. Now comes the best part...Open-Sourcing:
content.type: new_line
content.raw: - Released model weights and full technical details to the research community
content.type: text
content.value: - Released model weights and full technical details to the research community
content.type: new_line
content.type: new_line
content.raw: The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
content.type: text
content.value: The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
rawContent: Good folks at @nvidia have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
  
  Here is how they did it:
  
  1. Model Architecture Design:
  - Developed three model architectures:
  a) NVLM-D: Decoder-only architecture
  b) NVLM-X: Cross-attention-based architecture
  c) NVLM-H: Novel hybrid architecture
  
  2. Vision Encoder:
  - Used InternViT-6B-448px-V1-5 as the vision encoder
  - Implemented dynamic high-resolution (DHR) input handling
  
  3. Language Model:
  - Used Qwen2-72B-Instruct as the base LLM
  
  4. Training Data Curation:
  - Carefully curated high-quality pretraining and supervised fine-tuning datasets
  - Included diverse task-oriented datasets for various capabilities
  
  5. Pretraining:
  - Froze LLM and vision encoder
  - Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
  - Used a large batch size of 2048
  
  6. Supervised Fine-Tuning (SFT):
  - Unfroze LLM while keeping the vision encoder frozen
  - Trained on multimodal SFT datasets and high-quality text-only SFT data
  - Implemented 1-D tile tagging for dynamic high-resolution inputs
  
  7. Evaluation:
  - Evaluated on multiple vision-language benchmarks
  - Compared performance to leading proprietary and open-source models
  
  8. Optimization:
  - Iterated on model designs and training approaches
  - Used smaller 34B models for faster experimentation before scaling to 72B
  
  9. Now comes the best part...Open-Sourcing:
  - Released model weights and full technical details to the research community
  
  The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
reactions.reaction: 👍
reactions.users: Thunderbird2410 TacImpulse osanseviero AINovice2005 GuillaumeTHIBAULT
reactions.reaction: 👀
reactions.users: John6666 louisbrulenaudet