Good folks at 
text
Good folks at 
@nvidia
mention
nvidia
 have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
text
 have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
new_line
new_line
Here is how they did it:
text
Here is how they did it:
new_line
new_line
1. Model Architecture Design:
text
1. Model Architecture Design:
new_line
- Developed three model architectures:
text
- Developed three model architectures:
new_line
a) NVLM-D: Decoder-only architecture
text
a) NVLM-D: Decoder-only architecture
new_line
b) NVLM-X: Cross-attention-based architecture
text
b) NVLM-X: Cross-attention-based architecture
new_line
c) NVLM-H: Novel hybrid architecture
text
c) NVLM-H: Novel hybrid architecture
new_line
new_line
2. Vision Encoder:
text
2. Vision Encoder:
new_line
- Used InternViT-6B-448px-V1-5 as the vision encoder
text
- Used InternViT-6B-448px-V1-5 as the vision encoder
new_line
- Implemented dynamic high-resolution (DHR) input handling
text
- Implemented dynamic high-resolution (DHR) input handling
new_line
new_line
3. Language Model:
text
3. Language Model:
new_line
- Used Qwen2-72B-Instruct as the base LLM
text
- Used Qwen2-72B-Instruct as the base LLM
new_line
new_line
4. Training Data Curation:
text
4. Training Data Curation:
new_line
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
text
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
new_line
- Included diverse task-oriented datasets for various capabilities
text
- Included diverse task-oriented datasets for various capabilities
new_line
new_line
5. Pretraining:
text
5. Pretraining:
new_line
- Froze LLM and vision encoder
text
- Froze LLM and vision encoder
new_line
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
text
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
new_line
- Used a large batch size of 2048
text
- Used a large batch size of 2048
new_line
new_line
6. Supervised Fine-Tuning (SFT):
text
6. Supervised Fine-Tuning (SFT):
new_line
- Unfroze LLM while keeping the vision encoder frozen
text
- Unfroze LLM while keeping the vision encoder frozen
new_line
- Trained on multimodal SFT datasets and high-quality text-only SFT data
text
- Trained on multimodal SFT datasets and high-quality text-only SFT data
new_line
- Implemented 1-D tile tagging for dynamic high-resolution inputs
text
- Implemented 1-D tile tagging for dynamic high-resolution inputs
new_line
new_line
7. Evaluation:
text
7. Evaluation:
new_line
- Evaluated on multiple vision-language benchmarks
text
- Evaluated on multiple vision-language benchmarks
new_line
- Compared performance to leading proprietary and open-source models
text
- Compared performance to leading proprietary and open-source models
new_line
new_line
8. Optimization:
text
8. Optimization:
new_line
- Iterated on model designs and training approaches
text
- Iterated on model designs and training approaches
new_line
- Used smaller 34B models for faster experimentation before scaling to 72B
text
- Used smaller 34B models for faster experimentation before scaling to 72B
new_line
new_line
9. Now comes the best part...Open-Sourcing:
text
9. Now comes the best part...Open-Sourcing:
new_line
- Released model weights and full technical details to the research community
text
- Released model weights and full technical details to the research community
new_line
new_line
The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
text
The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
Good folks at @nvidia have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.

Here is how they did it:

1. Model Architecture Design:
- Developed three model architectures:
a) NVLM-D: Decoder-only architecture
b) NVLM-X: Cross-attention-based architecture
c) NVLM-H: Novel hybrid architecture

2. Vision Encoder:
- Used InternViT-6B-448px-V1-5 as the vision encoder
- Implemented dynamic high-resolution (DHR) input handling

3. Language Model:
- Used Qwen2-72B-Instruct as the base LLM

4. Training Data Curation:
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
- Included diverse task-oriented datasets for various capabilities

5. Pretraining:
- Froze LLM and vision encoder
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
- Used a large batch size of 2048

6. Supervised Fine-Tuning (SFT):
- Unfroze LLM while keeping the vision encoder frozen
- Trained on multimodal SFT datasets and high-quality text-only SFT data
- Implemented 1-D tile tagging for dynamic high-resolution inputs

7. Evaluation:
- Evaluated on multiple vision-language benchmarks
- Compared performance to leading proprietary and open-source models

8. Optimization:
- Iterated on model designs and training approaches
- Used smaller 34B models for faster experimentation before scaling to 72B

9. Now comes the best part...Open-Sourcing:
- Released model weights and full technical details to the research community

The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
👍
Thunderbird2410 TacImpulse osanseviero AINovice2005 GuillaumeTHIBAULT
👀
John6666 louisbrulenaudet