luodian commited on
Commit
18bf6d6
1 Parent(s): c6b7458

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +133 -0
README.md ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets:
3
+ - lmms-lab/LLaVA-OneVision-Data
4
+ language:
5
+ - en
6
+ - zh
7
+ library_name: transformers
8
+ license: apache-2.0
9
+ metrics:
10
+ - accuracy
11
+ tags:
12
+ - multimodal
13
+ ---
14
+
15
+ # LLaVA-OneVision
16
+
17
+ ![banner](https://i.postimg.cc/pL17YtG4/WX20240508-220230-2x.png)
18
+
19
+ Play with the model on the [LLaVA OneVision Chat](https://llava-onevision.lmms-lab.com/).
20
+
21
+ ## Table of Contents
22
+
23
+ 1. [Model Summary](##model-summary)
24
+ 2. [Use](##use)
25
+ 3. [Limitations](##limitations)
26
+ 4. [Training](##training)
27
+ 5. [License](##license)
28
+ 6. [Citation](##citation)
29
+
30
+ ## Model Summary
31
+
32
+ `llava-onevision-72b-ov-chat` is our latest model specifically designed for chat scenarios. It is built upon `llava-onevision-72b-ov` and has undergone iterative DPO training with human preference, making it well-suited for chat applications.
33
+
34
+ Research by [Tianyi Xiong](https://tyxiong23.github.io/) indicates that our iterative DPO training method enhances the model's chat capabilities while preserving its instruction-following abilities.
35
+
36
+ For further details, please refer to our upcoming blog or paper.
37
+
38
+ - **Repository:** [LLaVA-VL/LLaVA-NeXT](https://github.com/LLaVA-VL/LLaVA-NeXT?tab=readme-ov-file)
39
+ - **Project Website:** [llava-onevision.lmms-lab.com](llava-onevision.lmms-lab.com)
40
+ - **Paper:** [LLaVA-OneVision](arxiv.org/abs/2408.03326)
41
+ - **Point of Contact:** [Tianyi Xiong](https://tyxiong23.github.io/), [Bo Li](mailto:[email protected])
42
+ - **Languages:** English, Chinese
43
+
44
+ ## Benchmark Performance
45
+
46
+ To be released
47
+
48
+ ## Use
49
+
50
+ ### Intended use
51
+
52
+ The model was trained on [LLaVA-OneVision Dataset](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data) and have the ability to interact with images, multi-image and videos.
53
+
54
+ **Feel free to share your generations in the Community tab!**
55
+
56
+ ### Generation
57
+
58
+ ```python
59
+ # pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
60
+ from llava.model.builder import load_pretrained_model
61
+ from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
62
+ from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, IGNORE_INDEX
63
+ from llava.conversation import conv_templates, SeparatorStyle
64
+
65
+ from PIL import Image
66
+ import requests
67
+ import copy
68
+ import torch
69
+
70
+ import sys
71
+ import warnings
72
+
73
+ warnings.filterwarnings("ignore")
74
+ pretrained = "lmms-lab/llava-onevision-qwen2-0.5b-si"
75
+ model_name = "llava_qwen"
76
+ device = "cuda"
77
+ device_map = "auto"
78
+ tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map) # Add any other thing you want to pass in llava_model_args
79
+
80
+ model.eval()
81
+
82
+ url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
83
+ image = Image.open(requests.get(url, stream=True).raw)
84
+ image_tensor = process_images([image], image_processor, model.config)
85
+ image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
86
+
87
+ conv_template = "qwen_1_5" # Make sure you use correct chat template for different models
88
+ question = DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image?"
89
+ conv = copy.deepcopy(conv_templates[conv_template])
90
+ conv.append_message(conv.roles[0], question)
91
+ conv.append_message(conv.roles[1], None)
92
+ prompt_question = conv.get_prompt()
93
+
94
+ input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
95
+ image_sizes = [image.size]
96
+
97
+
98
+ cont = model.generate(
99
+ input_ids,
100
+ images=image_tensor,
101
+ image_sizes=image_sizes,
102
+ do_sample=False,
103
+ temperature=0,
104
+ max_new_tokens=4096,
105
+ )
106
+ text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
107
+ print(text_outputs)
108
+ ```
109
+
110
+ # Training
111
+
112
+ ## Model
113
+
114
+ - **Architecture:** SO400M + Qwen2
115
+ - **Pretraining Stage:** LCS-558K, 1 epoch, projector
116
+ - **Mid Stage:** A mixture of 4.7M high-quality synthetic data, 1 epoch, full model
117
+ - **Final-Image Stage:** A mixture of 3.6M single-image data, 1 epoch, full model
118
+ - **OneVision Stage:** A mixture of 1.6M single-image/multi-image/video data, 1 epoch, full model
119
+ - **Precision:** bfloat16
120
+
121
+ ## Hardware & Software
122
+
123
+ - **GPUs:** 256 \* Nvidia Tesla A100 (for whole model series training)
124
+ - **Orchestration:** [Huggingface Trainer](https://huggingface.co/docs/transformers/main_classes/trainer)
125
+ - **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)
126
+
127
+ # Citation
128
+
129
+ ```
130
+ @article{li2024llavaonevision,
131
+ title={LLaVA-OneVision},
132
+ }
133
+ ```