Update README.md

README.md

@@ -1,57 +1,71 @@
 ---
-base_model: HuggingFaceM4/Idefics3-8B-Llama3
-library_name: peft
+library_name: transformers
 license: apache-2.0
-tags:
-- generated_from_trainer
-model-index:
-- name: idefics3-llama-vqav2
-  results: []
+datasets:
+- merve/vqav2-small
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
 
-# idefics3-llama-vqav2
 
-This model is a fine-tuned version of [HuggingFaceM4/Idefics3-8B-Llama3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) on an unknown dataset.
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/6141a88b3a0ec78603c9e784/PebmPLcCig5BlpUS99VUc.png)
 
-## Model description
+# Idefics3Llama Fine-tuned using QLoRA on VQAv2
 
-More information needed
+- This is the [Idefics3Llama](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) model QLoRA fine-tuned on a very small part of [VQAv2](https://huggingface.co/datasets/merve/vqav2-small) dataset.
 
-## Intended uses & limitations
+- Find the fine-tuning notebook [here](https://github.com/merveenoyan/smol-vision/blob/main/Idefics_FT.ipynb). 
 
-More information needed
+## Usage
 
-## Training and evaluation data
+You can load and use this model as follows.
 
-More information needed
+```python
 
-## Training procedure
+from transformers import Idefics3ForConditionalGeneration, AutoProcessor
 
-### Training hyperparameters
+peft_model_id = "merve/idefics3llama-vqav2"
+base_model_id = "HuggingFaceM4/Idefics3-8B-Llama3"
+processor = AutoProcessor.from_pretrained(base_model_id)
+model = Idefics3ForConditionalGeneration.from_pretrained(base_model_id)
+model.load_adapter(peft_model_id).to("cuda")
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0001
-- train_batch_size: 4
-- eval_batch_size: 8
-- seed: 42
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 32
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 50
-- num_epochs: 1
+```
 
-### Training results
+This model was conditioned on a prompt "Answer briefly.".
 
+```python
+from PIL import Image
+import requests
+from transformers.image_utils import load_image
 
+DEVICE = "cuda"
 
-### Framework versions
+image = load_image("https://huggingface.co/spaces/merve/OWLSAM2/resolve/main/buddha.JPG")
 
-- PEFT 0.12.0
-- Transformers 4.45.0.dev0
-- Pytorch 2.3.1+cu121
-- Datasets 2.19.1
-- Tokenizers 0.19.1
+
+messages = [
+          {
+              "role": "user",
+              "content": [
+                  {"type": "text", "text": "Answer briefly."},
+                  {"type": "image"},
+                  {"type": "text", "text": "Which country is this located in?"}
+              ]
+          }
+      ]
+
+text = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(text=text, images=image, return_tensors="pt", padding=True).to("cuda")
+```
+
+We can infer.
+
+```python
+generated_ids = model.generate(**inputs, max_new_tokens=500)
+generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
+print(generated_texts)
+
+##['User: Answer briefly.<row_1_col_1><row_1_col_2><row_1_col_3><row_1_col_4>\n<row_2_col_1>
+# <row_2_col_2><row_2_col_3><row_2_col_4>\n<row_3_col_1><row_3_col_2><row_3_col_3>
+# <row_3_col_4>\n\n<global-img>Which country is this located in?\nAssistant: thailand\nAssistant: thailand']
+```