diff for compatibility

README.md

@@ -22,155 +22,23 @@ datasets:
 pipeline_tag: image-text-to-text
 ---
 
-# Bio-Medical-MultiModal-Llama-3-8B-V1
+<!-- header start -->
+<p align="center">
+  <img src="https://huggingface.co/datasets/FriendliAI/documentation-images/resolve/main/model-card-assets/friendliai.png" width="100%" alt="FriendliAI Logo">
+</p>
+<!-- header end -->
 
-![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/zPMUugzfOiwTiRw88jm7T.jpeg)
 
-This model is a fine-tuned Multimodal version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on our custom "BioMedData" text and image datasets.
+# ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1
 
-## Model details
+* Model creator: [ContactDoctor](https://huggingface.co/ContactDoctor)
+* Original model: [Bio-Medical-MultiModal-Llama-3-8B-V1](https://huggingface.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1)
 
-Model Name: Bio-Medical-MultiModal-Llama-3-8B-V1
+## Differences
 
-Base Model: Llama-3-8B-Instruct
-
-Parameter Count: 8 billion
-
-Training Data: Custom high-quality biomedical text and image dataset
-
-Number of Entries in Dataset: 500,000+
-
-Dataset Composition: The dataset comprises of text and image, both synthetic and manually curated samples, ensuring a diverse and comprehensive coverage of biomedical knowledge.
-
-## Model description
-
-Bio-Medical-MultiModal-Llama-3-8B-V1 is a specialized large language model designed for biomedical applications. It is finetuned from the Llama-3-8B-Instruct model using a custom dataset containing over 500,000 diverse entries. These entries include a mix of synthetic and manually curated data, ensuring high quality and broad coverage of biomedical topics.
-
-The model is trained to understand and generate text related to various biomedical fields, making it a valuable tool for researchers, clinicians, and other professionals in the biomedical domain.
+* Added missing eos_token (`<|eot_id|>`) to config.json.
 
 ## License
 
-This model is licensed under the [Bio-Medical-MultiModal-Llama-3-8B-V1 (Non-Commercial Use Only)](./LICENSE). Please review the terms and conditions before using the model.
-
-## Quick Demo
-
-<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/653f5b93cd52f288490edc83/RpdFKs3mBY9ZIxvUUWOKc.mp4"></video>
-
-## How to use
-
-import torch
-
-from PIL import Image
-
-from transformers import AutoModel, AutoTokenizer,BitsAndBytesConfig
-
-bnb_config = BitsAndBytesConfig(
-    load_in_4bit=True,
-    bnb_4bit_quant_type="nf4",
-    bnb_4bit_use_double_quant=True,
-    bnb_4bit_compute_dtype=torch.float16,
-)
-
-
-model = AutoModel.from_pretrained(
-	"ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1",
-	quantization_config=bnb_config,
-	device_map="auto",
-	torch_dtype=torch.float16,
-	trust_remote_code=True,
-	attn_implementation="flash_attention_2",
-)
-
-
-tokenizer = AutoTokenizer.from_pretrained("ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1", trust_remote_code=True)
-
-
-image = Image.open("Path to Your image").convert('RGB')
-
-question = 'Give the modality, organ, analysis, abnormalities (if any), treatment (if abnormalities are present)?'
-
-msgs = [{'role': 'user', 'content': [image, question]}]
-
-res = model.chat(
-    image=image,
-    msgs=msgs,
-    tokenizer=tokenizer,
-    sampling=True,
-    temperature=0.95,
-    stream=True
-)
-
-generated_text = ""
-
-for new_text in res:
-    generated_text += new_text
-    print(new_text, flush=True, end='')
-
-
-> Sample Response
-
-The modality is Magnetic Resonance Imaging (MRI), the organ being analyzed is the cervical spine, and there are no abnormalities present in the image.
-
-## Intended uses & limitations
-
-Bio-Medical-MultiModal-Llama-3-8B-V1 is intended for a wide range of applications within the biomedical field, including:
-
-1. Research Support: Assisting researchers in literature review and data extraction from biomedical texts.
-2. Clinical Decision Support: Providing information to support clinical decision-making processes.
-3. Educational Tool: Serving as a resource for medical students and professionals seeking to expand their knowledge base.
-
-## Limitations and Ethical Considerations
-
-Bio-Medical-MultiModal-Llama-3-8B-V1 performs well in various biomedical NLP tasks, users should be aware of the following limitations:
-
-1. Biases: The model may inherit biases present in the training data. Efforts have been made to curate a balanced dataset, but some biases may persist.
-2. Accuracy: The model's responses are based on patterns in the data it has seen and may not always be accurate or up-to-date. Users should verify critical information from reliable sources.
-3. Ethical Use: The model should be used responsibly, particularly in clinical settings where the stakes are high. It should complement, not replace, professional judgment and expertise.
-
-
-## Training and evaluation
-
-Bio-Medical-MultiModal-Llama-3-8B-V1 was trained using NVIDIA H100 GPU's, which provides the computational power necessary for handling large-scale data and model parameters efficiently. Rigorous evaluation protocols have been implemented to benchmark its performance against similar models, ensuring its robustness and reliability in real-world applications.
-
-The model was trained using **MiniCPM**, which allowed us to efficiently handle the multimodal data. MiniCPM provided the ability to process and learn from visual information.
-
-### Contact Information
-
-For further information, inquiries, or issues related to Biomed-LLM, please contact:
-
-Email: [email protected]
-
-Website: https://www.contactdoctor.in
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 4
-- eval_batch_size: 4
-- Number of epochs: 3
-- seed: 42
-- gradient_accumulation_steps: 4
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.03
-- mixed_precision_training: Native AMP
-
-### Framework versions
-
-- PEFT 0.11.0
-- Transformers 4.40.2
-- Pytorch 2.1.2
-- Datasets 2.19.1
-- Tokenizers 0.19.1
-
-### Citation
-
-If you use Bio-Medical-MultiModal-Llama-3-8B-V1 in your research or applications, please cite it as follows:
+Refer to the license of the original model card.
 
-@misc{ContactDoctor_MEDLLM,
-  author = ContactDoctor,
-  title = {Bio-Medical-MultiModal-Llama-3-8B-V1: A High-Performance Biomedical Multimodal LLM},
-  year = {2024},
-  howpublished = {https://huggingface.co/ContactDoctor/Bio-Medical-MultiModal-Llama-3-8B-V1},
-}

config.json

@@ -13,7 +13,10 @@
   "batch_vision_input": true,
   "bos_token_id": 128000,
   "drop_vision_last_layer": false,
-  "eos_token_id": 128001,
+  "eos_token_id": [
+    128001,
+    128009
+  ],
   "hidden_act": "silu",
   "hidden_size": 4096,
   "image_size": 448,
@@ -52,4 +55,4 @@
     "patch_size": 14
   },
   "vocab_size": 128256
-}
+}

generation_config.json

@@ -3,4 +3,4 @@
   "bos_token_id": 128000,
   "eos_token_id": 128001,
   "transformers_version": "4.41.2"
-}
+}

tokenizer_config.json

@@ -2058,7 +2058,7 @@
   "bos_token": "<|begin_of_text|>",
   "chat_template": "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}",
   "clean_up_tokenization_spaces": true,
-  "eos_token": "<|end_of_text|>",
+  "eos_token": "<|eot_id|>",
   "model_input_names": [
     "input_ids",
     "attention_mask"
@@ -2069,4 +2069,4 @@
   "tokenizer_class": "MiniCPMVTokenizerFast",
   "truncation_side": "right",
   "unk_token": "<unk>"
-}
+}