|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- medical |
|
|
- llama |
|
|
- unsloth |
|
|
- qlora |
|
|
- finetuned |
|
|
- chatbot |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- custom-medical-qa |
|
|
base_model: ContactDoctor/Bio-Medical-Llama-3-8B |
|
|
model_creator: khalednabawi11 |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
|
|
|
# Bio-Medical LLaMA 3 8B - Fine-Tuned |
|
|
|
|
|
🚀 **Fine-tuned version of [ContactDoctor/Bio-Medical-Llama-3-8B](https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B) using Unsloth for enhanced medical Q&A capabilities.** |
|
|
|
|
|
## 📌 Model Details |
|
|
|
|
|
- **Model Name:** Bio-Medical LLaMA 3 8B - Fine-Tuned |
|
|
- **Base Model:** ContactDoctor/Bio-Medical-Llama-3-8B |
|
|
- **Fine-Tuning Method:** QLoRA with Unsloth |
|
|
- **Domain:** Medical Question Answering |
|
|
- **Dataset:** Medical Q&A dataset (MQA.json) |
|
|
|
|
|
## 🛠️ Training Configuration |
|
|
|
|
|
- **Epochs:** 4 |
|
|
- **Batch Size:** 2 |
|
|
- **Gradient Accumulation:** 4 |
|
|
- **Learning Rate:** 2e-4 |
|
|
- **Optimizer:** AdamW (8-bit) |
|
|
- **Weight Decay:** 0.01 |
|
|
- **Warmup Steps:** 50 |
|
|
|
|
|
## 🔧 LoRA Parameters |
|
|
|
|
|
- **LoRA Rank (r):** 16 |
|
|
- **LoRA Alpha:** 16 |
|
|
- **LoRA Dropout:** 0 |
|
|
- **Bias:** None |
|
|
- **Target Layers:** |
|
|
- q_proj |
|
|
- k_proj |
|
|
- v_proj |
|
|
- o_proj |
|
|
- gate_proj |
|
|
- up_proj |
|
|
- down_proj |
|
|
- **Gradient Checkpointing:** Enabled (Unsloth) |
|
|
- **Random Seed:** 3407 |
|
|
|
|
|
## 🚀 Model Capabilities |
|
|
|
|
|
- Optimized for **low-memory inference** |
|
|
- Supports **long medical queries** |
|
|
- Efficient **parameter-efficient tuning (LoRA)** |
|
|
|
|
|
## 📊 Usage |
|
|
|
|
|
This model is suitable for **medical question answering**, **clinical chatbot applications**, and **biomedical research assistance**. |
|
|
|
|
|
## 🔗 References |
|
|
|
|
|
- [Unsloth Documentation](https://github.com/unslothai/unsloth) |
|
|
- [Hugging Face Transformers](https://huggingface.co/docs/transformers/index) |
|
|
|
|
|
--- |
|
|
💡 **Contributions & Feedback**: Open to collaboration! Feel free to reach out. |
|
|
|
