--- license: apache-2.0 base_model: BioMistral/BioMistral-7B tags: - medical - biomedical - healthcare - question-answering - lora - peft - medquad - biomistral - instruction-tuning language: - en datasets: - jpmiller/medquad library_name: peft pipeline_tag: text-generation model_type: mistral --- # BioMistral-7B LoRA Fine-tuned on MedQuAD This model is a LoRA (Low-Rank Adaptation) fine-tuned version of [BioMistral/BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B) for medical question answering, trained on the MedQuAD dataset from Kaggle. ## Model Description - **Base Model**: BioMistral/BioMistral-7B - **Fine-tuning Method**: LoRA (Low-Rank Adaptation) - **Model Type**: Causal Language Model - **Training Dataset**: MedQuAD (Medical Question Answering Dataset) - **Domain**: Medical/Biomedical - **Language**: English - **License**: Apache 2.0 (inherited from base model) ## Dataset Information ### MedQuAD Dataset - **Source**: [MedQuAD on Kaggle](https://www.kaggle.com/datasets/jpmiller/medquad) - **Full Name**: Medical Question Answering Dataset - **Description**: A collection of medical questions and answers from trusted medical sources - **Training Examples**: 14,770 question-answer pairs - **Validation Examples**: 1,642 question-answer pairs - **Format**: Instruction-Input-Output triplets for medical Q&A ### Data Sources (MedQuAD) The MedQuAD dataset contains medical information from various authoritative sources including: - National Institutes of Health (NIH) - National Cancer Institute (NCI) - National Institute of Mental Health (NIMH) - Centers for Disease Control and Prevention (CDC) - And other trusted medical organizations ## Training Details ### Training Configuration - **Training Steps**: 2,772 (3 epochs) - **Batch Size**: 2 per device - **Gradient Accumulation**: 8 steps - **Effective Batch Size**: 16 - **Learning Rate**: 2e-4 - **Warmup Steps**: 100 - **Max Sequence Length**: 512 - **Optimizer**: AdamW - **Precision**: FP16 ### LoRA Configuration - **LoRA Rank (r)**: 16 - **LoRA Alpha**: 32 - **LoRA Dropout**: 0.1 - **Target Modules**: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj - **Trainable Parameters**: ~0.1% of total parameters ### Training Results | Step | Training Loss | Validation Loss | |------|---------------|-----------------| | 500 | 0.8277 | 0.8332 | | 1000 | 0.5424 | 0.8180 | | 1500 | 0.5696 | 0.7986 | | 2000 | 0.3430 | 0.8451 | | 2500 | 0.3184 | 0.8488 | **Final Validation Loss**: 0.8488 ## Installation ```bash pip install transformers peft torch accelerate bitsandbytes ``` ## Usage ### Using LoRA Adapters (Recommended) ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch # Load base model base_model_name = "BioMistral/BioMistral-7B" tokenizer = AutoTokenizer.from_pretrained(base_model_name) base_model = AutoModelForCausalLM.from_pretrained( base_model_name, device_map="auto", torch_dtype=torch.float16 ) # Load LoRA adapters lora_model_name = "ayuwal12/biomistral-7b-lora-adapters" model = PeftModel.from_pretrained(base_model, lora_model_name) # Set pad token if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token def generate_medical_response(question, context="", max_length=256): if context.strip(): prompt = f"### Instruction:\n{question}\n\n### Input:\n{context}\n\n### Response:\n" else: prompt = f"### Instruction:\n{question}\n\n### Response:\n" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_length, temperature=0.7, do_sample=True, pad_token_id=tokenizer.eos_token_id ) response = tokenizer.decode(outputs[0], skip_special_tokens=True) return response.split("### Response:\n")[-1].strip() # Example usage response = generate_medical_response("What are the symptoms of hypertension?") print(response) ``` ## Example Medical Questions ### General Medical Questions ```python question = "What is hypertension and how is it diagnosed?" response = generate_medical_response(question) ``` ### Symptoms and Conditions ```python question = "What are the common symptoms of type 2 diabetes?" response = generate_medical_response(question) ``` ### Treatment and Management ```python question = "How is high blood pressure treated?" response = generate_medical_response(question) ``` ### With Medical Context ```python question = "What should I know about this condition?" context = "Patient has been diagnosed with stage 1 hypertension" response = generate_medical_response(question, context) ``` ## Model Performance - **Training Loss**: Decreased from 0.83 to 0.32 over 3 epochs - **Validation Loss**: Stabilized around 0.85 - **Convergence**: Model shows good learning with minimal overfitting - **Memory Efficiency**: Uses ~0.1% trainable parameters via LoRA - **Domain**: Specialized for medical question answering ## Capabilities This model excels at: - ✅ **Medical Question Answering**: Trained specifically on medical Q&A pairs - ✅ **Disease Information**: Provides information about various medical conditions - ✅ **Symptom Analysis**: Explains symptoms and their significance - ✅ **Treatment Overview**: Discusses general treatment approaches - ✅ **Medical Terminology**: Understands and explains medical terms ## Limitations - Based on BioMistral-7B, inherits its limitations - Trained on MedQuAD dataset, may not cover all medical domains equally - **Not for diagnosis**: Cannot replace professional medical evaluation - **Information only**: Provides general medical information, not personalized advice - May not have the most recent medical research (depends on training data cutoff) ## Intended Use This model is designed for: - 📚 **Educational purposes** in medical and healthcare domains - 🔬 **Research applications** in biomedical NLP - 💡 **Medical information retrieval** systems - 🏥 **Healthcare chatbots** (with appropriate disclaimers) - 📖 **Medical knowledge base** applications ## Ethical Considerations & Medical Disclaimer ⚠️ **IMPORTANT MEDICAL DISCLAIMER**: - This model is for **educational and research purposes only** - **NOT for medical diagnosis** or treatment decisions - Always consult qualified healthcare professionals for medical advice - AI-generated medical content may contain errors or biases - Do not use this model for emergency medical situations - Individual medical conditions require personalized professional care ## Dataset Citation ```bibtex @misc{medquad, title={MedQuAD: Medical Question Answering Dataset}, author={Ben Abacha, Asma and Mrabet, Yassine and Zhang, Yuhao and Shivade, Chaitanya and Langlotz, Curtis and Demner-Fushman, Dina}, year={2019}, howpublished={Available on Kaggle: https://www.kaggle.com/datasets/jpmiller/medquad} } ``` ## Model Citation If you use this model, please cite: ```bibtex @misc{biomistral-medquad-lora, title={BioMistral-7B LoRA Fine-tuned on MedQuAD}, author={Ayuwal}, year={2024}, howpublished={https://huggingface.co/ayuwal12/biomistral-7b-lora-adapters}, } ``` ## Acknowledgments - **Base model**: [BioMistral/BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B) - **Training dataset**: [MedQuAD](https://www.kaggle.com/datasets/jpmiller/medquad) - **LoRA implementation**: [PEFT](https://github.com/huggingface/peft) - **Training framework**: [Transformers](https://github.com/huggingface/transformers) - **Original MedQuAD authors**: Ben Abacha et al. ## Contact For questions or issues, please open an issue on the model repository. --- *This model was trained on the MedQuAD dataset and is intended for educational and research purposes in the medical domain. Always consult healthcare professionals for medical advice.*