README.md

---
license: apache-2.0
base_model: BioMistral/BioMistral-7B
tags:
- medical
- biomedical
- healthcare
- question-answering
- lora
- peft
- medquad
- biomistral
- instruction-tuning
language:
- en
datasets:
- jpmiller/medquad
library_name: peft
pipeline_tag: text-generation
model_type: mistral
---

# BioMistral-7B LoRA Fine-tuned on MedQuAD

This model is a LoRA (Low-Rank Adaptation) fine-tuned version of [BioMistral/BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B) for medical question answering, trained on the MedQuAD dataset from Kaggle.

## Model Description

- **Base Model**: BioMistral/BioMistral-7B
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
- **Model Type**: Causal Language Model
- **Training Dataset**: MedQuAD (Medical Question Answering Dataset)
- **Domain**: Medical/Biomedical
- **Language**: English
- **License**: Apache 2.0 (inherited from base model)

## Dataset Information

### MedQuAD Dataset
- **Source**: [MedQuAD on Kaggle](https://www.kaggle.com/datasets/jpmiller/medquad)
- **Full Name**: Medical Question Answering Dataset
- **Description**: A collection of medical questions and answers from trusted medical sources
- **Training Examples**: 14,770 question-answer pairs
- **Validation Examples**: 1,642 question-answer pairs
- **Format**: Instruction-Input-Output triplets for medical Q&A

### Data Sources (MedQuAD)
The MedQuAD dataset contains medical information from various authoritative sources including:
- National Institutes of Health (NIH)
- National Cancer Institute (NCI)
- National Institute of Mental Health (NIMH)
- Centers for Disease Control and Prevention (CDC)
- And other trusted medical organizations

## Training Details

### Training Configuration
- **Training Steps**: 2,772 (3 epochs)
- **Batch Size**: 2 per device
- **Gradient Accumulation**: 8 steps
- **Effective Batch Size**: 16
- **Learning Rate**: 2e-4
- **Warmup Steps**: 100
- **Max Sequence Length**: 512
- **Optimizer**: AdamW
- **Precision**: FP16

### LoRA Configuration
- **LoRA Rank (r)**: 16
- **LoRA Alpha**: 32
- **LoRA Dropout**: 0.1
- **Target Modules**: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj
- **Trainable Parameters**: ~0.1% of total parameters

### Training Results
| Step | Training Loss | Validation Loss |
|------|---------------|-----------------|
| 500  | 0.8277        | 0.8332          |
| 1000 | 0.5424        | 0.8180          |
| 1500 | 0.5696        | 0.7986          |
| 2000 | 0.3430        | 0.8451          |
| 2500 | 0.3184        | 0.8488          |

**Final Validation Loss**: 0.8488

## Installation

```bash
pip install transformers peft torch accelerate bitsandbytes
```

## Usage

### Using LoRA Adapters (Recommended)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model_name = "BioMistral/BioMistral-7B"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    torch_dtype=torch.float16
)

# Load LoRA adapters
lora_model_name = "ayuwal12/biomistral-7b-lora-adapters"
model = PeftModel.from_pretrained(base_model, lora_model_name)

# Set pad token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

def generate_medical_response(question, context="", max_length=256):
    if context.strip():
        prompt = f"### Instruction:\n{question}\n\n### Input:\n{context}\n\n### Response:\n"
    else:
        prompt = f"### Instruction:\n{question}\n\n### Response:\n"
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_length,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:\n")[-1].strip()

# Example usage
response = generate_medical_response("What are the symptoms of hypertension?")
print(response)
```

## Example Medical Questions

### General Medical Questions
```python
question = "What is hypertension and how is it diagnosed?"
response = generate_medical_response(question)
```

### Symptoms and Conditions
```python
question = "What are the common symptoms of type 2 diabetes?"
response = generate_medical_response(question)
```

### Treatment and Management
```python
question = "How is high blood pressure treated?"
response = generate_medical_response(question)
```

### With Medical Context
```python
question = "What should I know about this condition?"
context = "Patient has been diagnosed with stage 1 hypertension"
response = generate_medical_response(question, context)
```

## Model Performance

- **Training Loss**: Decreased from 0.83 to 0.32 over 3 epochs
- **Validation Loss**: Stabilized around 0.85
- **Convergence**: Model shows good learning with minimal overfitting
- **Memory Efficiency**: Uses ~0.1% trainable parameters via LoRA
- **Domain**: Specialized for medical question answering

## Capabilities

This model excels at:
- ✅ **Medical Question Answering**: Trained specifically on medical Q&A pairs
- ✅ **Disease Information**: Provides information about various medical conditions
- ✅ **Symptom Analysis**: Explains symptoms and their significance
- ✅ **Treatment Overview**: Discusses general treatment approaches
- ✅ **Medical Terminology**: Understands and explains medical terms

## Limitations

- Based on BioMistral-7B, inherits its limitations
- Trained on MedQuAD dataset, may not cover all medical domains equally
- **Not for diagnosis**: Cannot replace professional medical evaluation
- **Information only**: Provides general medical information, not personalized advice
- May not have the most recent medical research (depends on training data cutoff)

## Intended Use

This model is designed for:
- 📚 **Educational purposes** in medical and healthcare domains
- 🔬 **Research applications** in biomedical NLP
- 💡 **Medical information retrieval** systems
- 🏥 **Healthcare chatbots** (with appropriate disclaimers)
- 📖 **Medical knowledge base** applications

## Ethical Considerations & Medical Disclaimer

⚠️ **IMPORTANT MEDICAL DISCLAIMER**:
- This model is for **educational and research purposes only**
- **NOT for medical diagnosis** or treatment decisions
- Always consult qualified healthcare professionals for medical advice
- AI-generated medical content may contain errors or biases
- Do not use this model for emergency medical situations
- Individual medical conditions require personalized professional care

## Dataset Citation

```bibtex
@misc{medquad,
  title={MedQuAD: Medical Question Answering Dataset},
  author={Ben Abacha, Asma and Mrabet, Yassine and Zhang, Yuhao and Shivade, Chaitanya and Langlotz, Curtis and Demner-Fushman, Dina},
  year={2019},
  howpublished={Available on Kaggle: https://www.kaggle.com/datasets/jpmiller/medquad}
}
```

## Model Citation

If you use this model, please cite:

```bibtex
@misc{biomistral-medquad-lora,
  title={BioMistral-7B LoRA Fine-tuned on MedQuAD},
  author={Ayuwal},
  year={2024},
  howpublished={https://huggingface.co/ayuwal12/biomistral-7b-lora-adapters},
}
```

## Acknowledgments

- **Base model**: [BioMistral/BioMistral-7B](https://huggingface.co/BioMistral/BioMistral-7B)
- **Training dataset**: [MedQuAD](https://www.kaggle.com/datasets/jpmiller/medquad)
- **LoRA implementation**: [PEFT](https://github.com/huggingface/peft)
- **Training framework**: [Transformers](https://github.com/huggingface/transformers)
- **Original MedQuAD authors**: Ben Abacha et al.

## Contact

For questions or issues, please open an issue on the model repository.

---

*This model was trained on the MedQuAD dataset and is intended for educational and research purposes in the medical domain. Always consult healthcare professionals for medical advice.*