BioMistral-7B LoRA Fine-tuned on MedQuAD

This model is a LoRA (Low-Rank Adaptation) fine-tuned version of BioMistral/BioMistral-7B for medical question answering, trained on the MedQuAD dataset from Kaggle.

Model Description

Base Model: BioMistral/BioMistral-7B
Fine-tuning Method: LoRA (Low-Rank Adaptation)
Model Type: Causal Language Model
Training Dataset: MedQuAD (Medical Question Answering Dataset)
Domain: Medical/Biomedical
Language: English
License: Apache 2.0 (inherited from base model)

Dataset Information

MedQuAD Dataset

Source: MedQuAD on Kaggle
Full Name: Medical Question Answering Dataset
Description: A collection of medical questions and answers from trusted medical sources
Training Examples: 14,770 question-answer pairs
Validation Examples: 1,642 question-answer pairs
Format: Instruction-Input-Output triplets for medical Q&A

Data Sources (MedQuAD)

The MedQuAD dataset contains medical information from various authoritative sources including:

National Institutes of Health (NIH)
National Cancer Institute (NCI)
National Institute of Mental Health (NIMH)
Centers for Disease Control and Prevention (CDC)
And other trusted medical organizations

Training Details

Training Configuration

Training Steps: 2,772 (3 epochs)
Batch Size: 2 per device
Gradient Accumulation: 8 steps
Effective Batch Size: 16
Learning Rate: 2e-4
Warmup Steps: 100
Max Sequence Length: 512
Optimizer: AdamW
Precision: FP16

LoRA Configuration

LoRA Rank (r): 16
LoRA Alpha: 32
LoRA Dropout: 0.1
Target Modules: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj
Trainable Parameters: ~0.1% of total parameters

Training Results

Step	Training Loss	Validation Loss
500	0.8277	0.8332
1000	0.5424	0.8180
1500	0.5696	0.7986
2000	0.3430	0.8451
2500	0.3184	0.8488

Final Validation Loss: 0.8488

Installation

pip install transformers peft torch accelerate bitsandbytes

Usage

Using LoRA Adapters (Recommended)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model_name = "BioMistral/BioMistral-7B"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto",
    torch_dtype=torch.float16
)

# Load LoRA adapters
lora_model_name = "ayuwal12/biomistral-7b-lora-adapters"
model = PeftModel.from_pretrained(base_model, lora_model_name)

# Set pad token
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

def generate_medical_response(question, context="", max_length=256):
    if context.strip():
        prompt = f"### Instruction:\n{question}\n\n### Input:\n{context}\n\n### Response:\n"
    else:
        prompt = f"### Instruction:\n{question}\n\n### Response:\n"
    
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_length,
            temperature=0.7,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("### Response:\n")[-1].strip()

# Example usage
response = generate_medical_response("What are the symptoms of hypertension?")
print(response)

Example Medical Questions

General Medical Questions

question = "What is hypertension and how is it diagnosed?"
response = generate_medical_response(question)

Symptoms and Conditions

question = "What are the common symptoms of type 2 diabetes?"
response = generate_medical_response(question)

Treatment and Management

question = "How is high blood pressure treated?"
response = generate_medical_response(question)

With Medical Context

question = "What should I know about this condition?"
context = "Patient has been diagnosed with stage 1 hypertension"
response = generate_medical_response(question, context)

Model Performance

Training Loss: Decreased from 0.83 to 0.32 over 3 epochs
Validation Loss: Stabilized around 0.85
Convergence: Model shows good learning with minimal overfitting
Memory Efficiency: Uses ~0.1% trainable parameters via LoRA
Domain: Specialized for medical question answering

Capabilities

This model excels at:

✅ Medical Question Answering: Trained specifically on medical Q&A pairs
✅ Disease Information: Provides information about various medical conditions
✅ Symptom Analysis: Explains symptoms and their significance
✅ Treatment Overview: Discusses general treatment approaches
✅ Medical Terminology: Understands and explains medical terms

Limitations

Based on BioMistral-7B, inherits its limitations
Trained on MedQuAD dataset, may not cover all medical domains equally
Not for diagnosis: Cannot replace professional medical evaluation
Information only: Provides general medical information, not personalized advice
May not have the most recent medical research (depends on training data cutoff)

Intended Use

This model is designed for:

📚 Educational purposes in medical and healthcare domains
🔬 Research applications in biomedical NLP
💡 Medical information retrieval systems
🏥 Healthcare chatbots (with appropriate disclaimers)
📖 Medical knowledge base applications

Ethical Considerations & Medical Disclaimer

⚠️ IMPORTANT MEDICAL DISCLAIMER:

This model is for educational and research purposes only
NOT for medical diagnosis or treatment decisions
Always consult qualified healthcare professionals for medical advice
AI-generated medical content may contain errors or biases
Do not use this model for emergency medical situations
Individual medical conditions require personalized professional care

Dataset Citation

@misc{medquad,
  title={MedQuAD: Medical Question Answering Dataset},
  author={Ben Abacha, Asma and Mrabet, Yassine and Zhang, Yuhao and Shivade, Chaitanya and Langlotz, Curtis and Demner-Fushman, Dina},
  year={2019},
  howpublished={Available on Kaggle: https://www.kaggle.com/datasets/jpmiller/medquad}
}

Model Citation

If you use this model, please cite:

@misc{biomistral-medquad-lora,
  title={BioMistral-7B LoRA Fine-tuned on MedQuAD},
  author={Ayuwal},
  year={2024},
  howpublished={https://huggingface.co/ayuwal12/biomistral-7b-lora-adapters},
}

Acknowledgments

Base model: BioMistral/BioMistral-7B
Training dataset: MedQuAD
LoRA implementation: PEFT
Training framework: Transformers
Original MedQuAD authors: Ben Abacha et al.

Contact

For questions or issues, please open an issue on the model repository.

This model was trained on the MedQuAD dataset and is intended for educational and research purposes in the medical domain. Always consult healthcare professionals for medical advice.

ayuwal12
/

biomistral-7b-lora-adapters