ik-ram28/MedMistral-SFT-7B

Model Description

MedMistral-SFT-7B is a French medical language model based on Mistral-7B-v0.1, adapted for medical domain applications through direct Supervised Fine-Tuning (SFT).

Model Details

• Model Type: Causal Language Model
• Base Model: Mistral-7B-v0.1
• Language: French
• Domain: Medical/Healthcare
• License: Apache 2.0
• Paper: Adaptation des connaissances médicales pour les grands modèles de langue : Stratégies et analyse comparative

Training Details

Supervised Fine-Tuning (SFT)

• Dataset: 30K French medical question-answer pairs
  • 10K native French medical questions
  • 10K translated medical questions from English resources
  • 10K generated questions from French medical texts
• Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
• Training Duration: 10 epochs
• Rank: 16
• Alpha: 16
• LoRA Dropout: 0.05
• Learning Rate: 2e-5
• Batch Size: 4
• Gradient Accumulation: 2
• Optimizer: AdamW
• Scheduler: Cosine
• Target Modules: QKVOGUD

Computational Impact

• Training Time: 53 hours
• Carbon Emissions: 2.62 kgCO2e

Ethical Considerations

• Medical Accuracy: This model is for research and educational purposes only. All outputs should be verified by qualified medical professionals
• Bias: Training data may contain biases present in medical literature and online medical resources

Citation

If you use this model, please cite:

Contact

For questions about this model, please contact: [email protected]