vetgpt-2-7b: A Veterinary Medicine Fine-Tuned Language Model

vetgpt-2-7b is a specialized conversational AI model derived from Meta AI's meta-llama/Llama-2-7b-hf, meticulously fine-tuned using the Low-Rank Adaptation (LoRA) technique to cater to the unique needs of veterinary medicine. Built upon the robust foundation of the Llama-2 family, this model has been tailored to assist veterinarians, animal health professionals, researchers, and enthusiasts by providing domain-specific insights into animal care, diagnostics, treatment recommendations, and veterinary education. With its 7 billion parameters, vetgpt-2-7b leverages the power of advanced natural language processing to deliver precise, context-aware responses in the field of veterinary science.

Developed with a focus on practical utility, this model aims to bridge the gap between cutting-edge AI technology and real-world veterinary applications. Whether you're a practicing veterinarian seeking quick guidance on a complex case or a student studying animal pathology, vetgpt-2-7b offers a reliable and knowledgeable assistant tailored to your needs.

Model Overview

Base Model

vetgpt-2-7b is built upon meta-llama/Llama-2-7b-hf, a highly efficient and performant member of the Llama-2 family developed by Meta AI. The original Llama-2 models were designed for research and commercial use, excelling in natural language understanding and generation across a wide range of tasks. The -hf variant, optimized for compatibility with the Hugging Face ecosystem, serves as the backbone for this fine-tuned adaptation.

Fine-Tuning Methodology

To adapt the model for veterinary medicine, we employed LoRA, a parameter-efficient fine-tuning approach that modifies only a small subset of the model's weights. This technique allowed us to retain the general-purpose capabilities of the base model while injecting specialized knowledge from the veterinary domain. The LoRA configuration includes:

Rank (r): 8
LoRA Alpha: 16
Dropout: 0.1
Target Modules: q_proj, v_proj

This setup ensures computational efficiency while achieving significant improvements in domain-specific performance.

Training Data

The fine-tuning process utilized a custom dataset, dataset_cleaned_ALL_1700.jsonl, comprising approximately 1700 carefully curated examples. This dataset includes:

Instructions: Veterinary-related prompts and queries (e.g., "How to treat a dog's persistent cough?").
Inputs: Contextual information or case descriptions provided by veterinary professionals.
Outputs: Detailed, accurate responses grounded in veterinary science.

The data was sourced from a variety of veterinary resources, including clinical case studies, treatment protocols, and educational materials, ensuring a comprehensive representation of the field. While the dataset is primarily in Turkish and English, the model's multilingual capabilities allow it to handle queries in both languages effectively.

Training Environment

The model was trained on Google Colab using an NVIDIA A100 GPU, a state-of-the-art accelerator with 40 GB of HBM3 memory. To optimize memory usage and training speed, we applied 8-bit quantization via BitsAndBytesConfig and leveraged bfloat16 precision, both fully supported by the A100's hardware. The training process spanned 3 epochs, balancing performance gains with computational efficiency.

Technical Specifications

Parameter Count: 7 billion
Architecture: Auto-regressive transformer with grouped-query attention (GQA), as inherited from Llama-2.
Context Length: 256 tokens (configurable up to 512 or more with A100's memory capacity).
Quantization: 8-bit integer weights for memory efficiency.
Precision: bfloat16 for mixed-precision training and inference.
Vocabulary Size: 32,000 tokens, consistent with the Llama-2 tokenizer.
Training Duration: Approximately [insert duration after training, e.g., "6 hours"] on an A100 GPU.

The model retains the efficiency and scalability of the original Llama-2 design while incorporating veterinary-specific adaptations, making it both powerful and practical for real-world use.

Purpose and Capabilities

vetgpt-2-7b was developed to serve as a dedicated assistant for the veterinary community. Unlike general-purpose language models, it excels in understanding and generating responses tailored to animal health and welfare. Its key capabilities include:

Diagnostic Support: Assisting with symptom analysis and suggesting potential diagnoses based on veterinary knowledge.
Treatment Recommendations: Providing guidance on medications, dosages, and care protocols for various animal species.
Educational Tool: Supporting veterinary students and professionals with explanations of complex concepts, case studies, and best practices.
Multilingual Interaction: Handling queries in Turkish and English, with potential for expansion to other languages depending on the dataset.

While the base Llama-2-7b-chat-hf model was optimized for conversational tasks with an emphasis on safety and helpfulness, vetgpt-2-7b shifts this focus toward veterinary precision and utility, making it an invaluable resource for professionals working with animals.

Performance

The original Llama-2-7b-chat model was pre-trained on 2 trillion tokens from publicly available internet texts and fine-tuned with over 1 million human-annotated examples using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF). It outperformed many open-source chat models and approached the capabilities of proprietary systems like ChatGPT in general conversational benchmarks.

vetgpt-2-7b builds on this foundation but shifts the performance focus to veterinary medicine. While it retains strong general language skills, its fine-tuning on a targeted dataset ensures superior accuracy and relevance in veterinary contexts. Preliminary evaluations (based on the training data) suggest that it surpasses the base model in tasks such as:

Answering domain-specific questions (e.g., "What antibiotics are safe for cats with kidney issues?").
Generating coherent and detailed veterinary care instructions.

Formal benchmarks are pending, but users are encouraged to test the model with their own veterinary datasets and share results to further refine its capabilities.

Requirements

transformers>=4.35.0 torch>=2.0.0 bitsandbytes (for 8-bit quantization) GPU with CUDA support recommended (e.g., NVIDIA A100, T4, or higher). License vetgpt-2-7b is derived from meta-llama/Llama-2-7b-hf and is therefore subject to the Meta AI Llama 2 License. Users must review and accept this license before downloading or deploying the model. The license permits both research and commercial use under specific conditions, but redistribution of the base model weights is restricted. The fine-tuned LoRA adapters are shared under the same terms, with no additional restrictions unless specified by the veterinary dataset's ownership rights (if applicable).

Limitations and Disclaimer

While vetgpt-2-7b is a powerful tool for veterinary applications, it is not without limitations:

Domain Specificity: Its performance is optimized for veterinary medicine and may not match general-purpose models in unrelated tasks. Data Dependency: The model's knowledge is limited to the dataset_cleaned_ALL_1700.jsonl dataset and the pre-training of Llama-2. It may not cover rare or emerging veterinary conditions absent from this data. No Real-Time Updates: As a static model, it cannot access real-time veterinary research or updates post-training (March 24, 2025 cutoff assumed). Disclaimer: This model is intended as an assistive tool for veterinary professionals and should not replace qualified veterinarians. Outputs are provided "AS IS" and should not be solely relied upon for critical medical decisions involving animal health. Always consult a licensed veterinarian for diagnosis and treatment.

Acknowledgments

This project builds upon the groundbreaking work of Meta AI's Llama-2 team, whose efforts in developing efficient and performant language models have enabled domain-specific adaptations like vetgpt-2-7b. We also acknowledge the Hugging Face community for providing the tools and infrastructure to fine-tune and share this model seamlessly.

Contact and Contributions

For feedback, questions, or contributions, please reach out via the Hugging Face repository discussion tab at melihyuksel01/vetgpt-2-7b. We welcome suggestions for improving the model, expanding the dataset, or adding new veterinary use cases. If you have additional veterinary data or benchmarks to share, feel free to contribute to make vetgpt-2-7b an even more valuable resource for the global veterinary community.

Future Directions

We envision vetgpt-2-7b as the foundation for a series of veterinary AI tools. Potential future enhancements include:

Expanding the dataset with more diverse veterinary cases and languages. Integrating multimodal capabilities (e.g., analyzing veterinary images or lab reports). Fine-tuning with RLHF to improve response helpfulness and safety in veterinary contexts. Stay tuned for updates as we continue to refine and grow this model!

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 32
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 2
total_train_batch_size: 64
optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.4817	1.0	108	0.5623
0.4648	2.0	216	0.5531
0.4478	2.9767	321	0.5512

Framework versions

PEFT 0.14.0
Transformers 4.49.0
Pytorch 2.6.0+cu124
Datasets 3.4.1
Tokenizers 0.21.1

melihyuksel01
/

vetgpt-2-7b