metadata

license: llama3.2
datasets:
  - bigbio/med_qa
language:
  - en
base_model:
  - meta-llama/Llama-3.2-1B-Instruct
pipeline_tag: text-generation
tags:
  - medical
  - SandLogic
  - Meta
  - Conversational

SandLogic Technology - Quantized Llama-3.2-1B-Instruct-Medical-GGUF

Model Description

We have quantized the Llama-3.2-1B-Instruct-Medical-GGUF model into two variants:

Q5_KM
Q4_KM

These quantized models offer improved efficiency while maintaining performance in medical-related tasks.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

Base Model: Meta Llama 3.2 1B Instruct
Developer: Meta (base model)
Model Type: Multilingual large language model (LLM)
Architecture: Auto-regressive language model with optimized transformer architecture
Parameters: 1 billion
Training Approach: Supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF)

Fine-tuning Details

Dataset: bigbio/med_qa
Languages: English, simplified Chinese, and traditional Chinese
Dataset Size:
- English: 12,723 questions
- Simplified Chinese: 34,251 questions
- Traditional Chinese: 14,123 questions
Data Type: Free-form multiple-choice OpenQA for medical problems, collected from professional medical board exams

Model Capabilities

This model is optimized for medical-related dialogue and tasks, including:

Answering medical questions
Summarizing medical information
Assisting with medical problem-solving

Intended Use in Medical Domain

Medical Education: Assisting medical students in exam preparation and learning
Clinical Decision Support: Providing quick references for healthcare professionals
Patient Education: Explaining medical concepts in simple terms for patients
Medical Literature Review: Summarizing and extracting key information from medical texts
Differential Diagnosis: Assisting in generating potential diagnoses based on symptoms
Medical Coding: Aiding in the accurate coding of medical procedures and diagnoses
Drug Information: Providing information on medications, their uses, and potential interactions
Medical Translation: Assisting with medical translations across supported languages

Quantized Variants

Q5_KM: 5-bit quantization using the KM method
Q4_KM: 4-bit quantization using the KM method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Usage

pip install llama-cpp-python

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Llama-3.2-1B-Medical_Q4_KM.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm.create_chat_completion(
    messages =[
    {
        "role": "system",
        "content": """ You are a helpful, respectful and honest medical assistant. Yu are developed by SandLogic Technologies 
            Always answer as helpfully as possible, while being safe. 
            Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. 
            Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. 
            If you don’t know the answer to a question, please don’t share false information."""
            
        ,
    },
    {"role": "user", "content": "I have been experiencing a persistent cough for the last two weeks, along with a mild fever and fatigue. What could be the possible causes of these symptoms?"},
]
)

print(output["choices"][0]['message']['content'])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Llama-3.2-1B-Instruct-Medical-GGUF",
    filename="*Llama-3.2-1B-Medical_Q5_KM.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

Ethical Considerations and Limitations

This model is not a substitute for professional medical advice, diagnosis, or treatment
Users should be aware of potential biases in the training data
The model's knowledge cutoff date may limit its awareness of recent medical developments

Acknowledgements

We thank Meta for developing the original Llama-3.2-1B-Instruct model and the creators of the bigbio/med_qa dataset. Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at [email protected] or visit our support page.

Explore More

For any inquiries or support, please contact us at [email protected] or visit our support page.