metadata

license: apache-2.0
base_model:
  - mistralai/Voxtral-Small-24B-2507
tags:
  - mistral
  - quantized
  - 4bit
  - llm
  - language-model
  - transformers
  - mlx

VincentGOURBIN/voxtral-small-4bit-mixed

This is a 4-bit quantized version of the mistralai/Voxtral-Small-24B-2507 language model.
It is provided in standard Hugging Face Transformers format and compatible with mlx.voxtral.

🔧 About this model

Base model: mistralai/Voxtral-Small-24B-2507
Quantization: 4-bit mixed precision
Format: Transformers-compatible (safetensors), usable with MLX and Hugging Face

🙏 Acknowledgments

Huge thanks to:

Mistral AI for releasing the original Voxtral-Small model
mlx-voxtral for the quantization tooling and MLX support

This work is a quantized derivative of mistralai/Voxtral-Small-24B-2507, made easier by the amazing work of the voxtral project.

🚀 Usage

🤗 With Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "VincentGOURBIN/voxtral-small-4bit-mixed"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "What is the capital of France?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))