README.md

---

base_model: mistralai/Mistral-Small-24B-Instruct-2501

---
This is a quantization of the [Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).

Mistral Small 3 (2501) is a cutting-edge 24B parameter model that redefines the small LLM category under 70B, offering state-of-the-art performance comparable to larger models. Designed for fast conversational AI, low-latency function calling, and expert fine-tuning, it excels in multilingual support, advanced reasoning, and structured output generation. Released under an Apache 2.0 license, Mistral Small 3 embodies a commitment to open-source AI, serving as a versatile foundation for both community and enterprise use.
## Evaluations
This model provides an accuracy recovery of 99.56%. 

| __English__   |   __[Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)__ |   __[Mistral-Small-24B-Instruct-2501-FP8-Dynamic (this)](https://huggingface.co/cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic)__ |
|:--------------|----------------------------------------------------------------------------------------------------------:|---------------------------------------------------------------------------------------------------------------------------------------:|
| Avg.          |                                                                                                     76.04 |                                                                                                                                   75.7 |
| ARC           |                                                                                                     72.6  |                                                                                                                                   72.1 |
| Hellaswag     |                                                                                                     74.5  |                                                                                                                                   74.4 |
| MMLU          |                                                                                                     81.01 |                                                                                                                                   80.6 |

We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`. 
    
## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic --max-model-len 16000 --gpu-memory-utilization 0.9
```
Access the model:
```
curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic",
        "prompt": "San Francisco is a"
    } '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**2335 tokens per second** using one NVIDIA L40S ⚡