File size: 3,231 Bytes
491cbd7 0a0dd21 491cbd7 0a0dd21 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
---
base_model: mistralai/Mistral-Small-24B-Instruct-2501
---
This is a quantization of the [Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).
Mistral Small 3 (2501) is a cutting-edge 24B parameter model that redefines the small LLM category under 70B, offering state-of-the-art performance comparable to larger models. Designed for fast conversational AI, low-latency function calling, and expert fine-tuning, it excels in multilingual support, advanced reasoning, and structured output generation. Released under an Apache 2.0 license, Mistral Small 3 embodies a commitment to open-source AI, serving as a versatile foundation for both community and enterprise use.
## Evaluations
This model provides an accuracy recovery of 99.56%.
| __English__ | __[Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)__ | __[Mistral-Small-24B-Instruct-2501-FP8-Dynamic (this)](https://huggingface.co/cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic)__ |
|:--------------|----------------------------------------------------------------------------------------------------------:|---------------------------------------------------------------------------------------------------------------------------------------:|
| Avg. | 76.04 | 75.7 |
| ARC | 72.6 | 72.1 |
| Hellaswag | 74.5 | 74.4 |
| MMLU | 81.01 | 80.6 |
We did not check for data contamination.
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`.
## Usage
Install **vLLM** and
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic --max-model-len 16000 --gpu-memory-utilization 0.9
```
Access the model:
```
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' {
"model": "cortecs/Mistral-Small-24B-Instruct-2501-FP8-Dynamic",
"prompt": "San Francisco is a"
} '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**2335 tokens per second** using one NVIDIA L40S ⚡ |