---
language:
- en
- fr
- de
- es
- pt
- it
- ja
- ko
- ru
- zh
- ar
- fa
- id
- ms
- ne
- pl
- ro
- sr
- sv
- tr
- uk
- vi
- hi
- bn
license: apache-2.0
library_name: vllm
inference: false
---

# Model Card for Mistral-Small-3.1-24B-Base-2503 (TEXT ONLY)

This is the text-only variant of [mistralai/Mistral-Small-3.1-24B-Base-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503).
This also serves as the base-model for [mistralai/Devstral-Small-2505](https://huggingface.co/mistralai/Devstral-Small-2505), which had no official base model released.

Features:
- Text-only, no multimodality.
- 128k context length.

How was a text-only model achieved? The vision encoder was removed and the model architecture was converted from mistral3 to mistral. The tokenizer was not modified.

## Reproduced eval

Serve with vLLM:

```
vllm serve casperhansen/Mistral-Small-3.1-24B-Base-2503-Text-Only
```

The reproduced results can be seen below.

| Model                              | MMLU (0-shot)   |
|------------------------------------|-----------------|
| Small 3.1 24B Base (Text Only)     | 77.25% ± 0.0033 |
| Small 3.1 24B Base (Multimodal)    | 77.34% ± 0.0033 |

### Original Multimodal: Full MMLU (Reproduced)

```
lm_eval --model local-completions \
  --model_args "base_url=http://localhost:8000/v1/completions,model=mistralai/Mistral-Small-3.1-24B-Base-2503" \
  --tasks mmlu \
  --batch_size 128
```

|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.7734|±  |0.0033|
| - humanities                          |      2|none  |      |acc   |↑  |0.6820|±  |0.0062|
|  - formal_logic                       |      1|none  |     0|acc   |↑  |0.5714|±  |0.0443|
|  - high_school_european_history       |      1|none  |     0|acc   |↑  |0.8303|±  |0.0293|
|  - high_school_us_history             |      1|none  |     0|acc   |↑  |0.9363|±  |0.0171|
|  - high_school_world_history          |      1|none  |     0|acc   |↑  |0.9241|±  |0.0172|
|  - international_law                  |      1|none  |     0|acc   |↑  |0.9091|±  |0.0262|
|  - jurisprudence                      |      1|none  |     0|acc   |↑  |0.8148|±  |0.0376|
|  - logical_fallacies                  |      1|none  |     0|acc   |↑  |0.8589|±  |0.0274|
|  - moral_disputes                     |      1|none  |     0|acc   |↑  |0.8208|±  |0.0206|
|  - moral_scenarios                    |      1|none  |     0|acc   |↑  |0.3844|±  |0.0163|
|  - philosophy                         |      1|none  |     0|acc   |↑  |0.8296|±  |0.0214|
|  - prehistory                         |      1|none  |     0|acc   |↑  |0.8704|±  |0.0187|
|  - professional_law                   |      1|none  |     0|acc   |↑  |0.6095|±  |0.0125|
|  - world_religions                    |      1|none  |     0|acc   |↑  |0.8713|±  |0.0257|
| - other                               |      2|none  |      |acc   |↑  |0.8317|±  |0.0064|
|  - business_ethics                    |      1|none  |     0|acc   |↑  |0.8200|±  |0.0386|
|  - clinical_knowledge                 |      1|none  |     0|acc   |↑  |0.8679|±  |0.0208|
|  - college_medicine                   |      1|none  |     0|acc   |↑  |0.7803|±  |0.0316|
|  - global_facts                       |      1|none  |     0|acc   |↑  |0.6600|±  |0.0476|
|  - human_aging                        |      1|none  |     0|acc   |↑  |0.7982|±  |0.0269|
|  - management                         |      1|none  |     0|acc   |↑  |0.9029|±  |0.0293|
|  - marketing                          |      1|none  |     0|acc   |↑  |0.9359|±  |0.0160|
|  - medical_genetics                   |      1|none  |     0|acc   |↑  |0.8900|±  |0.0314|
|  - miscellaneous                      |      1|none  |     0|acc   |↑  |0.9183|±  |0.0098|
|  - nutrition                          |      1|none  |     0|acc   |↑  |0.8791|±  |0.0187|
|  - professional_accounting            |      1|none  |     0|acc   |↑  |0.6277|±  |0.0288|
|  - professional_medicine              |      1|none  |     0|acc   |↑  |0.8603|±  |0.0211|
|  - virology                           |      1|none  |     0|acc   |↑  |0.5602|±  |0.0386|
| - social sciences                     |      2|none  |      |acc   |↑  |0.8736|±  |0.0059|
|  - econometrics                       |      1|none  |     0|acc   |↑  |0.6491|±  |0.0449|
|  - high_school_geography              |      1|none  |     0|acc   |↑  |0.8990|±  |0.0215|
|  - high_school_government_and_politics|      1|none  |     0|acc   |↑  |0.9637|±  |0.0135|
|  - high_school_macroeconomics         |      1|none  |     0|acc   |↑  |0.8103|±  |0.0199|
|  - high_school_microeconomics         |      1|none  |     0|acc   |↑  |0.9034|±  |0.0192|
|  - high_school_psychology             |      1|none  |     0|acc   |↑  |0.9358|±  |0.0105|
|  - human_sexuality                    |      1|none  |     0|acc   |↑  |0.8855|±  |0.0279|
|  - professional_psychology            |      1|none  |     0|acc   |↑  |0.8578|±  |0.0141|
|  - public_relations                   |      1|none  |     0|acc   |↑  |0.7909|±  |0.0390|
|  - security_studies                   |      1|none  |     0|acc   |↑  |0.8327|±  |0.0239|
|  - sociology                          |      1|none  |     0|acc   |↑  |0.9154|±  |0.0197|
|  - us_foreign_policy                  |      1|none  |     0|acc   |↑  |0.9300|±  |0.0256|
| - stem                                |      2|none  |      |acc   |↑  |0.7545|±  |0.0073|
|  - abstract_algebra                   |      1|none  |     0|acc   |↑  |0.4600|±  |0.0501|
|  - anatomy                            |      1|none  |     0|acc   |↑  |0.8148|±  |0.0336|
|  - astronomy                          |      1|none  |     0|acc   |↑  |0.9211|±  |0.0219|
|  - college_biology                    |      1|none  |     0|acc   |↑  |0.9444|±  |0.0192|
|  - college_chemistry                  |      1|none  |     0|acc   |↑  |0.5700|±  |0.0498|
|  - college_computer_science           |      1|none  |     0|acc   |↑  |0.7100|±  |0.0456|
|  - college_mathematics                |      1|none  |     0|acc   |↑  |0.6200|±  |0.0488|
|  - college_physics                    |      1|none  |     0|acc   |↑  |0.6569|±  |0.0472|
|  - computer_security                  |      1|none  |     0|acc   |↑  |0.8300|±  |0.0378|
|  - conceptual_physics                 |      1|none  |     0|acc   |↑  |0.8170|±  |0.0253|
|  - electrical_engineering             |      1|none  |     0|acc   |↑  |0.7931|±  |0.0338|
|  - elementary_mathematics             |      1|none  |     0|acc   |↑  |0.7910|±  |0.0209|
|  - high_school_biology                |      1|none  |     0|acc   |↑  |0.9323|±  |0.0143|
|  - high_school_chemistry              |      1|none  |     0|acc   |↑  |0.7586|±  |0.0301|
|  - high_school_computer_science       |      1|none  |     0|acc   |↑  |0.8900|±  |0.0314|
|  - high_school_mathematics            |      1|none  |     0|acc   |↑  |0.5185|±  |0.0305|
|  - high_school_physics                |      1|none  |     0|acc   |↑  |0.6291|±  |0.0394|
|  - high_school_statistics             |      1|none  |     0|acc   |↑  |0.7593|±  |0.0292|
|  - machine_learning                   |      1|none  |     0|acc   |↑  |0.6250|±  |0.0460|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.7734|±  |0.0033|
| - humanities     |      2|none  |      |acc   |↑  |0.6820|±  |0.0062|
| - other          |      2|none  |      |acc   |↑  |0.8317|±  |0.0064|
| - social sciences|      2|none  |      |acc   |↑  |0.8736|±  |0.0059|
| - stem           |      2|none  |      |acc   |↑  |0.7545|±  |0.0073|

### Text Only: Full MMLU

```
lm_eval --model local-completions \
  --model_args "base_url=http://localhost:8000/v1/completions,model=casperhansen/Mistral-Small-3.1-24B-Base-2503-Text-Only" \
  --tasks mmlu \
  --batch_size 128
```

|                 Tasks                 |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|------|---|-----:|---|-----:|
|mmlu                                   |      2|none  |      |acc   |↑  |0.7725|±  |0.0033|
| - humanities                          |      2|none  |      |acc   |↑  |0.6793|±  |0.0062|
|  - formal_logic                       |      1|none  |     0|acc   |↑  |0.5397|±  |0.0446|
|  - high_school_european_history       |      1|none  |     0|acc   |↑  |0.8364|±  |0.0289|
|  - high_school_us_history             |      1|none  |     0|acc   |↑  |0.9363|±  |0.0171|
|  - high_school_world_history          |      1|none  |     0|acc   |↑  |0.9198|±  |0.0177|
|  - international_law                  |      1|none  |     0|acc   |↑  |0.9008|±  |0.0273|
|  - jurisprudence                      |      1|none  |     0|acc   |↑  |0.8148|±  |0.0376|
|  - logical_fallacies                  |      1|none  |     0|acc   |↑  |0.8405|±  |0.0288|
|  - moral_disputes                     |      1|none  |     0|acc   |↑  |0.8237|±  |0.0205|
|  - moral_scenarios                    |      1|none  |     0|acc   |↑  |0.3765|±  |0.0162|
|  - philosophy                         |      1|none  |     0|acc   |↑  |0.8264|±  |0.0215|
|  - prehistory                         |      1|none  |     0|acc   |↑  |0.8704|±  |0.0187|
|  - professional_law                   |      1|none  |     0|acc   |↑  |0.6108|±  |0.0125|
|  - world_religions                    |      1|none  |     0|acc   |↑  |0.8713|±  |0.0257|
| - other                               |      2|none  |      |acc   |↑  |0.8339|±  |0.0064|
|  - business_ethics                    |      1|none  |     0|acc   |↑  |0.8300|±  |0.0378|
|  - clinical_knowledge                 |      1|none  |     0|acc   |↑  |0.8679|±  |0.0208|
|  - college_medicine                   |      1|none  |     0|acc   |↑  |0.7746|±  |0.0319|
|  - global_facts                       |      1|none  |     0|acc   |↑  |0.6800|±  |0.0469|
|  - human_aging                        |      1|none  |     0|acc   |↑  |0.8027|±  |0.0267|
|  - management                         |      1|none  |     0|acc   |↑  |0.9029|±  |0.0293|
|  - marketing                          |      1|none  |     0|acc   |↑  |0.9402|±  |0.0155|
|  - medical_genetics                   |      1|none  |     0|acc   |↑  |0.8900|±  |0.0314|
|  - miscellaneous                      |      1|none  |     0|acc   |↑  |0.9208|±  |0.0097|
|  - nutrition                          |      1|none  |     0|acc   |↑  |0.8791|±  |0.0187|
|  - professional_accounting            |      1|none  |     0|acc   |↑  |0.6312|±  |0.0288|
|  - professional_medicine              |      1|none  |     0|acc   |↑  |0.8603|±  |0.0211|
|  - virology                           |      1|none  |     0|acc   |↑  |0.5602|±  |0.0386|
| - social sciences                     |      2|none  |      |acc   |↑  |0.8739|±  |0.0059|
|  - econometrics                       |      1|none  |     0|acc   |↑  |0.6667|±  |0.0443|
|  - high_school_geography              |      1|none  |     0|acc   |↑  |0.8939|±  |0.0219|
|  - high_school_government_and_politics|      1|none  |     0|acc   |↑  |0.9585|±  |0.0144|
|  - high_school_macroeconomics         |      1|none  |     0|acc   |↑  |0.8103|±  |0.0199|
|  - high_school_microeconomics         |      1|none  |     0|acc   |↑  |0.9076|±  |0.0188|
|  - high_school_psychology             |      1|none  |     0|acc   |↑  |0.9358|±  |0.0105|
|  - human_sexuality                    |      1|none  |     0|acc   |↑  |0.8855|±  |0.0279|
|  - professional_psychology            |      1|none  |     0|acc   |↑  |0.8578|±  |0.0141|
|  - public_relations                   |      1|none  |     0|acc   |↑  |0.7909|±  |0.0390|
|  - security_studies                   |      1|none  |     0|acc   |↑  |0.8327|±  |0.0239|
|  - sociology                          |      1|none  |     0|acc   |↑  |0.9104|±  |0.0202|
|  - us_foreign_policy                  |      1|none  |     0|acc   |↑  |0.9400|±  |0.0239|
| - stem                                |      2|none  |      |acc   |↑  |0.7520|±  |0.0073|
|  - abstract_algebra                   |      1|none  |     0|acc   |↑  |0.4500|±  |0.0500|
|  - anatomy                            |      1|none  |     0|acc   |↑  |0.8296|±  |0.0325|
|  - astronomy                          |      1|none  |     0|acc   |↑  |0.9211|±  |0.0219|
|  - college_biology                    |      1|none  |     0|acc   |↑  |0.9444|±  |0.0192|
|  - college_chemistry                  |      1|none  |     0|acc   |↑  |0.5600|±  |0.0499|
|  - college_computer_science           |      1|none  |     0|acc   |↑  |0.7100|±  |0.0456|
|  - college_mathematics                |      1|none  |     0|acc   |↑  |0.6200|±  |0.0488|
|  - college_physics                    |      1|none  |     0|acc   |↑  |0.6569|±  |0.0472|
|  - computer_security                  |      1|none  |     0|acc   |↑  |0.8300|±  |0.0378|
|  - conceptual_physics                 |      1|none  |     0|acc   |↑  |0.8213|±  |0.0250|
|  - electrical_engineering             |      1|none  |     0|acc   |↑  |0.7862|±  |0.0342|
|  - elementary_mathematics             |      1|none  |     0|acc   |↑  |0.7804|±  |0.0213|
|  - high_school_biology                |      1|none  |     0|acc   |↑  |0.9290|±  |0.0146|
|  - high_school_chemistry              |      1|none  |     0|acc   |↑  |0.7488|±  |0.0305|
|  - high_school_computer_science       |      1|none  |     0|acc   |↑  |0.8900|±  |0.0314|
|  - high_school_mathematics            |      1|none  |     0|acc   |↑  |0.5222|±  |0.0305|
|  - high_school_physics                |      1|none  |     0|acc   |↑  |0.6225|±  |0.0396|
|  - high_school_statistics             |      1|none  |     0|acc   |↑  |0.7500|±  |0.0295|
|  - machine_learning                   |      1|none  |     0|acc   |↑  |0.6339|±  |0.0457|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      2|none  |      |acc   |↑  |0.7725|±  |0.0033|
| - humanities     |      2|none  |      |acc   |↑  |0.6793|±  |0.0062|
| - other          |      2|none  |      |acc   |↑  |0.8339|±  |0.0064|
| - social sciences|      2|none  |      |acc   |↑  |0.8739|±  |0.0059|
| - stem           |      2|none  |      |acc   |↑  |0.7520|±  |0.0073|