Magistral-Small-2507-Rebased-Vision

This model was created by taking Mistral-Small-3.2-24B-Instruct-2506 and replacing the weights under the language_model with the weights from Magistral-Small-2507. The result is Magistral with vision capabilities, but you should expect a small quality degradation.

Notes: I used unsloth's uploads of these models for convenience, since they include some extra files and configs too. I didn't name this "-Vision" because it was not trained or finetuned after weight rebase, and in case a future version by mistralai has vision.

The code will be released soon.

Evaluation

Evaluation was performed on 7 benchmarks using lm_eval and sglang. Scripts and other details will also be released with the code. This is not a comprehensive evaluation, and it's not directly comparable to the official benchmark numbers from Mistral, the goal was to approximate quality degradation. Make sure to test on your own downstream tasks!

Tasks	Metric	Magistral-Small-2507	Magistral-Small-2507-rebased	Relative Loss	Relative Stderr
arc_challenge_chat	exact_match	0.8933	0.8908	0.28%	±1.01%
eq_bench	eqbench	68.2823	69.3298	-1.53%	±3.48%
gsm8k	exact_match	0.9083	0.9113	-0.33%	±0.88%
gsm8k	exact_match	0.9037	0.9083	-0.51%	±0.90%
ifeval	inst_level_loose_acc	0.7578	0.753	0.63%	N/A
ifeval	inst_level_strict_acc	0.6871	0.6859	0.17%	N/A
ifeval	prompt_level_loose_acc	0.6617	0.6617	0.00%	±3.08%
ifeval	prompt_level_strict_acc	0.5767	0.5804	-0.64%	±3.69%
mbpp	pass_at_1	0.636	0.63	0.94%	±3.38%
mmlu_pro	exact_match	0.6141	0.6142	-0.02%	±0.70%
triviaqa	exact_match	0.4801	0.48	0.02%	±0.77%