mistralai/Magistral-Small-2506 · Error with vLLM docker image

3 days ago

Hi I tried to host this model with vLLM docker, tried different vLLM images versions, like latest, v0.9.0, v0.8.5, but all gave me different errors, examples below

ValueError: Unknown version: v11 in /llm/Magistral-Small-2506/tekken.json. Make sure to use a valid version string: ['v1', 'v2', 'v3', 'v7']

api_server.py: error: argument --load-format: invalid choice: 'mistral' (choose from 'auto', 'pt', 'safetensors', 'npcache', 'dummy', 'tensorizer', 'bitsandbytes')

api_server.py: error: argument --tokenizer-mode: invalid choice: 'mistral' (choose from 'auto', 'slow')

Here is the docker command I used

docker run -d --name Magistral-Small-2506 --gpus "device=0" -p 9080:8000 --ipc=host -v /opt/llm:/llm vllm/vllm-openai:v0.8.5 --model /llm/Magistral-Small-2506 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2

Could you kindly confirm the correct version of vLLM image to use? Thanks

juliendenize

Mistral AI_ org 3 days ago

•

edited 3 days ago

Hi,

Currently we recommend installing vLLM to have the latest features as follows:

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

As soon as a new release of vLLM comes out, their released docker image will support mistral-common with tokenizer v11 !

mhwang093

3 days ago

Thank you so much for your quick response, Julien!