RedHatAI

/

Mistral-Small-3.2-24B-Instruct-2506-FP8

like 1

Follow

Red Hat AI 1.29k

I have the same issue. I guess this is a vllm bug rather than a bug with this repo though.

It works if you pass " --tokenizer_mode mistral --config_format mistral --load_format mistral" (found these on the usnloth model page: https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-FP8)

Please note the top section of the README where it has instructions:

Deploy this FP8 quantized variant with vLLM using:

vllm serve RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --limit_mm_per_prompt 'image=10'

Are you sure? I mean I am getting "Error retrieving safetensors" error even with all the additional tags. Running vllm-openai container, v0.9.2

It still works fine right? That is a noisy false positive where I see it continue fine for 0.9.2

vllm serve RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --limit_mm_per_prompt 'image=10'

INFO 07-11 18:43:17 [__init__.py:244] Automatically detected platform cuda.
INFO 07-11 18:43:20 [api_server.py:1395] vLLM API server version 0.9.2
INFO 07-11 18:43:20 [cli_args.py:325] non-default args: {'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8', 'tokenizer_mode': 'mistral', 'config_format': 'mistral', 'load_format': 'mistral', 'limit_mm_per_prompt': {'image': 10}}
params.json: 1.43kB [00:00, 4.43MB/s]
INFO 07-11 18:43:29 [config.py:841] This model supports multiple tasks: {'classify', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
ERROR 07-11 18:43:29 [config.py:130] Error retrieving safetensors: 'RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files., retrying 1 of 2
ERROR 07-11 18:43:31 [config.py:128] Error retrieving safetensors: 'RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files.
INFO 07-11 18:43:31 [config.py:3368] Downcasting torch.float32 to torch.bfloat16.
INFO 07-11 18:43:31 [config.py:1472] Using max model len 131072
INFO 07-11 18:43:32 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=8192.
tekken.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 19.4M/19.4M [00:20<00:00, 953kB/s]
/home/mgoin/venvs/vllm-rel/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/tekken.py:312: FutureWarning: The attributed `special_token_policy` is deprecated and will be removed in 1.10.0. Please pass a special token policy explicitly to the relevant methods.
  warnings.warn(
generation_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:00<00:00, 1.47MB/s]
INFO 07-11 18:43:58 [__init__.py:244] Automatically detected platform cuda.
INFO 07-11 18:44:00 [core.py:526] Waiting for init message from front-end.
INFO 07-11 18:44:00 [core.py:69] Initializing a V1 LLM engine (v0.9.2) with config: model='RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8', speculative_config=None, tokenizer='RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8', skip_tokenizer_init=False, tokenizer_mode=mistral, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.MISTRAL, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
INFO 07-11 18:44:01 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
/home/mgoin/venvs/vllm-rel/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/tekken.py:312: FutureWarning: The attributed `special_token_policy` is deprecated and will be removed in 1.10.0. Please pass a special token policy explicitly to the relevant methods.
  warnings.warn(
WARNING 07-11 18:44:03 [registry.py:183] PixtralProcessorAdapter did not return `BatchFeature`. Make sure to match the behaviour of `ProcessorMixin` when implementing custom processors.
/home/mgoin/venvs/vllm-rel/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/tekken.py:436: FutureWarning: Using the tokenizer's special token policy (SpecialTokenPolicy.IGNORE) is deprecated. It will be removed in 1.10.0. Please pass a special token policy explicitly. Future default will be SpecialTokenPolicy.IGNORE.
  warnings.warn(
WARNING 07-11 18:44:03 [topk_topp_sampler.py:59] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
INFO 07-11 18:44:03 [gpu_model_runner.py:1770] Starting to load model RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-FP8...
INFO 07-11 18:44:03 [gpu_model_runner.py:1775] Loading model from scratch...
INFO 07-11 18:44:03 [cuda.py:284] Using Flash Attention backend on V1 engine.
INFO 07-11 18:44:03 [weight_utils.py:292] Using model weights format ['consolidated*.safetensors', '*.pt']
consolidated.safetensors:   6%|██████████▍                                                                                                                                                                | 1.57G/25.8G [00:44<05:29, 73.4MB/s]