Qwen3 does not deploy on Endpoints
#23
by
zenfiric
- opened
I try to launch a model with HF Endpoints:
Endpoint encountered an error.
You can try restarting it using the "retry" button above. Check logs for more details.
[Server message]Endpoint failed to start
Exit code: 3. Reason: /routing.py", line 569, in __aenter__
await self._router.startup()
File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 670, in startup
await handler()
File "/app/webservice_starlette.py", line 62, in prepare_model_artifacts
inference_handler = get_inference_handler_either_custom_or_default_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/huggingface_inference_toolkit/handler.py", line 159, in get_inference_handler_either_custom_or_default_handler
return HuggingFaceHandler(model_dir=model_dir, task=task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/huggingface_inference_toolkit/handler.py", line 22, in __init__
self.pipeline = get_pipeline(
^^^^^^^^^^^^^
File "/app/huggingface_inference_toolkit/utils.py", line 252, in get_pipeline
hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/pipelines/__init__.py", line 849, in pipeline
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformers/models/auto/configuration_auto.py", line 1073, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type `qwen3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
Application startup failed. Exiting.