ValueError: There is no module or parameter named 'multi_modal_projector.ln_mid' in UltravoxModel
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] Exception in worker VllmWorkerProcess while processing method load_model.
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] Traceback (most recent call last):
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/executor/multiproc_worker_utils.py", line 236, in _run_worker_process
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/utils.py", line 2220, in run_method
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] return func(*args, **kwargs)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 183, in load_model
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] self.model_runner.load_model()
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] self.model = get_model(vllm_config=self.vllm_config)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] return loader.load_model(vllm_config=vllm_config)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 386, in load_model
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] loaded_weights = model.load_weights(
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242]
^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/ultravox.py", line 574, in load_weights
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] autoloaded_weights = set(self._load_module("", self.module, weights))
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] yield from self._load_module(prefix,
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 224, in _load_module
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] raise ValueError(msg)
(VllmWorkerProcess pid=4207) ERROR 02-18 07:04:30 multiproc_worker_utils.py:242] ValueError: There is no module or parameter named 'multi_modal_projector.ln_mid' in UltravoxModel
[rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/h/git/ultravox/vllm/examples/offline_inference_audio_language.py", line 127, in
[rank0]: main(args)
[rank0]: File "/mnt/h/git/ultravox/vllm/examples/offline_inference_audio_language.py", line 75, in main
[rank0]: llm, prompt, stop_token_ids = model_example_map[model](
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/vllm/examples/offline_inference_audio_language.py", line 39, in run_ultravox
[rank0]: llm = LLM(model=model_name, limit_mm_per_prompt={"audio": audio_count}, tensor_parallel_size=2)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/utils.py", line 1051, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 242, in init
[rank0]: self.llm_engine = self.engine_class.from_engine_args(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 484, in from_engine_args
[rank0]: engine = cls(
[rank0]: ^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 273, in init
[rank0]: self.model_executor = executor_class(vllm_config=vllm_config, )
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 262, in init
[rank0]: super().init(*args, **kwargs)
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 51, in init
[rank0]: self._init_executor()
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 125, in _init_executor
[rank0]: self._run_workers("load_model",
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
[rank0]: driver_worker_output = run_method(self.driver_worker, sent_method,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/utils.py", line 2220, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 183, in load_model
[rank0]: self.model_runner.load_model()
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1112, in load_model
[rank0]: self.model = get_model(vllm_config=self.vllm_config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/init.py", line 14, in get_model
[rank0]: return loader.load_model(vllm_config=vllm_config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 386, in load_model
[rank0]: loaded_weights = model.load_weights(
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/ultravox.py", line 574, in load_weights
[rank0]: return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 235, in load_weights
[rank0]: autoloaded_weights = set(self._load_module("", self.module, weights))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 196, in _load_module
[rank0]: yield from self._load_module(prefix,
[rank0]: File "/mnt/h/git/ultravox/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 224, in _load_module
[rank0]: raise ValueError(msg)
[rank0]: ValueError: There is no module or parameter named 'multi_modal_projector.ln_mid' in UltravoxModel
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:19<?, ?it/s]
[rank0]:[W218 07:04:31.212547979 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
/usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
any help with vllm/examples/offline_inference_audio_language.py