Looks like the GGUF files doesn't include vision model
Hi guys! Thank you very much for all your work made for the community!
Guys I've been trying to work with the quantizied versions of Qwen 2.5 VL, and after inspecting the tensors of different .gguf files, specifically:
Qwen2.5-VL-32B-Instruct-Q6_K.gguf and Qwen2.5-VL-32B-Instruct-UD-Q4_K_XL.gguf, I found that those gguf files doesn't contain the tenstors corresponding to the vision model. If you review for example the .gguf file of ollama, you'll see that they have tensors like v.blk* and v.merger.ln_q.weight
v.merger.mlp.0.bias
v.merger.mlp.0.weight
v.merger.mlp.2.bias
v.merger.mlp.2.weight
v.patch_embd_0.weight
v.patch_embd_1.weight
https://ollama.com/library/qwen2.5vl:32b-q4_K_M/blobs/043a363c6ca3
The lack of those tensors makes impossible to load the model into tools like ollama. BTW also the metadata of the gguf files is not well, on the unsloth files it uses qwen2vl instead of qwen25vl that is the used by ollama.
Thank you very much for your support!