Can't convert to gguf model
#1
by
felikz
- opened
Using llama.cpp project:
$ python convert_hf_to_gguf.py ../inf-retriever-v1-1.5b
ValueError: Can not map tensor 'embed_tokens.weight'
probably due to missing model.safetensors.index.json ?
It looks like llama.cpp currently only supports converting "Qwen2ForCausalLM" (https://github.com/ggml-org/llama.cpp/blob/193c3e03a63ccda3ac3d6a2999e41e6d1414fe23/convert_hf_to_gguf.py#L2259).
You can try reverting it from "Qwen2Model" to "Qwen2ForCausalLM" before converting.
from transformers import AutoModelForCausalLM, AutoModel, AutoConfig
config = AutoConfig.from_pretrained("../inf-retriever-v1-1.5b")
model_causal = AutoModelForCausalLM.from_config(config)
model = AutoModel.from_pretrained("../inf-retriever-v1-1.5b")
model_causal.model = model
model_causal.save_pretrained("../inf-retriever-v1-1.5b-causal")
That works, thank you!
For anyone googling this:
Make sure you have all files from the repository in place in -causal dir, except original model.safetensors!
felikz
changed discussion status to
closed