unsloth

/

DeepSeek-V3-bf16

like 9

Follow

Unsloth AI 1.55k

Dear developers and community users,

I tried to load this unsloth/DeepSeek-V3-bf16 via AutoModelForCausalLM.from_pretrained(" unsloth/DeepSeek-V3-bf16 ", trust_remote_code=True)

But encountered the below error:

Unknown quantization type, got fp8 - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'eetq', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet']

I tried with different transformers versions from 4.33.1, to 4.55, but none worked.

The config file of your model showed

https://huggingface.co/unsloth/DeepSeek-V3-bf16/blob/main/config.json

"torch_dtype": "bfloat16",
"transformers_version": "4.33.1",

Please advise of any fixes? Thanks

ohhh im not sure if hugging face implemented the support for it. :(

Dear developers and community users,

I tried to load this unsloth/DeepSeek-V3-bf16 via AutoModelForCausalLM.from_pretrained(" unsloth/DeepSeek-V3-bf16 ", trust_remote_code=True)

But encountered the below error:

Unknown quantization type, got fp8 - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'eetq', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet']

I tried with different transformers versions from 4.33.1, to 4.55, but none worked.

The config file of your model showed

https://huggingface.co/unsloth/DeepSeek-V3-bf16/blob/main/config.json

"torch_dtype": "bfloat16",
"transformers_version": "4.33.1",

Please advise of any fixes? Thanks

ohhh im not sure if hugging face implemented the support for it. :(

Could you please advise how I can load the model and run it? If HF's AutoModelForCausalLM.from_pretrained() doesn't support it?
thanks.

@ivanmanu

Try with vLLM or llama.cpp. Amazing tools overall!

I find vllm to be easier to start with: https://docs.vllm.ai/en/latest/

Though, llama.cpp has been great as well. I was able to fully run DeepSeek-V3-Q2_K_XS/DeepSeek-V3-Q2_K_XS.gguf on CPU exclusively. If you have the GPUs, you could go brrr.