---
license: apache-2.0
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
datasets:
- agentlans/common-crawl-sample
- bigcode/the-stack-smol-xl
- open-thoughts/OpenThoughts-Unverified-173k
- cognitivecomputations/dolphin-r1
tags:
- draft
- speculative-decoding
---

![russian dolls.webp](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/hAb6qi-c0wt4wA5pl4Qup.webp)

A `0.5B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).

**NOTE**: This is a draft model for the **full-sized** `DeepSeek-R1` model and not the smaller "distilled" models!

See [jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0) for the non-GGUF version, and a detailed explanation of how the model was created.

---

# Without `imatrix`

Link | Type | PPL | PPL vs BF16
-----| -----|-----|------------
[DeepSeek-R1-DRAFT-0.5B-BF16.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-BF16.gguf) | BF16 | 11.0267 ± 0.08658 | ---
[DeepSeek-R1-DRAFT-0.5B-F16.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-F16.gguf)| F16 | 11.0294 ± 0.08660 | +0.02%
[DeepSeek-R1-DRAFT-0.5B-Q8_0.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q8_0.gguf)| Q8_0 | 11.0450 ± 0.08675 | +0.17%
[DeepSeek-R1-DRAFT-0.5B-Q6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q6_K.gguf)| Q6_K | 11.1231 ± 0.08732 | +0.87%
[DeepSeek-R1-DRAFT-0.5B-Q5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q5_K_M.gguf)| Q5_K_M | 11.2727 ± 0.08902 | +2.23%
[DeepSeek-R1-DRAFT-0.5B-Q5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q5_K_S.gguf)| Q5_K_S | 11.2803 ± 0.08888 | +2.30%
[DeepSeek-R1-DRAFT-0.5B-Q4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q4_K_M.gguf)| Q4_K_M | 11.8171 ± 0.09319 | +7.17%
[DeepSeek-R1-DRAFT-0.5B-Q4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q4_K_S.gguf)| Q4_K_S | 11.9379 ± 0.09380 | +8.26%
[DeepSeek-R1-DRAFT-0.5B-IQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-IQ4_NL.gguf)| IQ4_NL | 11.8497 ± 0.09445 | +7.46%
[DeepSeek-R1-DRAFT-0.5B-IQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-IQ4_XS.gguf)| IQ4_XS | 11.8600 ± 0.09464 | +7.56%
[DeepSeek-R1-DRAFT-0.5B-Q5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q5_1.gguf)| Q5_1 | 11.3624 ± 0.08926 | +3.05%
[DeepSeek-R1-DRAFT-0.5B-Q5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q5_0.gguf)| Q5_0 | 11.5217 ± 0.09124 | +4.49%
[DeepSeek-R1-DRAFT-0.5B-Q4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q4_1.gguf)| Q4_1 | 12.3107 ± 0.09765 | +11.64%
[DeepSeek-R1-DRAFT-0.5B-Q4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-R1-DRAFT-0.5B-Q4_0.gguf)| Q4_0 | 12.6168 ± 0.10021 | +14.42%

# With `imatrix`

Link | Type | PPL | PPL vs BF16
-----| -----|-----|------------
[DeepSeek-R1-DRAFT-0.5B-iQ6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ6_K.gguf)| Q6_K | 11.0940 ± 0.08714 | +0.61%
[DeepSeek-R1-DRAFT-0.5B-iQ5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ5_K_M.gguf)| Q5_K_M | 11.2333 ± 0.08819 | +1.87%
[DeepSeek-R1-DRAFT-0.5B-iQ5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ5_K_S.gguf)| Q5_K_S | 11.2238 ± 0.08798 | +1.79%
[DeepSeek-R1-DRAFT-0.5B-iQ4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ4_K_M.gguf)| Q4_K_M | 11.6273 ± 0.09165 | +5.45%
[DeepSeek-R1-DRAFT-0.5B-iQ4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ4_K_S.gguf)| Q4_K_S | 11.7004 ± 0.09225 | +6.11%
[DeepSeek-R1-DRAFT-0.5B-iIQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iIQ4_NL.gguf)| IQ4_NL | 11.6495 ± 0.09192 | +5.65%
[DeepSeek-R1-DRAFT-0.5B-iIQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iIQ4_XS.gguf)| IQ4_XS | 11.6924 ± 0.09246 | +6.04%
[DeepSeek-R1-DRAFT-0.5B-iQ5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ5_1.gguf)| Q5_1 | 11.2001 ± 0.08792 | +1.57%
[DeepSeek-R1-DRAFT-0.5B-iQ5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ5_0.gguf)| Q5_0 | 11.3579 ± 0.08961 | +3.00%
[DeepSeek-R1-DRAFT-0.5B-iQ4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ4_1.gguf)| Q4_1 | 11.7469 ± 0.09250 | +6.53%
[DeepSeek-R1-DRAFT-0.5B-iQ4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-R1-DRAFT-0.5B-iQ4_0.gguf)| Q4_0 | 12.1546 ± 0.09619 | +10.23%

- Based on these results, my suggestion is to use `IQ4_XS` unless you have a good reason not to (ie: use `Q4_K_S` if `IQ4_XS` runs slow on your hardware, `Q4_0` may be a better choice if running on CPU, etc).
- I am not sure if the versions created using the `imatrix` file are actually better or worse in practice (more thorough testing is needeed; PPL might not be a good predictor of actual draft acceptance rates!).
- Both `deepseek-r1` and `qwen-2.5` use [YaRN](https://arxiv.org/abs/2309.00071) as their context window extension method. To get the best quality output it is advised to use smaller contexts (eg: `16k`) when you can. Due to the way `YaRN` is implemented in `llama.cpp`; just setting the context to a massive value **will** degrade both the draft and target models' outputs!
- Do not use less than 4-bit quants for speculative decoding - the large drop in quality will not be offset by the memory bandwidth reduction!

---

I have included the [imatrix file](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-R1-DRAFT-0.5B-BF16.imatrix) used to generate the `Q4_0`-`Q6_K` quants, along with the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-R1-DRAFT-imatrix-data.txt) used to create it.

I have included the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-R1-DRAFT-perplexity-test-data.txt) used to calculate the PPL using `llama-perplexity`'s default settings.