lym00/DEPRECATED-qwen-image-gguf-test

⚠️ Deprecation Notice

Please visit city96's repo https://huggingface.co/city96/Qwen-Image-gguf/tree/main for the full updated quants

This project is now deprecated and was intended for experimental use only.

This contains non-official and suboptimal patches.

---

Please visit city96's repo https://huggingface.co/city96/Qwen-Image-gguf/

---

ComfyUI Initial GGUF Tests

Update ComfyUI to pull the relevant updates (Initial support for qwen image model)

Update ComfyUI-GGUF nodes.

Type	Name	Location	Download
Main Model	Qwen_Image-GGUF	ComfyUI/models/unet	GGUF
Text Encoder	qwen_2.5_vl_7b	ComfyUI/models/text_encoders	Safetensors
VAE	qwen_image_vae	ComfyUI/models/vae	Safetensors

ComfyUI example workflow

---

References

Tensors: https://huggingface.co/Qwen/Qwen-Image/blob/main/transformer/diffusion_pytorch_model.safetensors.index.json

Tools: https://github.com/city96/ComfyUI-GGUF/tree/main/tools

Patches for unknown model: (referring last commit for cosmos)

ComfyUI Implementation: https://github.com/comfyanonymous/ComfyUI/commit/c012400240d4867cd63a45220eb791b91ad47617

---

Patches (non-official)

convert.py (ref)

class ModelQwenImage(ModelTemplate):
    arch = "qwen_image"
    keys_detect = [
        (
            "img_in.weight",
            "transformer_blocks.59.attn.add_q_proj.weight",
            "transformer_blocks.59.attn.add_k_proj.weight",
        )
    ]
    keys_banned = []  # Don't ban anything here

arch_list = [ModelQwenImage, ModelFlux, ModelSD3, ModelAura, ModelHiDream, ModelLTXV, ModelHyVid, ModelWan, ModelSDXL, ModelSD1]

loader.py (ref)

IMG_ARCH_LIST = {"qwen_image", "flux", "sd1", "sdxl", "sd3", "aura", "hidream", "cosmos", "ltxv", "hyvid", "wan"}

lcpp.patch (ref)

--- a/src/llama.cpp
+++ b/src/llama.cpp
@@ -205,6 +205,17 @@ enum llm_arch {
     LLM_ARCH_GRANITE,
     LLM_ARCH_GRANITE_MOE,
     LLM_ARCH_CHAMELEON,
+    LLM_ARCH_FLUX,
+    LLM_ARCH_SD1,
+    LLM_ARCH_SDXL,
+    LLM_ARCH_SD3,
+    LLM_ARCH_AURA,
+    LLM_ARCH_LTXV,
+    LLM_ARCH_HYVID,
+    LLM_ARCH_WAN,
+    LLM_ARCH_HIDREAM,
+    LLM_ARCH_COSMOS,
+    LLM_ARCH_QWEN_IMAGE,

@@ -258,6 +269,17 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
     { LLM_ARCH_GRANITE,         "granite"      },
     { LLM_ARCH_GRANITE_MOE,     "granitemoe"   },
     { LLM_ARCH_CHAMELEON,       "chameleon"    },
+    { LLM_ARCH_FLUX,            "flux"         },
+    { LLM_ARCH_SD1,             "sd1"          },
+    { LLM_ARCH_SDXL,            "sdxl"         },
+    { LLM_ARCH_SD3,             "sd3"          },
+    { LLM_ARCH_AURA,            "aura"         },
+    { LLM_ARCH_LTXV,            "ltxv"         },
+    { LLM_ARCH_HYVID,           "hyvid"        },
+    { LLM_ARCH_WAN,             "wan"          },
+    { LLM_ARCH_HIDREAM,         "hidream"      },
+    { LLM_ARCH_COSMOS,          "cosmos"       },
+    { LLM_ARCH_QWEN_IMAGE,      "qwen_image"    },
     { LLM_ARCH_UNKNOWN,         "(unknown)"    },

@@ -1531,6 +1553,17 @@ static const std::map<llm_arch, std::map<llm_tensor, const char *>> LLM_TENSOR_N
             { LLM_TENSOR_ATTN_K_NORM,     "blk.%d.attn_k_norm" },
         },
     },
+    { LLM_ARCH_FLUX,       {}},
+    { LLM_ARCH_SD1,        {}},
+    { LLM_ARCH_SDXL,       {}},
+    { LLM_ARCH_SD3,        {}},
+    { LLM_ARCH_AURA,       {}},
+    { LLM_ARCH_LTXV,       {}},
+    { LLM_ARCH_HYVID,      {}},
+    { LLM_ARCH_WAN,        {}},
+    { LLM_ARCH_HIDREAM,    {}},
+    { LLM_ARCH_COSMOS,     {}},
+    { LLM_ARCH_QWEN_IMAGE, {}},
     {
         LLM_ARCH_UNKNOWN,

@@ -5403,6 +5436,25 @@ static void llm_load_hparams(
     // get general kv
     ml.get_key(LLM_KV_GENERAL_NAME, model.name, false);
 
+    // Disable LLM metadata for image models
+    switch (model.arch) {
+        case LLM_ARCH_FLUX:
+        case LLM_ARCH_SD1:
+        case LLM_ARCH_SDXL:
+        case LLM_ARCH_SD3:
+        case LLM_ARCH_AURA:
+        case LLM_ARCH_LTXV:
+        case LLM_ARCH_HYVID:
+        case LLM_ARCH_WAN:
+        case LLM_ARCH_HIDREAM:
+        case LLM_ARCH_COSMOS:
+        case LLM_ARCH_QWEN_IMAGE:
+            model.ftype = ml.ftype;
+            return;
+        default:
+            break;
+    }

---

MSVC 19.44 Error and Fix

Error

error C2039: 'system_clock': is not a member of 'std::chrono'

common.cpp and log.cpp

#include <chrono>

---

Compile and Quantize

Apply patch, recompile, and quantize: https://github.com/city96/ComfyUI-GGUF/tree/main/tools#quantizing-using-custom-llamacpp