unslot-bnb-4bit for Qwen MoE models
In the article "Qwen3: How to Run & Fine-tune", in the "Dynamic 4-bit Safetensor (to fine-tune/deploy)" column of the model table, there is a link to unslot-bnb-4bit Qwen-30B-A3B (https://huggingface.co/unsloth/Qwen3-30B-A3B-unsloth-bnb-4bit) but this model does not exist. (While the bigger Qwen MoE is not even present in that column).
Does unslot-bnb-4bit quantization support MoE? Will those models add to unsloth model collections?
i downloaded the model and tried to load the shards but it took me like half hour to load all the shards
In the article "Qwen3: How to Run & Fine-tune", in the "Dynamic 4-bit Safetensor (to fine-tune/deploy)" column of the model table, there is a link to unslot-bnb-4bit Qwen-30B-A3B (https://huggingface.co/unsloth/Qwen3-30B-A3B-unsloth-bnb-4bit) but this model does not exist. (While the bigger Qwen MoE is not even present in that column).
Does unslot-bnb-4bit quantization support MoE? Will those models add to unsloth model collections?
I found this issue https://github.com/vllm-project/vllm/issues/17337#issuecomment-2838440466 it seems that bitsandbytes quantization does not support MoE
Yes that is correct, BnB does not support MOE hence we didn't upload our dynamic quant unfortunately 😞
Can you suggest solution for it ?