g023/Qwen3-8B-DMS-8x-4bit-NF4
Text Generation
•
8B
•
Updated
•
16
•
1
Piotr Nawrot
pnawrot
AI & ML interests
None yet
Recent Activity
liked
a model
1 day ago
g023/Qwen3-8B-DMS-8x-4bit-NF4
posted
an
update
6 days ago
We’ve just released Qwen3-8B-DMS-8x fine-tuned for 8x KV cache compression. It maintains dense model accuracy on demanding tasks like AIME24, and is perfect for inference-time scaling. The code on HF works out-of-the-box.
With DMS we fine-tune models end-to-end via distillation; this works much better than “token importance” proxies found in usual eviction methods. It’s state-of-art for KV eviction tailored for fast inference: adds negligible amount of parameters and computation to each KV head, and requires as little as 1K fine-tuning steps to reach 8x compression. It speeds-up both prefill and generation phase of Transformer LLMs, and can be combined with Sparse Attention methods such as DSA.
🎓Paper - https://neurips.cc/virtual/2025/loc/san-diego/poster/119605
💾 Checkpoint - https://huggingface.co/nvidia/Qwen3-8B-DMS-8x
📢 Article - https://ed.ac.uk/news/shrinking-ai-memory-boosts-accuracy
liked
a model
11 days ago
pnawrot/nanoT5-base
Organizations
liked
a
model
1 day ago
liked
2
models
11 days ago
liked
4
models
about 1 year ago
liked
a
model
almost 2 years ago
