PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs Paper • 2410.05265 • Published Oct 7, 2024 • 32
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation Paper • 2503.19950 • Published Mar 25 • 11
Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads Paper • 2501.15113 • Published Jan 25 • 1
view article Article Fast, High-Fidelity LLM Decoding with Regex Constraints By vivien • Feb 23, 2024 • 6
InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Paper • 2409.04992 • Published Sep 8, 2024 • 2
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval Paper • 2409.10516 • Published Sep 16, 2024 • 44
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits Paper • 2402.17764 • Published Feb 27, 2024 • 618
view article Article Fine-tuning LLMs to 1.58bit: extreme quantization made easy By medmekk and 5 others • Sep 18, 2024 • 246
Running 20 20 Zero Bubble Pipeline Parallellism 🏆 Optimize pipeline schedules for efficient computing
LLM in a flash: Efficient Large Language Model Inference with Limited Memory Paper • 2312.11514 • Published Dec 12, 2023 • 257