Llm, Model Quantization, Deep Learning, Machine Learning, Large Language Models.
quantizing activations can not only greatly reduce memory usage but also, when combined with weight quantization, make full use of integer computation to achieve performance improvement. KV cache: Quantizing the KV cache is crucial for improving the throughput of long sequence generation.