Quantization, Large Language Models, Llm, Deep Learning, ChatGPT.
quantization is focused on Post-training quantization (PTQ) — quantization after training. The likes of LLM.int8(), GPT-Q, SmoothQuantall fall into this category. For weights, we can calculate the quantization coefficients in advance before inference, and complete the quantization. However, for