Quantization, Large Language Models, Llm, Deep Learning, ChatGPT.
weight matrix. As shown in Figure 1(a): Figure 1: Definition of per-tensor, per-token, and perchannel quantization. Per-tensor quantization is the most efficient to implement. Per-token or Per-channel As shown in Figure 1(b), per-token for activation X: each row corresponds to a quantization