Quantization, Llm, Large Language Models, Langchain, Text Generation.
quantization is similar to zero-point, but instead of setting a custom range (-1,1), we set it as (-abs(max), abs(max)). Abs-max quantization Let’s take a look at how these practices are used in an example of matrix multiplication: Exact matrix multiplication Zero-point quantization: Zero-point