Quantization, Llm, Large Language Models, Langchain, Text Generation.
Zero-point and abs-max quantization In fact, we can, and there are several approaches for such quantization: Zero-point quantization saves half the memory by converting a fixed range (-1, 1) to int8 (-127, 127), followed by converting int8 back to bfloat16. Zero-point quantization Abs-max