Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
symmetric. We can compare these techniques by quantizing every layer in GPT-2 (linear layers, attention layers, etc.) and create two new models: model_abs and model_zp. To be precise, we will actually replace the original weights with de-quantized ones. This has two benefits: it allows us to 1/