Large Language Models, Quantization, Machine Learning, Data Science, Hands On Tutorials.
would seriously degrade the performance, while quantizing individual values would create a big overhead. In practice, we often prefer the vector-wise quantization, which considers the variability of values in rows and columns inside of the same tensor. However, even vector-wise quantization doesn’t