Attention Is Off By One
Some very interesting discussion of outlier features and quantization: <a href="https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/" rel="nofollow noreferrer">https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea...</a><p>* Outlier values are used to prune values.
* Transformers seem to undergo a "phase shift" in how outlier features are treated around 6.7B parameters. This could complicate research on removing them.<p>Maybe you and Tim Dettmers would have a lot to talk about :)