Attention Is Off By One
1. Summary<p>The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax).<p>The softmax inside an attention unit allows it to see key&#x2F;query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1&#x2F;0 output of a lookup, we get weights where a high weight = the desired key-value lookup).<p>Adding 1 to the denominator would change an attention unit by no longer working with a true probability vector of weights, but rather working with weights that add up to less than 1. The motivation is that the network can learn to provide high weights so that the adjusted softmax is very close to a probability vector; and it has a new option to provide all-low weights which give all-low output weights, meaning it can opt out of having high confidence in anything.<p>(switching to opinion mode)<p>2. How can we tell if this is good?<p>2a. We should just try it out: Train an LLM with this, see if it works.<p>2b. There are two reasons I suspect it won&#x27;t make a big difference.<p>First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Then we&#x27;re basically taking an average of a bunch of vectors (vs a weighted average that is more like choosing one of them). Statistically, we expect that averaged vector to be close to zero. In other words, the node already has a way to effectively opt-out by providing a near-zero output vector.<p>Second, in a transformer, each attention unit has many other learned weights that can support the ability to opt out. Both the V matrix and the feed-forward layer after the attention unit give that module a way to provide low values to the activation function after the feed-forward layer, which would result in a value as small as you like — again, a way to opt out.<p>3. I appreciate the non-academic tone of the article and the willingness to play around with fundamental ideas. Although I&#x27;m not totally convinced by the note, I&#x27;d love to read more stuff like this.