• Multi-Head (Self-)Attention. Multi-head self-attention extends the self-attention
mechanism by performing it multiple times in parallel, with diﬀerent sets of
learned parameters. This allows the model to capture diﬀerent types of relation-
ships and dependencies in the input sequence, providing a richer representation.
Mathematically, multi-head self-attention involves computing multiple sets of
query, key, and value matrices, and then concatenating the results from each head:

MultiHead(Q, K, V ) = Concat(cid:0)head1, head2, . . . , headh(cid:1)W O,

(2)

where headi = Attention(QW Q
i ) represents the output of the
i-th attention head, and W O is a learned linear transformation. Multi-head
self-attention enhances the model’s ability to capture both local and global
dependencies in the data.

i , KW K

, V W V

i
Beyond Efficiency