Qualia:

Thank you for your answer, I am also confused by point 2. I understand your answer, though, it doesn’t make sense to me that it works like that.
It is very weird to me to prevent a head from seeing parts of the features of the input. Going back to your example, you deprive each head from half of the feature of the input.


Your transformer model has a certain dimensionality, like 512. Each “head” gets parts of that vector to hold it’s representation. So if you have 512 dimensionality vector representation, and 8 heads, each head gets 512/8 = 64 numbers to store it’s representation. The dimensionality is not about the features the head gets access to, it’s about the amount of numbers the heads gets to use to store its representation.