Hi @AbdulsalamBande,
Thank you for your answer, I am also confused by point 2. I understand your answer, though, it doesn’t make sense to me that it works like that.
It is very weird to me to prevent a head from seeing parts of the features of the input. Going back to your example, you deprive each head from half of the feature of the input.
Extending your example, let’s say head 1 specializes in semantics and head 2 specializes in logic connectors in the sentence. If you give only half of the features to head 1, then it is difficult for the head to do its job because the split it receives may miss some important features from the input that contain a lot of information on semantics.
The same goes for head 2, the split it receives may miss some important features regarding logic.
I don’t understand how this architecture can be efficient, I am lacking the intuition here. What would make sense is that the split would happen AFTER the projection of Q, K and V. But then, we would not need embedding_dim to be divisible by num_heads.
Thanks