I am also confused with the PyTorch implementation.
In the multi_head_attention_forward 2 method Q, K and V matrices are first calculated with respect to the shapes of the source and target tensors (S,N,E) and (T,N,E), E being the size of each embedding.
But then 1 they are reshaped splitting the embedding dimension as @cgarciae just said
So why ? As the goal of MHA is to project these matrices on each head parameters and then concat them and project again on W_0 ?
The formula is even in the documentation 3 but is not applied properly ?
This is very unclear, is there something I totally missed ?
EDIT: I found what I missed, I got confused by the formula itself which doesn’t distingue the Q/K/V slices for each head