UP! I am very confused by this as well! Can anyone clarify?
Also: how can kdim be different from embed_dim (which is hardcoded to be qdim, which also makes no sense / is not very general)? In order to take the dot product between queries and keys, they have to have the same length! The only one between q,k,v that should be able to have a different length is v.
And vdim should also determine the size of the output, but then out_proj (the Matrix to projekt that output to the final output of the layer) is set up with the shape (embed_dim, embed_dim), when it should be (vdim, embed_dim), or better (vdim, out_dim) to be more general.
It seems that this layer can only work when embed_dim=kdim=vdim, which is the case any way when you do not specify kdim and vdim, so whats the point?
I have not yet tried to play around with it to see if it really doesn’t work when kdim or vdim are different from embed_dim, but just from looking at the code and the doc, this seems all a bit messy and not very well documented at all!