I don’t understand how nn.MultiheadAttention module work.


What is the meaning of k_dim and v_dim respectively in __init__? Are they related to key and value parameters in forward()?


Why must the embedding_dim be divisible by num_heads? What is the meaning of head_dim derived from such a fomulation?
I thought multi-head self-attention works this way:




embedding_dim are projected into query_dim, key_dim, value_dim respectively
after computing attention weights between query and key, the value_dim are collected according to the attention weights of each query, which forms exactly the output of current attention head.
Finally output of all heads are concatenated into one matrix of ~*head_num*value_dim and projected into embedding_dim. Thus, I don’t see where should the module divide embedding_dim by num_heads.


What is the difference between torchtext.nn.MultiheadAttention and torch.nn.MultiheadAttention?