In the current implementation of nn.MultiheadAttention, model_dim has to be equal to num_heads * kqv_dim, is it possible to break this limitation without reimplementing MHA? for example, I want the input and output of multi-head attention to be 512 dimentions, while having 24 heads with 64dim each head. And if that’s not possible, could someone recommend me a good implementation of multi-head attention?
I used this one https://github.com/pbloem/former/blob/master/former/modules.py 3 to train a small BERT style language model on wikitext103, but it’s not to converging for some reason.