But nn.MultiheadAttention the is just for sequence models.
I guess you meant some techniques to apply attention to convolution networks.
Attention is like a new wave for convnets.
You can do it either by changing the architecture or changing the loss function or both.
The problem with convolution is that it has local receptive field.
Opposite to that fc layers have the global receptive field. So the idea to combine that using SE blocks is here 44.
Also, there is an idea to concat convolution and attentional featuremaps here 64.
Same ideas but for GAN, you can find in SAGAN 18.
But these papers I think haven’t been implemented in PyTorch yet. It may take about few months for the good paper to be inside the PyTorch.