Hi all,
I recently started reading up on attention in the context of computer vision. In my research, I found a number of ways attention is applied for various CV tasks. However, it is still unclear to me as to what’s really happening.
When I say attention, I mean a mechanism that will focus on the important features of an image, similar to how it’s done in NLP (machine translation).
I’m looking for resources (blogs/gifs/videos) with PyTorch code that explains how to implement attention for, let’s say, a simple image classification task.
Alternatively, It would be great if you write a small implementation of only the attention mechanism in the following way -

Assume a tensor of size (h,w,c)
input tensor => attention => output

One example is the self-attention from the SAGAN paper 69 -

image829×344 54.7 KB

How would I go about implementing the self-attention shown above and eventually use it for image classification?
Thank you!