What is self-attention in transformers?
Self attention is a mechanism used by the transformer model for sequence modeling. It's similar to convolutions, but operates on sequential data instead of spatial. The input to the Transformer is broken up into multiple parts and each part is processed independently. Each processing step takes the query vector which then attends to all positions within that span where the mask allows it to calculate the final representation for these positional weighted elements, here weigthed exponentially moving average of them