Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
self-attention in BERT with Fourier transforms. The F-Net proposed in this paper does one modification to the transformer architecture, but this has quite the impact. Why not self-attention? For a bit of context here: The Transformer is composed of self-attention, followed by fully connected layers