Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
exactly what F-Net also does. It replaces the self-attention layers with Fourier transforms. ok, so let’s see how this replacement of attention works and what it brings. FNet architecture The authors of FNet take the classical transformer block and modify it by replacing the self-attention layer