Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
containing learnable parameters with an unparameterized, thus not a learnt transformation, being the Fourier transform. Only learnable parameters of the whole block are the weights of the feedforward network. FNet architecture (Source: Author) Fourier transform block applies a 2D DFT to its