Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
convolution in the time domain is equivalent to multiplication in the frequency domain. The feedforward network is performing kind of convolution operations on the embeddings. Then for the next block, the frequency domain embeddings go as input and get converted into time-domain again by Fourier