Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
transform. This conversion from time-domain to frequency-domain and vice-versa happens N times. N is the number of blocks in a transformer. Also, it can be thought of as alternating between multiplication and convolutions. Why Fourier? The authors have tried simple linear mixing schemes then went