Deep Learning, Attention, Transformers, Optical Computing, AI.
FNet: Left: standard transformer architecture. Right: FNet architecture. All else being equal, there are two computational advantages to FTs vs. attention. FTs can be calculated on graphics processing units (GPUs) fairly efficiently thanks to the Fast Fourier transform (FFT) algorithm. This allows