Deep Learning, Attention, Transformers, Optical Computing, AI.
replacing attention with FTs, as we do not need to store key, query and value matrices. Results The researchers reported that FNet trained faster (80% on GPUs, 70% on TPUs), and with much higher stability than BERT. The peak performance as measured by accuracy on the GLUE benchmark was 92% as good