Deep Learning, Attention, Transformers, Optical Computing, AI.
the quadratic algorithm written above to run in O(n log(n)) time using a divide-and-conquer approach. Attention cannot be accelerated in this way and thus has undesirable scaling properties of the order O(n²). FTs are not parameterised, meaning that we can reduce a model’s memory footprint by