Deep Learning, Attention, Transformers, Optical Computing, AI.
us with an insightful formalism to describe attention, as a case of parameterised token mixing. Attention is parameterised because it is calculated via application of learnable parameters in each matrix. Attention mixes tokens because each vector in the output matrix contains a representation of