Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
The Yannic kilcher review is quite good.https://youtu.be/iDulhoQ2proI can't ELI5 but I can ELI-junior-dev. Tl;dw:Transformers work by basically being a differentiable lookup/hash table.  First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame.Then there is an NxN matrix that is applied to your attention frame "performing the lookup query" over all other tokens in the attention frame, so every token gets a "contextual semantic understanding" that takes in both all the other stuff in the attention frame and it's relative position.Gpt is impressive because the N is really huge and it has many layers.  A big N means you can potentially access information farther away.  Each layer gives more opportunities to summarize and integrate long range information in a fractal process.Two key takeaways:- differentiable hash tables- encoding relative position using periodic functionsNB: the attention frame tokens are actually K-vectors (so the frame is a KxN matrix) and the query matrix is an NxNxK tensor IIRC but it's easier to describe it this way