Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
The Yannic kilcher review is quite good.<p><a href="https://youtu.be/iDulhoQ2pro" rel="nofollow">https://youtu.be/iDulhoQ2pro</a><p>I can't ELI5 but I can ELI-junior-dev. Tl;dw:<p>Transformers work by basically being a differentiable lookup/hash table.  First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity <i>and position</i> in the attention frame.<p>Then there is an NxN matrix that is applied to your attention frame "performing the lookup query" over all other tokens in the attention frame, so every token gets a "contextual semantic understanding" that takes in both all the other stuff in the attention frame and it's relative position.<p>Gpt is impressive because the N is really huge <i>and</i> it has many layers.  A big N means you can potentially access information farther away.  Each layer gives more opportunities to summarize and integrate long range information in a fractal process.<p>Two key takeaways:<p>- differentiable hash tables<p>- encoding relative position using periodic functions<p>NB: the attention frame tokens are actually K-vectors (so the frame is a KxN matrix) and the query matrix is an NxNxK tensor IIRC but it's easier to describe it this way