How Transformers Work
Honestly this single image does a better job of explaining transformers than <i>anything</i> else I've ever seen:<p><a href="https://people.idsia.ch/~juergen/fastweights754x288.png" rel="nofollow noreferrer">https://people.idsia.ch/~juergen/fastweights754x288.png</a><p>It's pretty damn simple: a linearized transformer is a "slow" neural net whose outputs determine the weights of another ("fast") neural net.<p>It's a NN that can use tools, subject to one major restriction: the tool must be another NN.  The restriction is is "the trick" that lets you backpropagate gradients through <i>both</i> NNs so you can train the slow NN based on an error function of the fast NN's outputs.<p>The only difference between linearized transformers and the kind that OpenAI uses is adding a softmax operation in one place.