How Transformers Work
Honestly this single image does a better job of explaining transformers than anything else I've ever seen:https://people.idsia.ch/~juergen/fastweights754x288.pngIt's pretty damn simple: a linearized transformer is a "slow" neural net whose outputs determine the weights of another ("fast") neural net.It's a NN that can use tools, subject to one major restriction: the tool must be another NN.  The restriction is is "the trick" that lets you backpropagate gradients through both NNs so you can train the slow NN based on an error function of the fast NN's outputs.The only difference between linearized transformers and the kind that OpenAI uses is adding a softmax operation in one place.