Machine Learning, Transformers, Ashish Vaswani, Lstm, Recurrent Neural Network.
heads in parallel. If we had enough parameters and attention heads, we could simulate a convolution easily. Stochastic gradient descent can be performed very effectively on attention models. Gradient dynamics of attention are very simple. (just linear combinations) References Vaswani, A., Shazeer,