Machine Learning, Transformers, Ashish Vaswani, Lstm, Recurrent Neural Network.
we get constant path lengths. Any two positions can interact with each other simultaneously. In RNNs, gates and multiplication are crucial in regulating the gradient flow. However, in self-attention, the gated structure is embedded in the design. Moreover, parallelizing self-attention is trivial.