Attention Mechanism, Neural Networks, Artificial Intelligence, Machine Learning, Deep Learning.
bidirectional RNNs or multiheaded and self-attention mechanisms. They also note that the attention-only architectures examined are akin to transformers, suggesting a similar dynamical behaviour may also hold for that popular non-recurrent architecture. The paper Understanding How Encoder-Decoder