Attention Mechanism, Neural Networks, Artificial Intelligence, Machine Learning, Deep Learning.
(worse) approximation for a random sample’s attention matrix. We discuss the dynamics of architectures with attention and/or recurrence and show how the input/temporal component behaviour differs across said architectures. We investigate the detailed temporal and input component dynamics in a