Answers.Body: <p>There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them.</p>
  
  <p>There is no one dominant attention approach and probably will not be. In fact, the earliest attention mechanism in common use in machines was likely the electromechanical fire alarm. In digital systems, it would be a vacuum tube electric eye driving an intruder alert followed by the first hardware interrupts in transistor microprocessor boards.</p>
  
  <p>The sophistication of hardware interrupts in contemporary computer systems is probably higher than attention mechanisms in neural nets as of this writing, but that may change. Currently the dictionary definition of attention is the only constraint we can place on these newer approaches in artificial networks.</p>
  
  <ul>
  <li>The four working links in the question (one is broken) lead to articles with some mathematical description, so the question half answers itself.</li>
  <li>This article, <a href="https://pdf.sciencedirectassets.com/271585/1-s2.0-S0004370200X00331/1-s2.0-S0004370297000659/main.pdf?x-amz-security-token=AgoJb3JpZ2luX2VjEOr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQCVfJlD30DAXF7w%2BJZTxVddGdAUgsgNTlP3Db8WMtDDlAIgUU7ITEmLZveMDUkKDLVV50AoHNwUQ0hqDzLJR48UKA4q4wMI4v%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgwwNTkwMDM1NDY4NjUiDOiCgCnrXcO4FYPkDCq3A9%2Bvw3GIADb%2FYrKeeImglZyeVuKDHCj5kt2lG7Zskv4scJPYJ4CH7CGL2A127Os3RvLzkHT87NUqZIUkKm%2FQPYSxJ2Cn6dmhmVgr2G44OcRT72nd84hBMN%2BYFzy3bcDcZax9AeM%2Br9fGEa1XtkDmsiMsvLfGEbByZ%2FoEZV0Kn5sxJBTBKaD5oHiKmdCDttXf9bzn0nZYEa2WA3BOzrQ8gBmtml%2BtdUa2arQXlvtsDJelhL5zjjPgJzJkOxGDLxOrEl28seWX75VNsE7010HYbpbteohQEFeAqLmmF0KLzQutYsjpfEHxeOmDuM9uzMlFYMlXdQA923o6dg%2F5YReXwon3fa84w%2FKn6cGztbDxt%2BS0n60P7FJnjNAZDqHwChSmKL3%2F61Pbie3lnfT7RAP%2B9kfKPIgfctjkjByVd2%2FB7y93X36wK%2FUwqcgnycNbfvdtY6rBGnqLXcEKG82l4bzVqTHQS8sYd791XxXa6dhw0jbAn%2Fk5aBM0ofELo4b67jfERsgj7oy4KwofjrH1WY6ExU5vMP%2FWSDPK0yPXayPWlQrOzsF0vWggkMUJ52A3jw%2FHvyU4DLotTNAwitLt5gU6tAHA%2Fy0ooqh979HVQpd5%2FGlH5eFEkiSVVRdi76GZWckrduAd1pSMPpfgKmIJfh0MDcWXUGZCx%2FR3qhq1nIzUjcgao3Iw6UYkI2smvO%2B0MyRnHxw%2BaVcWRImDeGTDhqtqfT9Y7ucVGPTDA4SYkH7nuWHxktkhSj8LtlT5Gk0iFdzMJRWIPCQx2aMf32l9qp%2FFDzpxn9gwPVickccSOHdjX9Y11MOyPVcydZ9L6ph3pntb2wd%2Fy8c%3D&amp;AWSAccessKeyId=ASIAQ3PHCVTY3MVAATUK&amp;Expires=1557889652&amp;Signature=BzzLZHNG66%2FkecI%2FtjnJCvPRF8Y%3D&amp;hash=9801a5a49f67b2342e82c00f4d9b746eb2951194644f233a0588e915f8d22bda&amp;host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&amp;pii=S0004370297000659&amp;tid=spdf-74d42e48-3195-473c-ab3e-61994c3c81c0&amp;sid=0ea5c4719d60084fe42944c4fd4f100bc2c0gxrqa&amp;type=client" rel="nofollow noreferrer"><em>Dynamic relevance: vision-based focus of attention using artificial neural networks</em></a>,
  Shumeet Baluja, Dean Pomerleau, 1995; revised 1996,
  (<a href="https://www.sciencedirect.com/science/article/pii/S0004370297000659" rel="nofollow noreferrer">abstract</a>),
  has only a few mathematical expressions but is filled with system
  theory that could be described mathematically with a little reading
  and thought.</li>
  <li>Same with <a href="http://www.diva-portal.org/smash/get/diva2:987589/FULLTEXT01.pdf" rel="nofollow noreferrer"><em>Self-Organization of an Artificial Neural Network
  Subjected to Attention Shift Impairments and Familiarity
  Preference, Characteristics Studied in Autism</em></a>, Lennart Gustafsson and Andrew P. Papliński, 2004.</li>
  <li>This one has clear and comprehensive mathematical treatment: <a href="https://www.degruyter.com/downloadpdf/j/amcs.2012.22.issue-2/v10006-012-0034-5/v10006-012-0034-5.pdf" rel="nofollow noreferrer"><em>Backpropagation Generalized Delta Rule for the Selective Attention Sigma-if Artificial Neural Network</em></a>, Maciej Huk, 2012</li>
  <li>This one too: <a href="https://doi.org/10.1371/journal.pone.0072035" rel="nofollow noreferrer"><em>Small and Dim Target Detection via Lateral Inhibition Filtering and Artificial Bee Colony Based Selective Visual Attention</em></a>, Haibin Duan, Yimin Deng, Xiaohua Wang, Chunfang Xu, 2013</li>
  <li>There is some here, but the visualizations and concepts are so theoretically relevant that one might not miss the more extensive mathematics: <a href="http://sprott.physics.wisc.edu/pubs/paper451.pdf" rel="nofollow noreferrer"><em>Using chaotic artificial neural networks to model memory in the brain</em></a>, Zainab Aram, Sajad Jafari, Jun Ma, Julien C. Sprott, Sareh Zendehrouh, Viet-Thanh Pham</li>
  <li>Solid math coverage here: <a href="https://iris.polito.it/retrieve/handle/11583/2303844/52344/cuda_v10.pdf" rel="nofollow noreferrer"><em>Parallel Implementation of Artificial Neural Network Training</em></a>, Stefano Scanzio, Sandro Cumani, Roberto Gemello, Franco Mana, P. Laface, 2010, (<a href="https://www.researchgate.net/publication/220732611_Parallel_implementation_of_artificial_neural_network_training" rel="nofollow noreferrer">abstract</a>)</li>
  <li>Also here: <a href="https://arxiv.org/pdf/1412.7755.pdf" rel="nofollow noreferrer"><em>Multiple Object Recognition with Visual Attention</em></a>, Jimmy Lei Ba, Volodymyr Mnih, Koray Kavukcuoglu, 2015 ICLR</li>
  <li>The mathematics for the very popular DiSAN is here: <a href="https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16126/16099" rel="nofollow noreferrer"><em>DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding</em></a>, Tao Shen, Jing Jiang, Tianyi Zhou, Shirui Pan, Guodong Long, Chengqi Zhang, 2018</li>
  <li>Mathematics and the algorithm is provided for this unique approach too: <a href="https://web.cs.wpi.edu/~xkong/publications/papers/kdd18b.pdf" rel="nofollow noreferrer"><em>Graph Classification using Structural Attention</em></a>, John Boaz Lee, Ryan Rossi, Xiangnan Kong, 2018</li>
  </ul>
  
  <p>It would be interesting to develop a taxonomy of attention approaches in AI, as that has probably not yet been done. It would take quite a study to see if any of the above bullet items match up with either of the two articles referenced in the question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-15T03:44:55.880Z
Answers.LastActivityDate: 2019-05-15T03:44:55.880Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in recurrent networks), should eventually lead to the network focusing on certain parts of the input/source. So what is attention mechanism really adding to the network? </p>
  
  <p>A potential answer in the case of Encoder-Decoder RNNs is:  </p>
  
  <blockquote>
    <p><a href="https://arxiv.org/pdf/1409.0473.pdf" rel="noreferrer">The most important distinguishing feature of this approach from the
    basic encoder–decoder is that it does not attempt to encode a whole
    input sentence into a single fixed-length vector. Instead, it encodes
    the input sentence into a sequence of vectors and chooses a subset of
    these vectors adaptively while decoding the translation. This frees a
    neural translation model from having to squash all the information of
    a source sentence, regardless of its length, into a fixed-length
    vector. We show this allows a model to cope better with long
    sentences.<br>
    - Neural Machine Translation by Jointly Learning to Align and Translate</a></p>
  </blockquote>
  
  <p>which made sense and the paper says that it worked better for NMT.</p>
  
  <p>A previous study indicated that breaking down the sentence into phrases could lead to better results:</p>
  
  <blockquote>
    <p><a href="https://arxiv.org/pdf/1409.1257.pdf" rel="noreferrer">In this paper, we propose a way to address this issue by automatically
    segmenting an input sentence into phrases that can be easily
    translated by the neural network translation model. Once each segment
    has been independently translated by the neural machine translation
    model, the translated clauses are concatenated to form a final
    translation. Empirical results show a significant improvement in
    translation quality for long sentences.<br>
     - Overcoming the Curse of Sentence Length for Neural Machine
    Translation using Automatic Segmentation</a></p>
  </blockquote>
  
  <p>which paved the way for further research resulting in attention models.</p>
  
  <p>I was also going through an article on <a href="https://medium.com/octavian-ai/attention-is-not-quite-all-you-need-cb605ef3aff6" rel="noreferrer">Attention is not quite all you need</a> where it said something similar:  </p>
  
  <blockquote>
    <p>An LSTM has to learn to sequentially retain past values together in a
    single internal state across multiple RNN iterations, whereas
    attention can recall past sequence values at any point in a single
    forward pass.</p>
  </blockquote>
  
  <p>and a more curated blog on the family of attention mechanism gives insight on how different ways have been formulated for implementing the concept: <a href="https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html" rel="noreferrer">Attention? Attention!</a></p>
  
  <p>Specifically, I want to know how attention mechanism is formulated for this task (aforementioned) or in general. A detailed mathematical insight would be helpful, probably somewhat on these lines: <a href="https://srome.github.io/Understanding-Attention-in-Neural-Networks-Mathematically/" rel="noreferrer">Understanding Attention in NN mathematically</a></p>
  
Tags: <neural-networks><machine-learning><deep-learning><recurrent-neural-networks><attention>
Title: A mathematical explanation of Attention Mechanism
UserDisplayName: PixelPioneer
ThreadText: # A mathematical explanation of Attention Mechanism
  
  (Asked by: username_0 on 2019-05-15)
  
  I am trying to understand why attention models are different than just using neural networks. Essentially the optimization of weights or using gates for protecting and controlling cell state (in recurrent networks), should eventually lead to the network focusing on certain parts of the input/source. So what is attention mechanism really adding to the network?
  
  A potential answer in the case of Encoder-Decoder RNNs is:
  
  > [The most important distinguishing feature of this approach from the
  > basic encoder–decoder is that it does not attempt to encode a whole
  > input sentence into a single fixed-length vector. Instead, it encodes
  > the input sentence into a sequence of vectors and chooses a subset of
  > these vectors adaptively while decoding the translation. This frees a
  > neural translation model from having to squash all the information of
  > a source sentence, regardless of its length, into a fixed-length
  > vector. We show this allows a model to cope better with long
  > sentences.  
  > - Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/pdf/1409.0473.pdf)
  
  which made sense and the paper says that it worked better for NMT.
  
  A previous study indicated that breaking down the sentence into phrases could lead to better results:
  
  > [In this paper, we propose a way to address this issue by automatically
  > segmenting an input sentence into phrases that can be easily
  > translated by the neural network translation model. Once each segment
  > has been independently translated by the neural machine translation
  > model, the translated clauses are concatenated to form a final
  > translation. Empirical results show a significant improvement in
  > translation quality for long sentences.  
  > - Overcoming the Curse of Sentence Length for Neural Machine
  > Translation using Automatic Segmentation](https://arxiv.org/pdf/1409.1257.pdf)
  
  which paved the way for further research resulting in attention models.
  
  I was also going through an article on [Attention is not quite all you need](https://medium.com/octavian-ai/attention-is-not-quite-all-you-need-cb605ef3aff6) where it said something similar:
  
  > An LSTM has to learn to sequentially retain past values together in a
  > single internal state across multiple RNN iterations, whereas
  > attention can recall past sequence values at any point in a single
  > forward pass.
  
  and a more curated blog on the family of attention mechanism gives insight on how different ways have been formulated for implementing the concept: [Attention? Attention!](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html)
  
  Specifically, I want to know how attention mechanism is formulated for this task (aforementioned) or in general. A detailed mathematical insight would be helpful, probably somewhat on these lines: [Understanding Attention in NN mathematically](https://srome.github.io/Understanding-Attention-in-Neural-Networks-Mathematically/)
  
  **Tags:** neural-networks, machine-learning, deep-learning, recurrent-neural-networks, attention
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-15. Score: 2 | Accepted answer
  
  There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them.
  
  There is no one dominant attention approach and probably will not be. In fact, the earliest attention mechanism in common use in machines was likely the electromechanical fire alarm. In digital systems, it would be a vacuum tube electric eye driving an intruder alert followed by the first hardware interrupts in transistor microprocessor boards.
  
  The sophistication of hardware interrupts in contemporary computer systems is probably higher than attention mechanisms in neural nets as of this writing, but that may change. Currently the dictionary definition of attention is the only constraint we can place on these newer approaches in artificial networks.
  
  * The four working links in the question (one is broken) lead to articles with some mathematical description, so the question half answers itself.
  * This article, [*Dynamic relevance: vision-based focus of attention using artificial neural networks*](https://pdf.sciencedirectassets.com/271585/1-s2.0-S0004370200X00331/1-s2.0-S0004370297000659/main.pdf?x-amz-security-token=AgoJb3JpZ2luX2VjEOr%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIQCVfJlD30DAXF7w%2BJZTxVddGdAUgsgNTlP3Db8WMtDDlAIgUU7ITEmLZveMDUkKDLVV50AoHNwUQ0hqDzLJR48UKA4q4wMI4v%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgwwNTkwMDM1NDY4NjUiDOiCgCnrXcO4FYPkDCq3A9%2Bvw3GIADb%2FYrKeeImglZyeVuKDHCj5kt2lG7Zskv4scJPYJ4CH7CGL2A127Os3RvLzkHT87NUqZIUkKm%2FQPYSxJ2Cn6dmhmVgr2G44OcRT72nd84hBMN%2BYFzy3bcDcZax9AeM%2Br9fGEa1XtkDmsiMsvLfGEbByZ%2FoEZV0Kn5sxJBTBKaD5oHiKmdCDttXf9bzn0nZYEa2WA3BOzrQ8gBmtml%2BtdUa2arQXlvtsDJelhL5zjjPgJzJkOxGDLxOrEl28seWX75VNsE7010HYbpbteohQEFeAqLmmF0KLzQutYsjpfEHxeOmDuM9uzMlFYMlXdQA923o6dg%2F5YReXwon3fa84w%2FKn6cGztbDxt%2BS0n60P7FJnjNAZDqHwChSmKL3%2F61Pbie3lnfT7RAP%2B9kfKPIgfctjkjByVd2%2FB7y93X36wK%2FUwqcgnycNbfvdtY6rBGnqLXcEKG82l4bzVqTHQS8sYd791XxXa6dhw0jbAn%2Fk5aBM0ofELo4b67jfERsgj7oy4KwofjrH1WY6ExU5vMP%2FWSDPK0yPXayPWlQrOzsF0vWggkMUJ52A3jw%2FHvyU4DLotTNAwitLt5gU6tAHA%2Fy0ooqh979HVQpd5%2FGlH5eFEkiSVVRdi76GZWckrduAd1pSMPpfgKmIJfh0MDcWXUGZCx%2FR3qhq1nIzUjcgao3Iw6UYkI2smvO%2B0MyRnHxw%2BaVcWRImDeGTDhqtqfT9Y7ucVGPTDA4SYkH7nuWHxktkhSj8LtlT5Gk0iFdzMJRWIPCQx2aMf32l9qp%2FFDzpxn9gwPVickccSOHdjX9Y11MOyPVcydZ9L6ph3pntb2wd%2Fy8c%3D&AWSAccessKeyId=ASIAQ3PHCVTY3MVAATUK&Expires=1557889652&Signature=BzzLZHNG66%2FkecI%2FtjnJCvPRF8Y%3D&hash=9801a5a49f67b2342e82c00f4d9b746eb2951194644f233a0588e915f8d22bda&host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&pii=S0004370297000659&tid=spdf-74d42e48-3195-473c-ab3e-61994c3c81c0&sid=0ea5c4719d60084fe42944c4fd4f100bc2c0gxrqa&type=client),
    Shumeet Baluja, Dean Pomerleau, 1995; revised 1996,
    ([abstract](https://www.sciencedirect.com/science/article/pii/S0004370297000659)),
    has only a few mathematical expressions but is filled with system
    theory that could be described mathematically with a little reading
    and thought.
  * Same with [*Self-Organization of an Artificial Neural Network
    Subjected to Attention Shift Impairments and Familiarity
    Preference, Characteristics Studied in Autism*](http://www.diva-portal.org/smash/get/diva2:987589/FULLTEXT01.pdf), Lennart Gustafsson and Andrew P. Papliński, 2004.
  * This one has clear and comprehensive mathematical treatment: [*Backpropagation Generalized Delta Rule for the Selective Attention Sigma-if Artificial Neural Network*](https://www.degruyter.com/downloadpdf/j/amcs.2012.22.issue-2/v10006-012-0034-5/v10006-012-0034-5.pdf), Maciej Huk, 2012
  * This one too: [*Small and Dim Target Detection via Lateral Inhibition Filtering and Artificial Bee Colony Based Selective Visual Attention*](https://doi.org/10.1371/journal.pone.0072035), Haibin Duan, Yimin Deng, Xiaohua Wang, Chunfang Xu, 2013
  * There is some here, but the visualizations and concepts are so theoretically relevant that one might not miss the more extensive mathematics: [*Using chaotic artificial neural networks to model memory in the brain*](http://sprott.physics.wisc.edu/pubs/paper451.pdf), Zainab Aram, Sajad Jafari, Jun Ma, Julien C. Sprott, Sareh Zendehrouh, Viet-Thanh Pham
  * Solid math coverage here: [*Parallel Implementation of Artificial Neural Network Training*](https://iris.polito.it/retrieve/handle/11583/2303844/52344/cuda_v10.pdf), Stefano Scanzio, Sandro Cumani, Roberto Gemello, Franco Mana, P. Laface, 2010, ([abstract](https://www.researchgate.net/publication/220732611_Parallel_implementation_of_artificial_neural_network_training))
  * Also here: [*Multiple Object Recognition with Visual Attention*](https://arxiv.org/pdf/1412.7755.pdf), Jimmy Lei Ba, Volodymyr Mnih, Koray Kavukcuoglu, 2015 ICLR
  * The mathematics for the very popular DiSAN is here: [*DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding*](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16126/16099), Tao Shen, Jing Jiang, Tianyi Zhou, Shirui Pan, Guodong Long, Chengqi Zhang, 2018
  * Mathematics and the algorithm is provided for this unique approach too: [*Graph Classification using Structural Attention*](https://web.cs.wpi.edu/~xkong/publications/papers/kdd18b.pdf), John Boaz Lee, Ryan Rossi, Xiangnan Kong, 2018
  
  It would be interesting to develop a taxonomy of attention approaches in AI, as that has probably not yet been done. It would take quite a study to see if any of the above bullet items match up with either of the two articles referenced in the question.
  
  
  ---
  
  