Answers.Body: <p>Pretty much anything can be interpreted as a distribution or some kind of energy model, that doesn't mean that everything is the same</p>
  <p>First of all, the softmax output of a Seq2Seq is just a formalization of the categorical distribution with which we model our problem</p>
  <p>Attention in other hand is more complicated: to begin with, the attention module learns a residual of the input, in other word is &quot;how should i correct the current input of the layer&quot;, so attention is actually a &quot;smart weighted sum for the residual&quot;</p>
  <p>At that point, how you implement the attention, is up to you, indeed there exist alternatives to softmax attention, such as linear attention, which definitely loses the notion of &quot;distribution&quot;, but somewhat keeps the  notion of &quot;attention&quot; (since you put high magnitude scores to something you want, and small magnitude scores to what you don't care)</p>
  <p>So yeah, at most you can say that attention is &quot;a probability distribution over the tokens that allows the network to improve the current input via a residual connection&quot;... not too sure that this is a helpful interpretation whatsoever</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-30T10:37:34.607Z
Answers.LastActivityDate: 2024-05-30T10:37:34.607Z
Answers.UserDisplayName: Alberto
Body: <p>Many people explained seq2seq model by explanatory description. However, in my opinion, that is just like a robot who could say something correctly but don't really understand it. Just like the AI did.</p>
  <p>In my opinion, the correct way to descript Seq2Seq model and others in NLP should start from a probability view.
  My probaility view is very simple. The output of encoder is a presentation of the probability distribution of the next word. And in each step of the Decoder, it just modify the distribution based on each word it predicted by the distribution and output the modified distribution. Then do this again and again.</p>
  <p>So from this probability view, how could we explain the attention mechanism used in Seq2seq?</p>
  
Tags: <natural-language-processing><attention><seq2seq>
Title: probability intepreter of attention mechanism in Seq2Seq
UserDisplayName: tangyao
ThreadText: # probability intepreter of attention mechanism in Seq2Seq
  
  (Asked by: username_0 on 2024-05-30)
  
  Many people explained seq2seq model by explanatory description. However, in my opinion, that is just like a robot who could say something correctly but don't really understand it. Just like the AI did.
  
  In my opinion, the correct way to descript Seq2Seq model and others in NLP should start from a probability view.
  My probaility view is very simple. The output of encoder is a presentation of the probability distribution of the next word. And in each step of the Decoder, it just modify the distribution based on each word it predicted by the distribution and output the modified distribution. Then do this again and again.
  
  So from this probability view, how could we explain the attention mechanism used in Seq2seq?
  
  **Tags:** natural-language-processing, attention, seq2seq
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-30. Score: 0
  
  Pretty much anything can be interpreted as a distribution or some kind of energy model, that doesn't mean that everything is the same
  
  First of all, the softmax output of a Seq2Seq is just a formalization of the categorical distribution with which we model our problem
  
  Attention in other hand is more complicated: to begin with, the attention module learns a residual of the input, in other word is "how should i correct the current input of the layer", so attention is actually a "smart weighted sum for the residual"
  
  At that point, how you implement the attention, is up to you, indeed there exist alternatives to softmax attention, such as linear attention, which definitely loses the notion of "distribution", but somewhat keeps the notion of "attention" (since you put high magnitude scores to something you want, and small magnitude scores to what you don't care)
  
  So yeah, at most you can say that attention is "a probability distribution over the tokens that allows the network to improve the current input via a residual connection"... not too sure that this is a helpful interpretation whatsoever
  
  
  ---
  
  