Body: <p>[LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation) and generating sequences that are distinctive with more context and coherent structure as well as capturing syntactic and structural information from the original sequences. Therefore I am dealing with a time series dataset. Similar work was reported in &quot;<a href="https://arxiv.org/pdf/2002.03854.pdf" rel="nofollow noreferrer">Attentional Networks for music generation</a>&quot; but in our case, we have a different model architecture and different dataset.
  It has been known that Transformer (attention) suffers in multivariate time series dataset (Source: <a href="https://towardsdatascience.com/attention-for-time-series-classification-and-forecasting-261723e0006d" rel="nofollow noreferrer">Attention for time series forecasting and classification</a>). But given these problems were reported two years ago, the SOTA should be better by now. For that reason, my target is to use the attention mechanism in a way to overcome these challenges.</p>
  <p>Recently I have been using the <a href="https://www.tensorflow.org/addons/api_docs/python/tfa/layers/MultiHeadAttention" rel="nofollow noreferrer">multiheaded attention layer from TF</a> and testing with head size between 128 and 3074 and head number from 1 to 10 and dropout from 0.1 to 0.5. Based on the results there was no noticeable improvement in the model performance, it seems that the multi-headed attention layer didn't have contribution during training.</p>
  <p>Therefore and after carefully reading the literature I found that autoregressive attention is the best option for this types of problem. Basically, by making the attention autoregressive, it will compute the attention over the previous (decoder) outputs in such a way as to avoid using future information to make current predictions (to preserve the notion of causality). So the attention has to designed so that at each time step it needs to be autoregressive, for example, use previously generated sequences as extra input while generating the next symbol.</p>
  <p>In &quot;<a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2755456.pdf" rel="nofollow noreferrer">Autoregressive Attention for Parallel Sequence Modeling</a>&quot; paper they introduced the autoregressive attention mechanism in order to maintain the causality in the decoder. I didn't understand what they mean in Section 3.3 which describe the implementation of autoregressive attention. My problem is in the autoregressive implementation, in the paper they stated that autoregressive mechanics was implemented using the masking technique which changes all of the elements in the upper-right triangle and the diagonal
  to −∞ 3 to ensure that all the scores that would introduce future information into the attention calculation are equal to 0 after the softmax. I was hoping to see how it was implemented in the code to get a better idea of how it works.</p>
  <p>Here is how the attention is implemented in tensorflow:</p>
  <pre><code>def multiHeadedAttentionLayer(cell_input):  
      cell_state = None    
  
      if cell_state is None:
          cell_state = cell_input
  
      mha = tfa.layers.MultiHeadAttention(head_size=128, num_heads=5, dropout = 0.5)  
      cell_output = mha([cell_input, cell_state])   
      cell_state = cell_input
  
      return cell_output
  </code></pre>
  <p>Then the function is recalled in the model architecture with the rest of the layers (below is a section of the model architecture only):</p>
  <pre><code>x = MaxPooling1D(pool_size=2)(x) # previous layer
  x = multiHeadedAttentionLayer(x) # attention layer
  x = LSTM(lstmneurons, kernel_regularizer=regularizers.l2(kreg3_rate), dropout=dropout3_rate, recurrent_dropout=dropout4_rate)(x) # following layer
  x = BatchNormalization()(x) # following layer
  </code></pre>
  <p>etc....</p>
  <p>Based on my intuition the autoregression should take the output results and feed them back to the input at every time step, so my questions are:</p>
  <p>Why do we need the masking technique?</p>
  <p>How to implement the masking technique in this case?</p>
  <p>Is there is a code for the autoregressive attention that I have a look at for reference?</p>
  <p>Is my current intuition about autoregressive attention correct as shown in the diagram?</p>
  <p><a href="https://i.sstatic.net/zldoC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zldoC.png" alt="Autoregressive Attention" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T11:56:32.887Z
Comments.Text: https://github.com/neqkir/attention-mechanism
Comments.UserDisplayName: kiriloff
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T12:19:01.360Z
Comments.Text: Thanks for your comment. if you post it as an answer to the post then I will make it the correct one
Comments.UserDisplayName: Sami
Tags: <neural-networks><deep-learning><python><transformer><attention>
Title: How do autoregressive attention mechanism work in multi-headed attention?
UserDisplayName: Sami
ThreadText: # How do autoregressive attention mechanism work in multi-headed attention?
  
  (Asked by: username_0 on 2021-06-05)
  
  [LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation) and generating sequences that are distinctive with more context and coherent structure as well as capturing syntactic and structural information from the original sequences. Therefore I am dealing with a time series dataset. Similar work was reported in "Attentional Networks for music generation" but in our case, we have a different model architecture and different dataset.
  It has been known that Transformer (attention) suffers in multivariate time series dataset (Source: Attention for time series forecasting and classification). But given these problems were reported two years ago, the SOTA should be better by now. For that reason, my target is to use the attention mechanism in a way to overcome these challenges.
  Recently I have been using the multiheaded attention layer from TF and testing with head size between 128 and 3074 and head number from 1 to 10 and dropout from 0.1 to 0.5. Based on the results there was no noticeable improvement in the model performance, it seems that the multi-headed attention layer didn't have contribution during training.
  Therefore and after carefully reading the literature I found that autoregressive attention is the best option for this types of problem. Basically, by making the attention autoregressive, it will compute the attention over the previous (decoder) outputs in such a way as to avoid using future information to make current predictions (to preserve the notion of causality). So the attention has to designed so that at each time step it needs to be autoregressive, for example, use previously generated sequences as extra input while generating the next symbol.
  In "Autoregressive Attention for Parallel Sequence Modeling" paper they introduced the autoregressive attention mechanism in order to maintain the causality in the decoder. I didn't understand what they mean in Section 3.3 which describe the implementation of autoregressive attention. My problem is in the autoregressive implementation, in the paper they stated that autoregressive mechanics was implemented using the masking technique which changes all of the elements in the upper-right triangle and the diagonal
  to −∞ 3 to ensure that all the scores that would introduce future information into the attention calculation are equal to 0 after the softmax. I was hoping to see how it was implemented in the code to get a better idea of how it works.
  Here is how the attention is implemented in tensorflow:
  def multiHeadedAttentionLayer(cell_input):  
      cell_state = None    
  
      if cell_state is None:
          cell_state = cell_input
  
      mha = tfa.layers.MultiHeadAttention(head_size=128, num_heads=5, dropout = 0.5)  
      cell_output = mha([cell_input, cell_state])   
      cell_state = cell_input
  
      return cell_output
  
  Then the function is recalled in the model architecture with the rest of the layers (below is a section of the model architecture only):
  x = MaxPooling1D(pool_size=2)(x) # previous layer
  x = multiHeadedAttentionLayer(x) # attention layer
  x = LSTM(lstmneurons, kernel_regularizer=regularizers.l2(kreg3_rate), dropout=dropout3_rate, recurrent_dropout=dropout4_rate)(x) # following layer
  x = BatchNormalization()(x) # following layer
  
  etc....
  Based on my intuition the autoregression should take the output results and feed them back to the input at every time step, so my questions are:
  Why do we need the masking technique?
  How to implement the masking technique in this case?
  Is there is a code for the autoregressive attention that I have a look at for reference?
  Is my current intuition about autoregressive attention correct as shown in the diagram?
  
  **Tags:** neural-networks, deep-learning, python, transformer, attention
  > **Comments:**
  > **username_1**: https://github.com/neqkir/attention-mechanism  
  > **username_0**: Thanks for your comment. if you post it as an answer to the post then I will make it the correct one  
  
  ---
  
  ## 0 answers
  
  