Answers.Body: <p>TL;DR YES.</p>
  <hr />
  <p>If the sequence length of <span class="math-container">$Q, K, V$</span> is <span class="math-container">$L$</span>, the embedding size is <span class="math-container">$E$</span>, and the number of heads is <span class="math-container">$H$</span>, then weight matrices are of the order <span class="math-container">$E \times (E // H)$</span> to transform <span class="math-container">$E$</span>-sized embeddings to <span class="math-container">$(E // H)$</span> sized embeddings. We will have <span class="math-container">$H$</span> such embeddings; we get the same <span class="math-container">$E$</span>-sized vectors upon concatenation. The attention weights will be of size <span class="math-container">$L \times L$</span> to enforce every token is attending to every other token in self-attention mode.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-27T11:20:31.810Z
Answers.LastActivityDate: 2024-02-29T05:44:50.530Z
Answers.LastEditDate: 2024-02-29T05:44:50.530Z
Answers.UserDisplayName: sudar
Answers.Body: <p>During the training, we would process the entire sequence at once and train the Transformer with the <a href="https://en.wikipedia.org/wiki/Teacher_forcing" rel="nofollow noreferrer">teacher-forcing algorithm</a> usually. If the input sequences vary in length, we would truncate or pad them to a same length (of 512 tokens for instance).</p>
  <p>During the inference, the generation of the output sequence is done via ancestral sampling. We start with a special token (SOS - Start-of-sequence) and sample the next token, and then feed this new token back in again and continue.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T11:41:50.487Z
Answers.LastActivityDate: 2024-02-21T11:41:50.487Z
Answers.UserDisplayName: entropy07
Answers.Body: <p>Yes, the query, key and value are the entire sequence. If the sequence is smaller than the maximum sequence length (which is the size of the linear layers), then you use padding tokens to complete the sequence and masking in the attention. If it's longer you cannot process the entire sequence with a standard self attention layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T09:34:19.760Z
Answers.LastActivityDate: 2024-02-21T09:34:19.760Z
Answers.UserDisplayName: c p
Body: <p>About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a clear answer. But, I thought. If we feed in each sequence straight into the Attention mechanism, then the linear layers, which are supposed to transform these inputs, won't be able to accept that, because sequences can be different length, and linear layers' input shape is fixed. And if we process each token independently, we have to store the other tokens somewhere, and then have loops to iterate through each token etc.</p>
  <p>So, I decided to find some code, where people create that Multi-Head attention, and here's what I found:</p>
  <pre><code>import tensorflow as tf
  from tensorflow.keras import layers
  
  
  class MultiHeadAttention(layers.Layer):
      def __init__(self, model_dim, n_heads, rate=0.1, initializer='glorot_uniform'):
          super(MultiHeadAttention, self).__init__()
          self.n_heads = n_heads
          self.model_dim = model_dim
  
          assert model_dim % self.n_heads == 0
  
          self.head_dim = model_dim // self.n_heads
  
          self.wq = layers.Dense(model_dim, kernel_initializer=initializer)
          self.wk = layers.Dense(model_dim, kernel_initializer=initializer)
          self.wv = layers.Dense(model_dim, kernel_initializer=initializer)
          
          self.dropout1 = layers.Dropout(rate)
          self.dropout2 = layers.Dropout(rate)
          
          self.wo = layers.Dense(model_dim, kernel_initializer=initializer)
  
      def split_heads(self, x, batch_size):
          x = tf.reshape(x, (batch_size, -1, self.n_heads, self.head_dim))
          return tf.transpose(x, perm=[0, 2, 1, 3])
  
      def call(self, q, k, v, mask=None):
          batch_size = tf.shape(q)[0]
  
          q = self.wq(q)  
          k = self.wk(k)  
          v = self.wv(v)  
  
          q = self.split_heads(q, batch_size) 
          k = self.split_heads(k, batch_size)  
          v = self.split_heads(v, batch_size) 
  
          dh = tf.cast(self.head_dim, tf.float32)
          qk = tf.matmul(q, k, transpose_b=True)
          scaled_qk =  qk / tf.math.sqrt(dh)
          
          if mask is not None:
              scaled_qk += (mask * -1e9) 
  
          attn = self.dropout1(tf.nn.softmax(scaled_qk, axis=-1))
          attn = tf.matmul(attn, v) 
  
          attn = tf.transpose(attn, perm=[0, 2, 1, 3]) 
          original_size_attention = tf.reshape(attn, (batch_size, -1, self.model_dim)) 
  
          output = self.dropout2(self.wo(original_size_attention))
          return output
  class TransformerBlock(layers.Layer):
      def __init__(self, emb_dim, n_heads, mlp_dim, 
                   rate=0.1, initializer='glorot_uniform', eps=1e-6, activation='gelu'):
          super(TransformerBlock, self).__init__()
          self.attn = MultiHeadAttention(emb_dim, n_heads, initializer=initializer)
          self.mlp = tf.keras.Sequential([
              layers.Dense(mlp_dim, activation=activation, kernel_initializer=initializer), 
              layers.Dense(emb_dim, kernel_initializer=initializer),
              layers.Dropout(rate)
          ])
          self.ln1 = layers.LayerNormalization(epsilon=eps)
          self.ln2 = layers.LayerNormalization(epsilon=eps)
  
      def call(self, inputs, mask=None):
          x = self.ln1(inputs)
          x = inputs + self.attn(x, x, x, mask) 
          x = x + self.mlp(self.ln2(x))
          return x
  </code></pre>
  <p>So, here I see that they basically feed in the entire sequence into the Multi-Head attention:</p>
  <p>x = self.ln1(inputs)
  x = inputs + self.attn(x, x, x, mask)</p>
  <p>But again, sequences can be different length, but linear layers accept inputs of fixed length. Is it me not understanding something? I also read about Padding and Masking, that is used during training. Is it also used during inference?</p>
  
Tags: <neural-networks><tensorflow><transformer><attention>
Title: Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?
UserDisplayName: CyberLight 64
ThreadText: # Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?
  
  (Asked by: username_0 on 2024-02-18)
  
  About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a clear answer. But, I thought. If we feed in each sequence straight into the Attention mechanism, then the linear layers, which are supposed to transform these inputs, won't be able to accept that, because sequences can be different length, and linear layers' input shape is fixed. And if we process each token independently, we have to store the other tokens somewhere, and then have loops to iterate through each token etc.
  So, I decided to find some code, where people create that Multi-Head attention, and here's what I found:
  import tensorflow as tf
  from tensorflow.keras import layers
  
  
  class MultiHeadAttention(layers.Layer):
      def __init__(self, model_dim, n_heads, rate=0.1, initializer='glorot_uniform'):
          super(MultiHeadAttention, self).__init__()
          self.n_heads = n_heads
          self.model_dim = model_dim
  
          assert model_dim % self.n_heads == 0
  
          self.head_dim = model_dim // self.n_heads
  
          self.wq = layers.Dense(model_dim, kernel_initializer=initializer)
          self.wk = layers.Dense(model_dim, kernel_initializer=initializer)
          self.wv = layers.Dense(model_dim, kernel_initializer=initializer)
          
          self.dropout1 = layers.Dropout(rate)
          self.dropout2 = layers.Dropout(rate)
          
          self.wo = layers.Dense(model_dim, kernel_initializer=initializer)
  
      def split_heads(self, x, batch_size):
          x = tf.reshape(x, (batch_size, -1, self.n_heads, self.head_dim))
          return tf.transpose(x, perm=[0, 2, 1, 3])
  
      def call(self, q, k, v, mask=None):
          batch_size = tf.shape(q)[0]
  
          q = self.wq(q)  
          k = self.wk(k)  
          v = self.wv(v)  
  
          q = self.split_heads(q, batch_size) 
          k = self.split_heads(k, batch_size)  
          v = self.split_heads(v, batch_size) 
  
          dh = tf.cast(self.head_dim, tf.float32)
          qk = tf.matmul(q, k, transpose_b=True)
          scaled_qk =  qk / tf.math.sqrt(dh)
          
          if mask is not None:
              scaled_qk += (mask * -1e9) 
  
          attn = self.dropout1(tf.nn.softmax(scaled_qk, axis=-1))
          attn = tf.matmul(attn, v) 
  
          attn = tf.transpose(attn, perm=[0, 2, 1, 3]) 
          original_size_attention = tf.reshape(attn, (batch_size, -1, self.model_dim)) 
  
          output = self.dropout2(self.wo(original_size_attention))
          return output
  class TransformerBlock(layers.Layer):
      def __init__(self, emb_dim, n_heads, mlp_dim, 
                   rate=0.1, initializer='glorot_uniform', eps=1e-6, activation='gelu'):
          super(TransformerBlock, self).__init__()
          self.attn = MultiHeadAttention(emb_dim, n_heads, initializer=initializer)
          self.mlp = tf.keras.Sequential([
              layers.Dense(mlp_dim, activation=activation, kernel_initializer=initializer), 
              layers.Dense(emb_dim, kernel_initializer=initializer),
              layers.Dropout(rate)
          ])
          self.ln1 = layers.LayerNormalization(epsilon=eps)
          self.ln2 = layers.LayerNormalization(epsilon=eps)
  
      def call(self, inputs, mask=None):
          x = self.ln1(inputs)
          x = inputs + self.attn(x, x, x, mask) 
          x = x + self.mlp(self.ln2(x))
          return x
  
  So, here I see that they basically feed in the entire sequence into the Multi-Head attention:
  x = self.ln1(inputs)
  x = inputs + self.attn(x, x, x, mask)
  But again, sequences can be different length, but linear layers accept inputs of fixed length. Is it me not understanding something? I also read about Padding and Masking, that is used during training. Is it also used during inference?
  
  **Tags:** neural-networks, tensorflow, transformer, attention
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2024-02-27. Score: 0 | Accepted answer
  
  TL;DR YES.
  
  ---
  
  If the sequence length of $Q, K, V$ is $L$, the embedding size is $E$, and the number of heads is $H$, then weight matrices are of the order $E \times (E // H)$ to transform $E$-sized embeddings to $(E // H)$ sized embeddings. We will have $H$ such embeddings; we get the same $E$-sized vectors upon concatenation. The attention weights will be of size $L \times L$ to enforce every token is attending to every other token in self-attention mode.
  
  
  ---
  
  ### Answer by username_2 on 2024-02-21. Score: 1
  
  During the training, we would process the entire sequence at once and train the Transformer with the [teacher-forcing algorithm](https://en.wikipedia.org/wiki/Teacher_forcing) usually. If the input sequences vary in length, we would truncate or pad them to a same length (of 512 tokens for instance).
  
  During the inference, the generation of the output sequence is done via ancestral sampling. We start with a special token (SOS - Start-of-sequence) and sample the next token, and then feed this new token back in again and continue.
  
  
  ---
  
  ### Answer by username_3 on 2024-02-21. Score: 0
  
  Yes, the query, key and value are the entire sequence. If the sequence is smaller than the maximum sequence length (which is the size of the linear layers), then you use padding tokens to complete the sequence and masking in the attention. If it's longer you cannot process the entire sequence with a standard self attention layer.
  
  
  ---
  
  