Answers.Body: <p>You're right that the operation performed by ggml_mul_mat() is not ordinary matrix multiplication, but a more general version of it. As the commenter to your question notes, the operation you're looking at is called batched matrix multiplication. It is a standard operation in deep learning frameworks and is commonly used in various types of neural networks, including CNNs and transformers.</p>
  <p>In batched matrix multiplication, you have two tensors, A and B, with shapes <span class="math-container">$(A_0, A_1, A_2, ..., A_n)$</span> and <span class="math-container">$(B_0, B_1, B_2, ..., B_m)$</span>, respectively. When you perform batched matrix multiplication, you multiply 2D matrices along certain dimensions while keeping the other dimensions fixed. In the specific case of ggml_mul_mat() in the LLaMA implementation, it performs batched matrix multiplication along dimensions 1 and 2, and the result is an output tensor with shape <span class="math-container">$(A_0, B_1, A_2, B_3)$</span>.</p>
  <p>With regards to the specific case you mentioned:</p>
  <ul>
  <li>Tensor <span class="math-container">$X$</span> with shape <span class="math-container">$(a, m, c, d)$</span></li>
  <li>Tensor <span class="math-container">$Y$</span> with shape <span class="math-container">$(a, n, c, d)$</span></li>
  </ul>
  <p>The batched matrix multiplication will result in a new tensor, <span class="math-container">$Z$</span>, with shape <span class="math-container">$(m, n, c, d)$</span>. The operation is performed as follows:</p>
  <p><span class="math-container">$Z[i, j, k, l] = \sum(X[:, i, k, l] \times Y[:, j, k, l])$</span></p>
  <p>In this equation, we perform element-wise multiplication and sum over the first dimension <span class="math-container">$(a)$</span>. The resulting tensor <span class="math-container">$Z$</span> will have shape <span class="math-container">$(m, n, c, d)$</span>.</p>
  <p>Batch matrix multiplication is also described in the documentation of the DL libraries:</p>
  <ul>
  <li>TensorFlow: tf.linalg.matmul</li>
  <li>PyTorch: torch.matmul or torch.bmm</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T15:16:00.550Z
Answers.LastActivityDate: 2023-04-27T16:09:39.337Z
Answers.LastEditDate: 2023-04-27T16:09:39.337Z
Answers.UserDisplayName: Hans-Peter Schrei
Body: <p>I’m reading the source code of <a href="https://github.com/antimatter15/alpaca.cpp" rel="nofollow noreferrer"><code>alpaca.cpp</code></a> in an attempt to understand how a large language model works. (I have a strong programming background, but almost no math, so it’s easier for me to start with code and then go backwards to the papers once I have some idea what’s going on.)</p>
  <p>Partway down <code>llama_eval</code>, it makes a call to <code>ggml_mul_mat()</code> which caught my attention:</p>
  <pre><code>// K * Q
  struct ggml_tensor * KQ = ggml_mul_mat(ctx0, K, Q);
  </code></pre>
  <p>At first I thought this was an ordinary matrix multiplication; the “Attention is all you need” paper certainly uses that for K and Q. But in this case <code>K</code> and <code>Q</code> appear to be 3d tensors, and my understanding is that matrix multiplication only works on 2d matrices.</p>
  <p>So I dug into the <code>ggml</code> code; <a href="https://github.com/antimatter15/alpaca.cpp/blob/899fa599542ac5d77efc7d909be5393b1dba375f/ggml.c#L3670-L3671" rel="nofollow noreferrer"><code>ggml_mul_mat</code> itself</a> is just a wrapper function, but the first thing I noticed is that it supports up to 4 dimensions, with an interleaved result:</p>
  <pre><code>static inline bool ggml_can_mul_mat(const struct ggml_tensor * t0, const struct ggml_tensor * t1) {
   return (t0-&gt;ne[0]  == t1-&gt;ne[0])  &amp;&amp;
          (t0-&gt;ne[2]  == t1-&gt;ne[2])  &amp;&amp;
          (t0-&gt;ne[3]  == t1-&gt;ne[3]);
  }
  
  const int ne[4] = { a-&gt;ne[1], b-&gt;ne[1], a-&gt;ne[2], b-&gt;ne[3] };
  struct ggml_tensor * result = ggml_new_tensor(ctx, GGML_TYPE_F32, MIN(a-&gt;n_dims, b-&gt;n_dims), ne);
  </code></pre>
  <p>...which, if I'm reading this right, means that it will take an A×M×C×D tensor, and an A×N×C×D tensor, and return an M×N×C×D tensor, truncating as needed, which <em>definitely</em> doesn’t sound like matrix multiplication already.</p>
  <p>The actual implementation (in <code>ggml_compute_forward_mul_mat_*()</code>) has a lot of special cases, and even <a href="https://github.com/antimatter15/alpaca.cpp/blob/899fa599542ac5d77efc7d909be5393b1dba375f/ggml.c#L5711-L5734" rel="nofollow noreferrer">the simplest case I could find</a> I’m having trouble reading without already understanding what it’s doing. But the best impression I’ve got is that it’s doing some sort of sideways matrix multiplication, piecewise along the other two dimensions.</p>
  <p>Does this operation have a name? Is it some kind of standard operation in CNNs, or is it something that GGML and/or LLaMA have made up? Where can I read more about what’s going on here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-18T06:31:38.770Z
Comments.Text: Could this just be batchwise matrix multiplication, the additional dimension being the batch dimension?
Comments.UserDisplayName: N. Kiefer
Tags: <math><large-language-models><tensor>
Title: What operation is ggml_mul_mat performing? (K×Q in LLaMA)
UserDisplayName: Wolfgang
ThreadText: # What operation is ggml_mul_mat performing? (K×Q in LLaMA)
  
  (Asked by: username_0 on 2023-04-18)
  
  I’m reading the source code of alpaca.cpp in an attempt to understand how a large language model works. (I have a strong programming background, but almost no math, so it’s easier for me to start with code and then go backwards to the papers once I have some idea what’s going on.)
  Partway down llama_eval, it makes a call to ggml_mul_mat() which caught my attention:
  // K * Q
  struct ggml_tensor * KQ = ggml_mul_mat(ctx0, K, Q);
  
  At first I thought this was an ordinary matrix multiplication; the “Attention is all you need” paper certainly uses that for K and Q. But in this case K and Q appear to be 3d tensors, and my understanding is that matrix multiplication only works on 2d matrices.
  So I dug into the ggml code; ggml_mul_mat itself is just a wrapper function, but the first thing I noticed is that it supports up to 4 dimensions, with an interleaved result:
  static inline bool ggml_can_mul_mat(const struct ggml_tensor * t0, const struct ggml_tensor * t1) {
   return (t0->ne[0]  == t1->ne[0])  &&
          (t0->ne[2]  == t1->ne[2])  &&
          (t0->ne[3]  == t1->ne[3]);
  }
  
  const int ne[4] = { a->ne[1], b->ne[1], a->ne[2], b->ne[3] };
  struct ggml_tensor * result = ggml_new_tensor(ctx, GGML_TYPE_F32, MIN(a->n_dims, b->n_dims), ne);
  
  ...which, if I'm reading this right, means that it will take an A×M×C×D tensor, and an A×N×C×D tensor, and return an M×N×C×D tensor, truncating as needed, which definitely doesn’t sound like matrix multiplication already.
  The actual implementation (in ggml_compute_forward_mul_mat_*()) has a lot of special cases, and even the simplest case I could find I’m having trouble reading without already understanding what it’s doing. But the best impression I’ve got is that it’s doing some sort of sideways matrix multiplication, piecewise along the other two dimensions.
  Does this operation have a name? Is it some kind of standard operation in CNNs, or is it something that GGML and/or LLaMA have made up? Where can I read more about what’s going on here?
  
  **Tags:** math, large-language-models, tensor
  > **Comments:**
  > **username_1**: Could this just be batchwise matrix multiplication, the additional dimension being the batch dimension?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-04-27. Score: 1
  
  You're right that the operation performed by ggml\_mul\_mat() is not ordinary matrix multiplication, but a more general version of it. As the commenter to your question notes, the operation you're looking at is called batched matrix multiplication. It is a standard operation in deep learning frameworks and is commonly used in various types of neural networks, including CNNs and transformers.
  
  In batched matrix multiplication, you have two tensors, A and B, with shapes $(A\_0, A\_1, A\_2, ..., A\_n)$ and $(B\_0, B\_1, B\_2, ..., B\_m)$, respectively. When you perform batched matrix multiplication, you multiply 2D matrices along certain dimensions while keeping the other dimensions fixed. In the specific case of ggml\_mul\_mat() in the LLaMA implementation, it performs batched matrix multiplication along dimensions 1 and 2, and the result is an output tensor with shape $(A\_0, B\_1, A\_2, B\_3)$.
  
  With regards to the specific case you mentioned:
  
  * Tensor $X$ with shape $(a, m, c, d)$
  * Tensor $Y$ with shape $(a, n, c, d)$
  
  The batched matrix multiplication will result in a new tensor, $Z$, with shape $(m, n, c, d)$. The operation is performed as follows:
  
  $Z[i, j, k, l] = \sum(X[:, i, k, l] \times Y[:, j, k, l])$
  
  In this equation, we perform element-wise multiplication and sum over the first dimension $(a)$. The resulting tensor $Z$ will have shape $(m, n, c, d)$.
  
  Batch matrix multiplication is also described in the documentation of the DL libraries:
  
  * TensorFlow: tf.linalg.matmul
  * PyTorch: torch.matmul or torch.bmm
  
  
  ---
  
  