Answers.Body: <p><span class="math-container">$$
  Q \in R^{T \times d}, K \in R^{T \times d} \\
  Q \cdot K^T \in R^{T \times d} \times  R^{d \times T}\\
  Q \cdot K^T \in R^{T \times T} 
  $$</span></p>
  <p>Which is exactly the dimension of <span class="math-container">$D$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-09T16:45:51.607Z
Answers.Text: Except that does not make sense. $C$ has to be a sequence of matrices transforming $V$, as mentioned in the original post, so it needs to be of dimension $T\times d\times d$. Strictly speaking what you wrote is the correct way to do the matrix product, but I think it should be read as performing an outer product between $Q_t$ and $K_t^T$ for each $t\in [1, T]$. This is also what is done in the non-parallel case.
Answers.UserDisplayName: Quaere Verum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-09T16:20:37.053Z
Answers.LastActivityDate: 2024-05-09T16:20:37.053Z
Answers.UserDisplayName: Alberto
Body: <p>In <a href="https://arxiv.org/abs/2405.04517" rel="nofollow noreferrer">this</a> recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version that's explained in the appendix (page 25-26). I feel like this page might contain a mistake, or maybe I'm missing something, so I wanted to check here.</p>
  <p>The issue is as follows. We consider an input sequence <span class="math-container">$X\in\mathbb{R}^{T\times d}$</span> and we obtain two matrices <span class="math-container">$F, I\in\mathbb{R}^{T\times T}$</span> (see the paper for details), which combine into <span class="math-container">$D\in\mathbb{R}^{T\times T}$</span>. All well and good. Now, it is stated that <span class="math-container">$Q, K, V\in\mathbb{R}^{T\times d}$</span>. I believe this should be <span class="math-container">$\mathbb{R}^{T\times e}$</span> where <span class="math-container">$e$</span> denotes the embedding dimension, but that is not the main source of confusion. The point is that we have the equation <span class="math-container">$C=QK^T\odot D$</span> in the paper, and subsequently, <span class="math-container">$H$</span> is obtained as <span class="math-container">$CV$</span>. But <span class="math-container">$QK^T\in \mathbb{R}^{T\times e\times e}$</span>, which makes sense since this is a sequence of linear transformations which will transform the sequence <span class="math-container">$V\in\mathbb{R}^{T\times e}$</span>. But then the dimensions of <span class="math-container">$QK^T$</span> do not match those of <span class="math-container">$D$</span>, so the equation <span class="math-container">$QK^T\odot D$</span> does not make sense to me.</p>
  <p>Have I missed something here, or is there an issue with the equation in the paper?</p>
  
Tags: <neural-networks><machine-learning><transformer><recurrent-neural-networks><long-short-term-memory>
Title: xLSTM parallel computation - mismatch in dimensions
UserDisplayName: Quaere Verum
ThreadText: # xLSTM parallel computation - mismatch in dimensions
  
  (Asked by: username_0 on 2024-05-09)
  
  In [this](https://arxiv.org/abs/2405.04517) recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version that's explained in the appendix (page 25-26). I feel like this page might contain a mistake, or maybe I'm missing something, so I wanted to check here.
  
  The issue is as follows. We consider an input sequence $X\in\mathbb{R}^{T\times d}$ and we obtain two matrices $F, I\in\mathbb{R}^{T\times T}$ (see the paper for details), which combine into $D\in\mathbb{R}^{T\times T}$. All well and good. Now, it is stated that $Q, K, V\in\mathbb{R}^{T\times d}$. I believe this should be $\mathbb{R}^{T\times e}$ where $e$ denotes the embedding dimension, but that is not the main source of confusion. The point is that we have the equation $C=QK^T\odot D$ in the paper, and subsequently, $H$ is obtained as $CV$. But $QK^T\in \mathbb{R}^{T\times e\times e}$, which makes sense since this is a sequence of linear transformations which will transform the sequence $V\in\mathbb{R}^{T\times e}$. But then the dimensions of $QK^T$ do not match those of $D$, so the equation $QK^T\odot D$ does not make sense to me.
  
  Have I missed something here, or is there an issue with the equation in the paper?
  
  **Tags:** neural-networks, machine-learning, transformer, recurrent-neural-networks, long-short-term-memory
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-09. Score: 0
  
  $$
  Q \in R^{T \times d}, K \in R^{T \times d} \\
  Q \cdot K^T \in R^{T \times d} \times R^{d \times T}\\
  Q \cdot K^T \in R^{T \times T}
  $$
  
  Which is exactly the dimension of $D$
  
  > **Comments:**
  > **username_0**: Except that does not make sense. $C$ has to be a sequence of matrices transforming $V$, as mentioned in the original post, so it needs to be of dimension $T\times d\times d$. Strictly speaking what you wrote is the correct way to do the matrix product, but I think it should be read as performing an outer product between $Q_t$ and $K_t^T$ for each $t\in [1, T]$. This is also what is done in the non-parallel case.  
  
  ---
  
  