Answers.Body: <p>This seems to be correct, <span class="math-container">$d$</span> is the dimension of the input vectors <span class="math-container">${\bf x}_i$</span>, <span class="math-container">$n$</span> is the amount of input vectors, <span class="math-container">$i=1,...,n$</span>, and <span class="math-container">$k,m$</span> is the amount of &quot;features&quot; or output dimension (for the current layer).</p>
  <p>This notation is a little inconsistent, as with <span class="math-container">$X\in \Bbb R^{n\times d}$</span> the <span class="math-container">${\bf x}_i\in \Bbb R^{1\times d}$</span> would be the rows of that matrix, so the matrix-vector products should be <span class="math-container">$W^T{\bf x}_i^T=({\bf x}_iW)^T=({\bf e}_iXW)^T$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-12T07:23:16.293Z
Answers.LastActivityDate: 2023-01-12T07:23:16.293Z
Answers.UserDisplayName: Lutz Lehmann
Body: <p>I was reading the following notes on the math behind transformers and was confused about what <span class="math-container">$\mathbf{x}_i$</span> is? If <span class="math-container">$\mathbf{x} \in \mathbb{R}^{n\times d}$</span>, then is the <span class="math-container">$i$</span> indexing the <span class="math-container">$n$</span> or the the <span class="math-container">$d$</span>? Am I correct to conclude that it has to be <span class="math-container">$n$</span> since the <span class="math-container">$W$</span>'s are in <span class="math-container">$\mathbb{R}^{d\times k}$</span>?</p>
  <p><a href="https://johnthickstun.com/docs/transformers.pdf" rel="nofollow noreferrer">https://johnthickstun.com/docs/transformers.pdf</a></p>
  
Tags: <neural-networks><math><transformer><linear-algebra><vector-space>
Title: Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?
UserDisplayName: play
ThreadText: # Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?
  
  (Asked by: username_0 on 2023-01-11)
  
  I was reading the following notes on the math behind transformers and was confused about what $\mathbf{x}\_i$ is? If $\mathbf{x} \in \mathbb{R}^{n\times d}$, then is the $i$ indexing the $n$ or the the $d$? Am I correct to conclude that it has to be $n$ since the $W$'s are in $\mathbb{R}^{d\times k}$?
  
  <https://johnthickstun.com/docs/transformers.pdf>
  
  **Tags:** neural-networks, math, transformer, linear-algebra, vector-space
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-12. Score: 0 | Accepted answer
  
  This seems to be correct, $d$ is the dimension of the input vectors ${\bf x}\_i$, $n$ is the amount of input vectors, $i=1,...,n$, and $k,m$ is the amount of "features" or output dimension (for the current layer).
  
  This notation is a little inconsistent, as with $X\in \Bbb R^{n\times d}$ the ${\bf x}\_i\in \Bbb R^{1\times d}$ would be the rows of that matrix, so the matrix-vector products should be $W^T{\bf x}\_i^T=({\bf x}\_iW)^T=({\bf e}\_iXW)^T$.
  
  
  ---
  
  