Answers.Body: <p>The biases are vectors, and their shape should be <span class="math-container">$b_1\in \mathbb R^{d_{ff}}$</span> and <span class="math-container">$b_2\in \mathbb R^{d_\text{model}}$</span>.</p>
  <p>To verify that let's compute the shapes of the above formula (for simplicity we can exclude the <span class="math-container">$\max$</span>):
  <span class="math-container">$$\begin{align}
  xW_1 &amp;= (n\times d_\text{model})\times (d_\text{model}\times d_{ff}) = n\times d_{ff} \\
  (xW_1)W_2 &amp;= (n\times d_{ff})\times (d_{ff}\times d_\text{model}) = n\times d_\text{model}
  \end{align}
  $$</span></p>
  <p>In practice, according to the underlying implementation the biases can be either:</p>
  <ol>
  <li>expanded to match the batch dimensions, i.e. the vectors can be replicated <span class="math-container">$n$</span> times (equal to the batch size) - as you wrote;</li>
  <li>summed element per element.</li>
  <li>added directly to the weight matrix: this requires adding an extra dimension to both <span class="math-container">$x$</span> (the inputs) and <span class="math-container">$W$</span>, like: <span class="math-container">$xW =  (n\times d_{model}+1)\times (d_{model}+1\times d_{ff})$</span>.</li>
  </ol>
  <p>In the first two cases, the biases are stored as vectors. I'm not aware of implementations that keeps the biases as matrices (i.e. replicated vectors) also because the batch size <span class="math-container">$n$</span> is not said to remain fixed: e.g. on inference it's tipically one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-24T21:15:37.803Z
Answers.LastActivityDate: 2023-05-24T21:15:37.803Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In transformer network (<a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Vaswani et al., 2017</a>), the feedforward networks have equation:</p>
  <p><span class="math-container">$$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$</span></p>
  <p>where <span class="math-container">$x \in \mathbb{R}^{n \times d_\mathrm{model}}$</span>, <span class="math-container">$W_1 \in\mathbb{R}^{d_\mathrm{model} \times d_{ff}}$</span>, <span class="math-container">$W_2 \in\mathbb{R}^{d_{ff} \times d_\mathrm{model}}$</span>.</p>
  <p>We know that the biases <span class="math-container">$b_1$</span> and <span class="math-container">$b_2$</span> are vectors.</p>
  <p>But, for the equation to work the shape of <span class="math-container">$b_1$</span> and <span class="math-container">$b_2$</span> must agree, i.e., <span class="math-container">$b_1 \in\mathbb{R}^{n \times d_{ff}}$</span> and <span class="math-container">$b_2 \in\mathbb{R}^{n \times d_\mathrm{model}}$</span>.</p>
  <p>My question: is it true that</p>
  <p><span class="math-container">$b_1 = \begin{bmatrix} (b_1)_{1} &amp; (b_1)_{2} &amp; \dots &amp; (b_1)_{d_{ff}}\\ (b_1)_{1} &amp; (b_1)_{2} &amp; \dots &amp; (b_1)_{d_{ff}} \\ \vdots &amp; \vdots &amp;  &amp; \vdots \\ (b_1)_{1} &amp; (b_1)_{2} &amp; \dots &amp; (b_1)_{d_{ff}} \end{bmatrix}$</span>
  and
  <span class="math-container">$b_2 = \begin{bmatrix} (b_2)_{1} &amp; (b_2)_{2} &amp; \dots &amp; (b_2)_{d_\mathrm{model}}\\ (b_2)_{1} &amp; (b_2)_{2} &amp; \dots &amp; (b_2)_{d_\mathrm{model}} \\ \vdots &amp; \vdots &amp;  &amp; \vdots \\ (b_2)_{1} &amp; (b_2)_{2} &amp; \dots &amp; (b_2)_{d_\mathrm{model}} \end{bmatrix}$</span> ?</p>
  
Tags: <transformer><feedforward-neural-networks><linear-algebra>
Title: Shape of biases in Transformer's Feedforward Network
UserDisplayName: poglhar
ThreadText: # Shape of biases in Transformer's Feedforward Network
  
  (Asked by: username_0 on 2023-05-14)
  
  In transformer network ([Vaswani et al., 2017](https://arxiv.org/abs/1706.03762)), the feedforward networks have equation:
  
  $$\mathrm{FNN}(x) = \max(0, xW\_1 + b\_1) W\_2 + b\_2$$
  
  where $x \in \mathbb{R}^{n \times d\_\mathrm{model}}$, $W\_1 \in\mathbb{R}^{d\_\mathrm{model} \times d\_{ff}}$, $W\_2 \in\mathbb{R}^{d\_{ff} \times d\_\mathrm{model}}$.
  
  We know that the biases $b\_1$ and $b\_2$ are vectors.
  
  But, for the equation to work the shape of $b\_1$ and $b\_2$ must agree, i.e., $b\_1 \in\mathbb{R}^{n \times d\_{ff}}$ and $b\_2 \in\mathbb{R}^{n \times d\_\mathrm{model}}$.
  
  My question: is it true that
  
  $b\_1 = \begin{bmatrix} (b\_1)\_{1} & (b\_1)\_{2} & \dots & (b\_1)\_{d\_{ff}}\\ (b\_1)\_{1} & (b\_1)\_{2} & \dots & (b\_1)\_{d\_{ff}} \\ \vdots & \vdots & & \vdots \\ (b\_1)\_{1} & (b\_1)\_{2} & \dots & (b\_1)\_{d\_{ff}} \end{bmatrix}$
  and
  $b\_2 = \begin{bmatrix} (b\_2)\_{1} & (b\_2)\_{2} & \dots & (b\_2)\_{d\_\mathrm{model}}\\ (b\_2)\_{1} & (b\_2)\_{2} & \dots & (b\_2)\_{d\_\mathrm{model}} \\ \vdots & \vdots & & \vdots \\ (b\_2)\_{1} & (b\_2)\_{2} & \dots & (b\_2)\_{d\_\mathrm{model}} \end{bmatrix}$ ?
  
  **Tags:** transformer, feedforward-neural-networks, linear-algebra
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-24. Score: 0 | Accepted answer
  
  The biases are vectors, and their shape should be $b\_1\in \mathbb R^{d\_{ff}}$ and $b\_2\in \mathbb R^{d\_\text{model}}$.
  
  To verify that let's compute the shapes of the above formula (for simplicity we can exclude the $\max$):
  $$\begin{align}
  xW\_1 &= (n\times d\_\text{model})\times (d\_\text{model}\times d\_{ff}) = n\times d\_{ff} \\
  (xW\_1)W\_2 &= (n\times d\_{ff})\times (d\_{ff}\times d\_\text{model}) = n\times d\_\text{model}
  \end{align}
  $$
  
  In practice, according to the underlying implementation the biases can be either:
  
  1. expanded to match the batch dimensions, i.e. the vectors can be replicated $n$ times (equal to the batch size) - as you wrote;
  2. summed element per element.
  3. added directly to the weight matrix: this requires adding an extra dimension to both $x$ (the inputs) and $W$, like: $xW = (n\times d\_{model}+1)\times (d\_{model}+1\times d\_{ff})$.
  
  In the first two cases, the biases are stored as vectors. I'm not aware of implementations that keeps the biases as matrices (i.e. replicated vectors) also because the batch size $n$ is not said to remain fixed: e.g. on inference it's tipically one.
  
  
  ---
  
  