Answers.Body: <p>The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.</p>
  <p>Some examples include:</p>
  <ol>
  <li><p>In <a href="https://arxiv.org/abs/1801.05894" rel="nofollow noreferrer"><em>Deep Learning: An Introduction for Applied Mathematicians</em></a>, a layer with input <span class="math-container">$x \in \mathbb R^n$</span> and output <span class="math-container">$f(x) \in \mathbb R^m$</span> is computed by
  <span class="math-container">$$ f(x) = \sigma(Wx + b)$$</span>
  for a matrix <span class="math-container">$W \in \mathbb R^{m \times n}$</span> of weights, a vector <span class="math-container">$b \in \mathbb R^m$</span> of biases and an activation function <span class="math-container">$\sigma \colon \mathbb R^m \to \mathbb R^m$</span>.</p>
  </li>
  <li><p>In <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer"><em>Deep Learning</em></a> by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using</p>
  </li>
  </ol>
  <p><span class="math-container">$$ f(x) = \max \{ 0, W^T x + c \},$$</span></p>
  <p>where the <span class="math-container">$\max$</span> function is applied componentwise and gives the ReLU activation.</p>
  <ol start="3">
  <li>In the paper <a href="https://arxiv.org/pdf/2002.06440.pdf" rel="nofollow noreferrer"><em>Federated Learning with Matched Averaging</em></a>, the authors describe a fully-connected layer using pre-multiplication,</li>
  </ol>
  <p><span class="math-container">$$f(x) = \sigma(xW),$$</span></p>
  <p>where they omit the bias to simplify notation and as before apply <span class="math-container">$\sigma$</span> entrywise. In this case of course we need to view <span class="math-container">$x$</span> as a row vector and will obtain a row vector as the result.</p>
  <p>Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-07T10:43:33.047Z
Answers.LastActivityDate: 2021-02-07T13:26:04.130Z
Answers.LastEditDate: 2021-02-07T13:26:04.130Z
Answers.UserDisplayName: htl
Body: <p>Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.</p>
  <p>There are two options:</p>
  <p>Option 1:</p>
  <p><span class="math-container">$$\begin{bmatrix}i_1 &amp; i_2\end{bmatrix} \times \begin{bmatrix} w_1 &amp; w_2 &amp; w_3\\w_4 &amp; w_5 &amp; w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 &amp; i_1*w_2+i_2*w_5 &amp;i_1*w_3+i_2*w_6\end{bmatrix}$$</span></p>
  <p>Option 2:</p>
  <p><span class="math-container">$$\begin{bmatrix} w_1 &amp; w_4\\ w_2 &amp; w_5\\ w_3 &amp; w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix}  = \begin{bmatrix}i_1*w_1 + i_2*w_4 &amp; i_1*w_2+i_2*w_5 &amp;i_1*w_3+i_2*w_6\end{bmatrix}$$</span></p>
  
Tags: <neural-networks><implementation><weights>
Title: Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
UserDisplayName: Pluxyy
ThreadText: # Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
  
  (Asked by: username_0 on 2021-02-07)
  
  Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.
  
  There are two options:
  
  Option 1:
  
  $$\begin{bmatrix}i\_1 & i\_2\end{bmatrix} \times \begin{bmatrix} w\_1 & w\_2 & w\_3\\w\_4 & w\_5 & w\_6\end{bmatrix} = \begin{bmatrix}i\_1\*w\_1 + i\_2\*w\_4 & i\_1\*w\_2+i\_2\*w\_5 &i\_1\*w\_3+i\_2\*w\_6\end{bmatrix}$$
  
  Option 2:
  
  $$\begin{bmatrix} w\_1 & w\_4\\ w\_2 & w\_5\\ w\_3 & w\_6\end{bmatrix} \times \begin{bmatrix}i\_1 \\ i\_2\end{bmatrix} = \begin{bmatrix}i\_1\*w\_1 + i\_2\*w\_4 & i\_1\*w\_2+i\_2\*w\_5 &i\_1\*w\_3+i\_2\*w\_6\end{bmatrix}$$
  
  **Tags:** neural-networks, implementation, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-07. Score: 2
  
  The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.
  
  Some examples include:
  
  1. In [*Deep Learning: An Introduction for Applied Mathematicians*](https://arxiv.org/abs/1801.05894), a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by
     $$ f(x) = \sigma(Wx + b)$$
     for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.
  2. In [*Deep Learning*](https://www.deeplearningbook.org/) by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using
  
  $$ f(x) = \max \{ 0, W^T x + c \},$$
  
  where the $\max$ function is applied componentwise and gives the ReLU activation.
  
  3. In the paper [*Federated Learning with Matched Averaging*](https://arxiv.org/pdf/2002.06440.pdf), the authors describe a fully-connected layer using pre-multiplication,
  
  $$f(x) = \sigma(xW),$$
  
  where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.
  
  Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.
  
  
  ---
  
  