Answers.Body: <p>It helps to think of each output dimension separately.</p>
  <p>You have <span class="math-container">$X$</span> which is an <span class="math-container">$(2 \times 1)$</span> vector, and <span class="math-container">$W_1$</span> is a <span class="math-container">$(1 \times 2 )$</span> vector. Their product is a scalar, of which we then take the sigmoid to get our output <span class="math-container">$Y_1$</span>.</p>
  <p>The gradient of this w.r.t. <span class="math-container">$W_1$</span> will be <span class="math-container">$f(WX) (1 - f(WX)) X^T$</span>, which has the appropriate dimensions.</p>
  <p>Then, you just stack these for all your outputs, giving you the shape you got.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-10T21:13:15.407Z
Answers.LastActivityDate: 2020-09-10T21:13:15.407Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>Let's suppose we have calculated the gradient and it came out to be <span class="math-container">$f(WX)(1-f(W X))X$</span>, where <span class="math-container">$f()$</span> is the sigmoid function, <span class="math-container">$W$</span> of order <span class="math-container">$2\times2$</span> is the weight matrix, and <span class="math-container">$X$</span> is an input vector of order <span class="math-container">$2\times 1$</span>. For ease let <span class="math-container">$f(WX)(1-f(W X))=\Bigg[
  \begin{array}{c}
  0.3 \\
  0.8 \\
  \end{array}\Bigg]$</span> and <span class="math-container">$X=\Bigg[
  \begin{array}{c}
  1 \\
  0 \\
  \end{array}\Bigg]$</span>. When we multiply these vectors we will multiply them as <span class="math-container">$f(WX)(1-f(W X))\times X^T$</span> i.e <span class="math-container">$\Bigg[
  \begin{array}{c}
  0.3 \\
  0.8 \\
  \end{array}\Bigg]\times[1 \quad0]$</span>. I do this because I know that we need this gradient to update a <span class="math-container">$2\times 2$</span> weight matrix, hence, the gradient should have size <span class="math-container">$2\times 2$</span>. But, I don't know the law/rule behind this, if I was just given the values and had no knowledge that we need the solution to update the weight matrix, then, I might have done something like <span class="math-container">$[0.3 \quad 0.8]\times\Bigg[
  \begin{array}{c}
  1 \\
  0 \\
  \end{array}\Bigg]$</span> which will return a scalar. For a long chain of such operations (multiple derivatives in applying chain rule, resulting in many vectors), how do we know if the multiplication of two vectors should return a vector or matrix (dot or cross product)?</p>
  
Tags: <neural-networks><recurrent-neural-networks><backpropagation><deep-neural-networks><gradient-descent>
Title: What are the rules behind vector product in gradient?
UserDisplayName: ZZ1
ThreadText: # What are the rules behind vector product in gradient?
  
  (Asked by: username_0 on 2020-09-09)
  
  Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of order $2\times 1$. For ease let $f(WX)(1-f(W X))=\Bigg[
  \begin{array}{c}
  0.3 \\
  0.8 \\
  \end{array}\Bigg]$ and $X=\Bigg[
  \begin{array}{c}
  1 \\
  0 \\
  \end{array}\Bigg]$. When we multiply these vectors we will multiply them as $f(WX)(1-f(W X))\times X^T$ i.e $\Bigg[
  \begin{array}{c}
  0.3 \\
  0.8 \\
  \end{array}\Bigg]\times[1 \quad0]$. I do this because I know that we need this gradient to update a $2\times 2$ weight matrix, hence, the gradient should have size $2\times 2$. But, I don't know the law/rule behind this, if I was just given the values and had no knowledge that we need the solution to update the weight matrix, then, I might have done something like $[0.3 \quad 0.8]\times\Bigg[
  \begin{array}{c}
  1 \\
  0 \\
  \end{array}\Bigg]$ which will return a scalar. For a long chain of such operations (multiple derivatives in applying chain rule, resulting in many vectors), how do we know if the multiplication of two vectors should return a vector or matrix (dot or cross product)?
  
  **Tags:** neural-networks, recurrent-neural-networks, backpropagation, deep-neural-networks, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-10. Score: 0 | Accepted answer
  
  It helps to think of each output dimension separately.
  
  You have $X$ which is an $(2 \times 1)$ vector, and $W\_1$ is a $(1 \times 2 )$ vector. Their product is a scalar, of which we then take the sigmoid to get our output $Y\_1$.
  
  The gradient of this w.r.t. $W\_1$ will be $f(WX) (1 - f(WX)) X^T$, which has the appropriate dimensions.
  
  Then, you just stack these for all your outputs, giving you the shape you got.
  
  
  ---
  
  