Answers.Body: <p>I think I understand the process now. Let <span class="math-container">$x$</span> has the dimension <span class="math-container">$(n, p)$</span> and W has the dimension <span class="math-container">$(p, q)$</span>. In neural network, <span class="math-container">$n$</span> denotes the samples in the batch and <span class="math-container">$p, q$</span> denotes input dimension and output dimension of each layer, respectively. We don't use bias <span class="math-container">$b$</span> here just for simplicity.</p>
  <p>I had trouble in understanding the process of differentiation because of some confusion in vector(matrix) derivative. By changing matrix as set of column vector, now I can solve the problem.</p>
  <p>Let <span class="math-container">$x$</span> = <span class="math-container">$\begin{bmatrix}
  x_{11}, &amp;x_{12}, &amp;\cdots, &amp;x_{1p}\\
  x_{21}, &amp;x_{22}, &amp;\cdots, &amp;x_{2p}\\
  \vdots, &amp;\vdots, &amp;\vdots, &amp;\vdots\\
  x_{n1}, &amp;x_{n2}, &amp;\cdots, &amp;x_{np}
  \end{bmatrix}$</span>, which can be written as <span class="math-container">$\begin{bmatrix}
  \vec{x}_{1}, &amp;\vec{x}_{2}, &amp;\cdots, &amp;\vec{x}_{p}
  \end{bmatrix}$</span>.</p>
  <p>Similarly, <span class="math-container">$W$</span> = <span class="math-container">$\begin{bmatrix}
  w_{11}, &amp;w_{12}, &amp;\cdots, &amp;w_{1q}\\
  w_{21}, &amp;w_{22}, &amp;\cdots, &amp;w_{2q}\\
  \vdots, &amp;\vdots, &amp;\vdots, &amp;\vdots\\
  w_{p1}, &amp;w_{n2}, &amp;\cdots, &amp;w_{pq}
  \end{bmatrix}$</span> = <span class="math-container">$\begin{bmatrix}
  \vec{w}_{1}\\ \vec{w}_{2}, \\ \vdots, \\ \vec{w}_{p}
  \end{bmatrix}$</span>.</p>
  <p>Then <span class="math-container">$x\times W$</span> = <span class="math-container">$\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$</span>. So <span class="math-container">$\frac{\partial(\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p)}{\partial{W}} = \begin{bmatrix}
  \vec{x}_{1}, &amp;\vec{x}_{2}, &amp;\cdots, &amp;\vec{x}_{p}
  \end{bmatrix}^T.$</span></p>
  <p>I still have a little trouble in understanding the last part since actually, <span class="math-container">$\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$</span> is not scalar, it's <span class="math-container">$(n, q)$</span> matrix, but at least now I understand why it is written as above result.</p>
  <p>(Also hope to fully understand the last part.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-19T02:09:44.630Z
Answers.LastActivityDate: 2021-04-19T02:09:44.630Z
Answers.UserDisplayName: 최선웅
Body: <p>Let <span class="math-container">$\sigma(x)$</span> be sigmoid function. Consider the case where <span class="math-container">$\text{out}=\sigma(\vec{x} \times W + \vec{b})$</span>, and we want to compute <span class="math-container">$\frac{\partial{\text{out}}}{\partial{w}
  }.$</span><br />
  Set the dimension as belows:<br />
  <span class="math-container">$\vec{x}$</span>: <span class="math-container">$(n, n_{\text{in}})$</span>, <span class="math-container">$W$</span>: <span class="math-container">$(n_{\text{in}}, n_{\text{out}})$</span>, <span class="math-container">$\vec{b}$</span>: <span class="math-container">$(1, n_{\text{out}})$</span>.<br />
  Then <span class="math-container">$\text{out}$</span> has the dimension <span class="math-container">$(n, n_{\text{out}})$</span>. So we need to calculate the matrix by matrix derivative, as I know there is no such way to define that.
  I know that finally it is calculated as <span class="math-container">$\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$</span>.<br />
  But I can't still get the exact procedure of calculation, why it should be <span class="math-container">$\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$</span>, not <span class="math-container">$(\text{out}\cdot(1-\text{out})) \times \vec{x}^T$</span>,I know it by considering dimension, but not by calculation.</p>
  <p>My intuition about this problem is that all calculation can be considerd as vector by vector differentiation since <span class="math-container">$n$</span> is a batch size number, we can calculate matrix differentiation by considering each column vector.</p>
  <p>I'm not sure about my intuition yet, and I need some exact mathematical calculation procedure for the problem,</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T12:25:46.037Z
Comments.Text: What is $n$ in your context? The number of samples/examples in your batch that you pass to the net? When you say "I know that finally it is calculated as...", what exactly are you talking about? Do you mean the derivative that you want to calculate? Where does that formula come from?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-19T00:51:26.450Z
Comments.Text: 1.  menas the number of samples. 
  2. I know that ∂out / ∂ has that result, because of 'derivative of composition function' and I saw it on my textbook(which is based on some internet resources). It came from the 'derivative of sigmoid w.r.t (xW + b)' * 'derivative of xW + b w.r.t w ( = x.T)'.
  I'm just wondering that how can matrix-by-matrix derivative be defined, and ∂out / ∂ have those result.
  Simply, I can understand it when  = 1, but not in other case. Sorry for my english nonfluency.
Comments.UserDisplayName: 최선웅
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-19T02:10:16.813Z
Comments.Text: I self-answered to my question, will it be helpful for your question?
Comments.UserDisplayName: 최선웅
Tags: <gradient-descent><linear-algebra><calculus>
Title: How can the gradient of the weight be calculated in the viewpoint of matrix calculus?
UserDisplayName: 최선웅
ThreadText: # How can the gradient of the weight be calculated in the viewpoint of matrix calculus?
  
  (Asked by: username_0 on 2021-04-14)
  
  Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w}
  }.$  
  Set the dimension as belows:  
  $\vec{x}$: $(n, n\_{\text{in}})$, $W$: $(n\_{\text{in}}, n\_{\text{out}})$, $\vec{b}$: $(1, n\_{\text{out}})$.  
  Then $\text{out}$ has the dimension $(n, n\_{\text{out}})$. So we need to calculate the matrix by matrix derivative, as I know there is no such way to define that.
  I know that finally it is calculated as $\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$.  
  But I can't still get the exact procedure of calculation, why it should be $\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$, not $(\text{out}\cdot(1-\text{out})) \times \vec{x}^T$,I know it by considering dimension, but not by calculation.
  
  My intuition about this problem is that all calculation can be considerd as vector by vector differentiation since $n$ is a batch size number, we can calculate matrix differentiation by considering each column vector.
  
  I'm not sure about my intuition yet, and I need some exact mathematical calculation procedure for the problem,
  
  **Tags:** gradient-descent, linear-algebra, calculus
  
  > **Comments:**
  > **username_1**: What is $n$ in your context? The number of samples/examples in your batch that you pass to the net? When you say "I know that finally it is calculated as...", what exactly are you talking about? Do you mean the derivative that you want to calculate? Where does that formula come from?  
  > **username_0**: 1.  menas the number of samples.  2. I know that ∂out / ∂ has that result, because of 'derivative of composition function' and I saw it on my textbook(which is based on some internet resources). It came from the 'derivative of sigmoid w.r.t (xW + b)' * 'derivative of xW + b w.r.t w ( = x.T)'. I'm just wondering that how can matrix-by-matrix derivative be defined, and ∂out / ∂ have those result. Simply, I can understand it when  = 1, but not in other case. Sorry for my english nonfluency.  
  > **username_0**: I self-answered to my question, will it be helpful for your question?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-04-19. Score: 0
  
  I think I understand the process now. Let $x$ has the dimension $(n, p)$ and W has the dimension $(p, q)$. In neural network, $n$ denotes the samples in the batch and $p, q$ denotes input dimension and output dimension of each layer, respectively. We don't use bias $b$ here just for simplicity.
  
  I had trouble in understanding the process of differentiation because of some confusion in vector(matrix) derivative. By changing matrix as set of column vector, now I can solve the problem.
  
  Let $x$ = $\begin{bmatrix}
  x\_{11}, &x\_{12}, &\cdots, &x\_{1p}\\
  x\_{21}, &x\_{22}, &\cdots, &x\_{2p}\\
  \vdots, &\vdots, &\vdots, &\vdots\\
  x\_{n1}, &x\_{n2}, &\cdots, &x\_{np}
  \end{bmatrix}$, which can be written as $\begin{bmatrix}
  \vec{x}\_{1}, &\vec{x}\_{2}, &\cdots, &\vec{x}\_{p}
  \end{bmatrix}$.
  
  Similarly, $W$ = $\begin{bmatrix}
  w\_{11}, &w\_{12}, &\cdots, &w\_{1q}\\
  w\_{21}, &w\_{22}, &\cdots, &w\_{2q}\\
  \vdots, &\vdots, &\vdots, &\vdots\\
  w\_{p1}, &w\_{n2}, &\cdots, &w\_{pq}
  \end{bmatrix}$ = $\begin{bmatrix}
  \vec{w}\_{1}\\ \vec{w}\_{2}, \\ \vdots, \\ \vec{w}\_{p}
  \end{bmatrix}$.
  
  Then $x\times W$ = $\vec{x}\_1 \vec{w}\_1 + \vec{x}\_2 \vec{w}\_2 + \cdots \vec{x}\_p \vec{w}\_p$. So $\frac{\partial(\vec{x}\_1 \vec{w}\_1 + \vec{x}\_2 \vec{w}\_2 + \cdots \vec{x}\_p \vec{w}\_p)}{\partial{W}} = \begin{bmatrix}
  \vec{x}\_{1}, &\vec{x}\_{2}, &\cdots, &\vec{x}\_{p}
  \end{bmatrix}^T.$
  
  I still have a little trouble in understanding the last part since actually, $\vec{x}\_1 \vec{w}\_1 + \vec{x}\_2 \vec{w}\_2 + \cdots \vec{x}\_p \vec{w}\_p$ is not scalar, it's $(n, q)$ matrix, but at least now I understand why it is written as above result.
  
  (Also hope to fully understand the last part.)
  
  
  ---
  
  