Answers.Body: <p>As you understand, <span class="math-container">$E$</span> is the definition of loss function.  This function defines square of the difference between weights applied to <span class="math-container">$X_i$</span>, namely output of the perception, and <span class="math-container">$Y_i$</span> the desired target value (as a ground truth). So, for finding the optimum <span class="math-container">$W$</span> we should find a one that make the derivative of the loss function zero, based on the given <span class="math-container">$X$</span>s and <span class="math-container">$Y$</span>s.</p>
  <p>As, the error function <span class="math-container">$E$</span> is a convex function (a square function which is always positive), our finding from making the error function zero must minimize the error function.</p>
  <p>So, just we need to compute <span class="math-container">$\frac{d}{dW} E(W)$</span> (derivative of the error function over <span class="math-container">$W$</span>). In general derivative of a function in the form of <span class="math-container">$(zx+b)^2$</span> based on <span class="math-container">$z$</span> is equal to <span class="math-container">$2 \times (zx+b) \times x$</span>. Similar to this case, you can do the same for matrices. Hence the derivative of each term in <span class="math-container">$E$</span> can be phrased as mentioned in the question, i.e., by getting derivative of each term of the sum in <span class="math-container">$E$</span> and rewriting in a matrix format (the mapping between your question and this example is <span class="math-container">$z = W$</span>, <span class="math-container">$x = X_i$</span>, and <span class="math-container">$b = -Y_i$</span>). Note that multiplication of <span class="math-container">$2$</span> in each term of the result has been removed in the case, as it does not impact on the root of the equation as a constant factor.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T15:34:34.397Z
Answers.LastActivityDate: 2024-01-03T15:46:38.070Z
Answers.LastEditDate: 2024-01-03T15:46:38.070Z
Answers.UserDisplayName: OmG
Answers.Body: <p>The idea is correct, the last formula is wrong. In general <span class="math-container">$X$</span> will not be square, usually one has much more data than parameters. The data points will also be in general position, so that <span class="math-container">$X$</span> has maximal rank and <span class="math-container">$XX^T$</span> is invertible. The equation then transforms to
  <span class="math-container">$$
  WXX^T=YX^T\implies W=YX^T(XX^T)^{-1}.
  $$</span>
  There is no way to get an <span class="math-container">$X^2$</span> into the formula.</p>
  <p>With a SVD <span class="math-container">$X=U\Sigma V^T$</span> where <span class="math-container">$U$</span> is square, <span class="math-container">$\Sigma$</span> invertible and <span class="math-container">$V$</span> isometric one gets <span class="math-container">$W=YV\Sigma^{-1}U^T$</span>, which is connected with the idea of a pseudo-inverse.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-04T07:27:36.057Z
Answers.Text: Thank you very much. But also can you propose a good reference about these operation to me? I want to study more in this topic.
Answers.UserDisplayName: hasanghaforian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-04T08:29:12.410Z
Answers.Text: What you are doing here is just linear regression. This should be in most linear algebra and statistics texts. See Moore-Penrose pseudo-inverse.
Answers.UserDisplayName: Lutz Lehmann
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T21:10:37.923Z
Answers.LastActivityDate: 2024-01-03T21:10:37.923Z
Answers.UserDisplayName: Lutz Lehmann
Body: <p>Suppose we have a perceptron without bias and <span class="math-container">$f(x) = x$</span> as activation function and matrices <span class="math-container">$X,Y,W$</span> that input training data are columns of matrix <span class="math-container">$X$</span>, <span class="math-container">$Y$</span> is targets matrix (columns are ordered with attention to the related inputs) and <span class="math-container">$W$</span> is the weights matrix in perceptron. Also <span class="math-container">$X_i$</span> denotes the <span class="math-container">$i$</span>th column in matrix <span class="math-container">$X$</span> and so on for other matrices. We know we want to minimize this:</p>
  <p><span class="math-container">$$
  E(W) = \sum\limits_{i} ||WX_i - Y_i||^{2} 
  $$</span></p>
  <p>In a class I saw someone said we can find the optimum <span class="math-container">$W$</span> by by calculating  <span class="math-container">$\frac{d}{dW}E(W)$</span>. Then with attention to the
  <span class="math-container">\begin{align*}
  ||V||^{2} = V^{T}V \ \ \ \ \ \ \ \ \ \ \ \ \ \ &amp; (1)
  \end{align*}</span>
  , when <span class="math-container">$V$</span> is a columnar vector, he continued:</p>
  <p><span class="math-container">\begin{align*}
  \frac{d}{dW}E(W) = 0 &amp; \Rightarrow (WX - Y)X^{T} = 0 \\
  &amp; \Rightarrow W = YX^{T}(X^{2})^{-1}
  \end{align*}</span></p>
  <p>He claimed that above formula gets the optimum <span class="math-container">$W$</span>. But I do not know is that true and how does he calculate <span class="math-container">$\frac{d}{dW}E(W)$</span>? If it is not true, is there any correct direct formula for achieving <span class="math-container">$W$</span>? Also can you provide references for more studying?</p>
  
Tags: <neural-networks><machine-learning><perceptron><derivative><single-layer-perceptron>
Title: Direct formula for calculating the optimum matrix which minimizes the perceptron error
UserDisplayName: hasanghaforian
ThreadText: # Direct formula for calculating the optimum matrix which minimizes the perceptron error
  
  (Asked by: username_0 on 2024-01-03)
  
  Suppose we have a perceptron without bias and $f(x) = x$ as activation function and matrices $X,Y,W$ that input training data are columns of matrix $X$, $Y$ is targets matrix (columns are ordered with attention to the related inputs) and $W$ is the weights matrix in perceptron. Also $X\_i$ denotes the $i$th column in matrix $X$ and so on for other matrices. We know we want to minimize this:
  
  $$
  E(W) = \sum\limits\_{i} ||WX\_i - Y\_i||^{2}
  $$
  
  In a class I saw someone said we can find the optimum $W$ by by calculating $\frac{d}{dW}E(W)$. Then with attention to the
  \begin{align\*}
  ||V||^{2} = V^{T}V \ \ \ \ \ \ \ \ \ \ \ \ \ \ & (1)
  \end{align\*}
  , when $V$ is a columnar vector, he continued:
  
  \begin{align\*}
  \frac{d}{dW}E(W) = 0 & \Rightarrow (WX - Y)X^{T} = 0 \\
  & \Rightarrow W = YX^{T}(X^{2})^{-1}
  \end{align\*}
  
  He claimed that above formula gets the optimum $W$. But I do not know is that true and how does he calculate $\frac{d}{dW}E(W)$? If it is not true, is there any correct direct formula for achieving $W$? Also can you provide references for more studying?
  
  **Tags:** neural-networks, machine-learning, perceptron, derivative, single-layer-perceptron
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-01-03. Score: 3 | Accepted answer
  
  The idea is correct, the last formula is wrong. In general $X$ will not be square, usually one has much more data than parameters. The data points will also be in general position, so that $X$ has maximal rank and $XX^T$ is invertible. The equation then transforms to
  $$
  WXX^T=YX^T\implies W=YX^T(XX^T)^{-1}.
  $$
  There is no way to get an $X^2$ into the formula.
  
  With a SVD $X=U\Sigma V^T$ where $U$ is square, $\Sigma$ invertible and $V$ isometric one gets $W=YV\Sigma^{-1}U^T$, which is connected with the idea of a pseudo-inverse.
  
  > **Comments:**
  > **username_0**: Thank you very much. But also can you propose a good reference about these operation to me? I want to study more in this topic.  
  > **username_2**: What you are doing here is just linear regression. This should be in most linear algebra and statistics texts. See Moore-Penrose pseudo-inverse.  
  
  ---
  
  ### Answer by username_1 on 2024-01-03. Score: 2
  
  As you understand, $E$ is the definition of loss function. This function defines square of the difference between weights applied to $X\_i$, namely output of the perception, and $Y\_i$ the desired target value (as a ground truth). So, for finding the optimum $W$ we should find a one that make the derivative of the loss function zero, based on the given $X$s and $Y$s.
  
  As, the error function $E$ is a convex function (a square function which is always positive), our finding from making the error function zero must minimize the error function.
  
  So, just we need to compute $\frac{d}{dW} E(W)$ (derivative of the error function over $W$). In general derivative of a function in the form of $(zx+b)^2$ based on $z$ is equal to $2 \times (zx+b) \times x$. Similar to this case, you can do the same for matrices. Hence the derivative of each term in $E$ can be phrased as mentioned in the question, i.e., by getting derivative of each term of the sum in $E$ and rewriting in a matrix format (the mapping between your question and this example is $z = W$, $x = X\_i$, and $b = -Y\_i$). Note that multiplication of $2$ in each term of the result has been removed in the case, as it does not impact on the root of the equation as a constant factor.
  
  
  ---
  
  