Answers.Body: <p>As you say, the outputs are modeled as a vector, each output in one vector component.</p>
  <p><strong>In regression problems:</strong></p>
  <p>The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:</p>
  <p><span class="math-container">$$E=\sum_i ||\mathbf{y_i}-\mathbf{\hat{y_i}}||^2 = \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}})$$</span></p>
  <p>where:</p>
  <ul>
  <li><span class="math-container">$\mathbf{y_i}$</span> vector is expected value for sample <span class="math-container">$i$</span> (note I do not use the same naming convention than the question).</li>
  <li><span class="math-container">$\mathbf{\hat{y_i}}$</span> vector is network output for same sample</li>
  <li>||.|| is vector norm</li>
  <li>product of vectors is scalar/inner product.</li>
  </ul>
  <p>The derivative respect some NN parameter <span class="math-container">$w$</span> is:</p>
  <p><span class="math-container">$$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}}) = -2 \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})\frac{\partial \mathbf{\hat{y_i}}}{\partial w} $$</span></p>
  <p>being <span class="math-container">$\frac{\partial \mathbf{\hat{y_i}}}{\partial w}$</span> the term that backtracking algorithm evaluates.</p>
  <p><strong>Multi-class classification problems:</strong></p>
  <p>Two options appears as most usual ones:</p>
  <ul>
  <li>a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.</li>
  <li>b) optimize entropy. In this case, an usual loss function is cross-entropy:</li>
  </ul>
  <p><span class="math-container">$$ E = - \sum_c p(c) log(\hat p(c)) \text{  [definition]} $$</span>
  <span class="math-container">$$ E = - \sum_i \sum_c p_i(c) log(\hat p_i(c)) = - \sum_i log(\hat p_i(c_i)) \text{  [average]} $$</span></p>
  <p>where:</p>
  <ul>
  <li><span class="math-container">$c$</span> is some class</li>
  <li><span class="math-container">$p(c)$</span> is probability of class <span class="math-container">$c$</span> in train dataset</li>
  <li><span class="math-container">$\hat p(c)$</span> is probability of class <span class="math-container">$c$</span> in network output</li>
  <li><span class="math-container">$i$</span> is sample number</li>
  <li><span class="math-container">$c_i$</span> is expected (correct) class of sample <span class="math-container">$i$</span></li>
  <li><span class="math-container">$p_i(c)$</span> is ground truth, usually taken as 1 if <span class="math-container">$c=c_i$</span>, 0 otherwise as in latest expression.</li>
  <li><span class="math-container">$\hat p_i(c)$</span> is net output for sample <span class="math-container">$i$</span> and class <span class="math-container">$c$</span>.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-21T09:21:44.280Z
Answers.LastActivityDate: 2020-10-22T09:21:56.770Z
Answers.LastEditDate: 2020-10-22T09:21:56.770Z
Answers.UserDisplayName: pasaba por aqui
Answers.Body: <p>Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: <a href="https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation">https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation</a></p>
  <p>Edit:</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-21T16:19:30.160Z
Answers.Text: Rather than linking to another post (which can be lost, e.g. if the author of that post deletes it), please, explain (preferably, in your own words, i.e. without just copying and pasting) the concepts (in this case, the cross-entropy loss function).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-21T04:45:35.200Z
Answers.LastActivityDate: 2020-10-21T12:10:16.293Z
Answers.LastEditDate: 2020-10-21T12:10:16.293Z
Answers.UserDisplayName: Cameron Chandler
Body: <p>Machine Learning books generally explains that the error calculated for a given sample <span class="math-container">$i$</span> is:</p>
  <p><span class="math-container">$e_i = y_i - \hat{y_i}$</span></p>
  <p>Where <span class="math-container">$\hat{y}$</span> is the target output and <span class="math-container">$y$</span> is the actual output given by the network. So, a loss function <span class="math-container">$L$</span> is calculated:</p>
  <p><span class="math-container">$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$</span></p>
  <p>The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with <span class="math-container">$m$</span> neurons in the output layer for a multiclass classification problem (generally one neuron per class).</p>
  <p>What changes in the equations above? Since we now have multiple outputs, both <span class="math-container">$e_i$</span> and <span class="math-container">$y_i$</span> should be a vector?</p>
  
Tags: <neural-networks><backpropagation><feedforward-neural-networks><multilayer-perceptrons>
Title: How is the error calculated with multiple output neurons in the neural network?
UserDisplayName: joann2555
ThreadText: # How is the error calculated with multiple output neurons in the neural network?
  
  (Asked by: username_0 on 2020-10-20)
  
  Machine Learning books generally explains that the error calculated for a given sample $i$ is:
  
  $e\_i = y\_i - \hat{y\_i}$
  
  Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:
  
  $L = \frac{1}{2N}\sum^{N}\_{i=1}(e\_i)^2$
  
  The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).
  
  What changes in the equations above? Since we now have multiple outputs, both $e\_i$ and $y\_i$ should be a vector?
  
  **Tags:** neural-networks, backpropagation, feedforward-neural-networks, multilayer-perceptrons
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-10-21. Score: 1
  
  As you say, the outputs are modeled as a vector, each output in one vector component.
  
  **In regression problems:**
  
  The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:
  
  $$E=\sum\_i ||\mathbf{y\_i}-\mathbf{\hat{y\_i}}||^2 = \sum\_i (\mathbf{y\_i}-\mathbf{\hat{y\_i}})(\mathbf{y\_i}-\mathbf{\hat{y\_i}})$$
  
  where:
  
  * $\mathbf{y\_i}$ vector is expected value for sample $i$ (note I do not use the same naming convention than the question).
  * $\mathbf{\hat{y\_i}}$ vector is network output for same sample
  * ||.|| is vector norm
  * product of vectors is scalar/inner product.
  
  The derivative respect some NN parameter $w$ is:
  
  $$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum\_i (\mathbf{y\_i}-\mathbf{\hat{y\_i}})(\mathbf{y\_i}-\mathbf{\hat{y\_i}}) = -2 \sum\_i (\mathbf{y\_i}-\mathbf{\hat{y\_i}})\frac{\partial \mathbf{\hat{y\_i}}}{\partial w} $$
  
  being $\frac{\partial \mathbf{\hat{y\_i}}}{\partial w}$ the term that backtracking algorithm evaluates.
  
  **Multi-class classification problems:**
  
  Two options appears as most usual ones:
  
  * a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.
  * b) optimize entropy. In this case, an usual loss function is cross-entropy:
  
  $$ E = - \sum\_c p(c) log(\hat p(c)) \text{ [definition]} $$
  $$ E = - \sum\_i \sum\_c p\_i(c) log(\hat p\_i(c)) = - \sum\_i log(\hat p\_i(c\_i)) \text{ [average]} $$
  
  where:
  
  * $c$ is some class
  * $p(c)$ is probability of class $c$ in train dataset
  * $\hat p(c)$ is probability of class $c$ in network output
  * $i$ is sample number
  * $c\_i$ is expected (correct) class of sample $i$
  * $p\_i(c)$ is ground truth, usually taken as 1 if $c=c\_i$, 0 otherwise as in latest expression.
  * $\hat p\_i(c)$ is net output for sample $i$ and class $c$.
  
  
  ---
  
  ### Answer by username_2 on 2020-10-21. Score: 1
  
  Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: <https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation>
  
  Edit:
  
  > **Comments:**
  > **username_3**: Rather than linking to another post (which can be lost, e.g. if the author of that post deletes it), please, explain (preferably, in your own words, i.e. without just copying and pasting) the concepts (in this case, the cross-entropy loss function).  
  
  ---
  
  