Answers.Body: <p>So, two things:</p>
  <ol>
  <li><p>Yes you technically calculate the accumulated error for each neuron, but in practice we calculate all neurons at once by treating it as a matrix instead of individual vectors. Likewise, you do not have to calculate error gradients for the individual hidden neurons, but rather you calculate the gradients on a layer-wise basis which can be decomposed into individual neurons.</p>
  </li>
  <li><p>The size of your minibatches and the number of samples per update step are not necessarily the same thing. While you can perform backprop and weight updates after each minibatch, you can also perform what's called gradient accumulation where, as the name says, you accumulate the gradients between several (or all) minibatches before performing your update step. Accumulating the gradients for all minibatches before performing your update step effectively makes it so you're doing batch gradient descent, even if you don't have the hardware to do &quot;true&quot; batch gradient descent on data of your size.</p>
  </li>
  </ol>
  <p>How you select your minibatch size and number of gradient accumulation steps (minibatches per update step) is an empirical process, for now.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-26T17:47:10.640Z
Answers.Text: But then, if I'm not using matrices, does this mean that after processing each sample (in feedforward), I calculate the error of that sample for each of the output neurons individually?  And so at the end of the minibatch, I would have the following errors accumulated from each output neuron individually, for example if I have 2 output neurons:
  
  errorN1 = sum(...samplesErrorsN1), errorN2 = sum(...samplesErrorsN2), etc...
  
   In minibatch, For each output neuron I calculate the sum of the errors of all samples, so I have the accumulated error of each output neuron individually? Am I correct?
Answers.UserDisplayName: will The J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-26T16:24:08.713Z
Answers.LastActivityDate: 2023-12-26T16:24:08.713Z
Answers.UserDisplayName: Andy
Body: <p>In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the accumulated error.  In other neural networks, for example in Adaline, I know that something similar is done to update the weights, for example in this video: <a href="https://youtu.be/MTe2qsS56MQ?si=Sky-220_zA15l7T8" rel="nofollow noreferrer">https://youtu.be/MTe2qsS56MQ?si=Sky-220_zA15l7T8</a>
  In this case, it shows the &quot; total error&quot;, from what it seems to me, the weights are only adjusted at the end of each epoch, using the accumulated error (summed, that is, the total error) of the samples.  In the example in the video, he used only one output neuron, as it was just one Adaline, and in the way it was done, it seems like it was just a single batch, as he made a sum of the squared errors of each sample and only at the end I updated the weights using this accumulated error as I mentioned above, it was just a batch.  But for me it's confusing, because, <strong>if I have a Multilayer Perceptron using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?</strong></p>
  <p>For example, if I divide 10 samples into 2 batches, how would the weights be updated?  Would it add up the error of all samples from each batch, and then update the weights (after the current batch)?  When using multiple classes, what would this weight update look like?  I would have to add the error (that is, obtain the accumulated error) of all output neurons, for example: &quot;total error of class 1&quot;, &quot;total error of class 2&quot;, &quot;total error of class 3&quot;, etc?</p>
  <p><strong>In Multilayer Perceptron,using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?</strong></p>
  
Tags: <neural-networks><mini-batch-gradient-descent>
Title: In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?
UserDisplayName: will The J
ThreadText: # In MLP, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?
  
  (Asked by: username_0 on 2023-12-24)
  
  In Multilayer Perceptron neural networks, I know that there are two types of training: online training, and batch training, which consists of dividing the samples and updating the weights using the accumulated error. In other neural networks, for example in Adaline, I know that something similar is done to update the weights, for example in this video: <https://youtu.be/MTe2qsS56MQ?si=Sky-220_zA15l7T8>
  In this case, it shows the " total error", from what it seems to me, the weights are only adjusted at the end of each epoch, using the accumulated error (summed, that is, the total error) of the samples. In the example in the video, he used only one output neuron, as it was just one Adaline, and in the way it was done, it seems like it was just a single batch, as he made a sum of the squared errors of each sample and only at the end I updated the weights using this accumulated error as I mentioned above, it was just a batch. But for me it's confusing, because, **if I have a Multilayer Perceptron using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?**
  
  For example, if I divide 10 samples into 2 batches, how would the weights be updated? Would it add up the error of all samples from each batch, and then update the weights (after the current batch)? When using multiple classes, what would this weight update look like? I would have to add the error (that is, obtain the accumulated error) of all output neurons, for example: "total error of class 1", "total error of class 2", "total error of class 3", etc?
  
  **In Multilayer Perceptron,using several output classes, how would I update the weights using batches? Would I have to calculate the accumulated error(of all samples) of each output neuron?**
  
  **Tags:** neural-networks, mini-batch-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-26. Score: 2 | Accepted answer
  
  So, two things:
  
  1. Yes you technically calculate the accumulated error for each neuron, but in practice we calculate all neurons at once by treating it as a matrix instead of individual vectors. Likewise, you do not have to calculate error gradients for the individual hidden neurons, but rather you calculate the gradients on a layer-wise basis which can be decomposed into individual neurons.
  2. The size of your minibatches and the number of samples per update step are not necessarily the same thing. While you can perform backprop and weight updates after each minibatch, you can also perform what's called gradient accumulation where, as the name says, you accumulate the gradients between several (or all) minibatches before performing your update step. Accumulating the gradients for all minibatches before performing your update step effectively makes it so you're doing batch gradient descent, even if you don't have the hardware to do "true" batch gradient descent on data of your size.
  
  How you select your minibatch size and number of gradient accumulation steps (minibatches per update step) is an empirical process, for now.
  
  > **Comments:**
  > **username_0**: But then, if I'm not using matrices, does this mean that after processing each sample (in feedforward), I calculate the error of that sample for each of the output neurons individually?  And so at the end of the minibatch, I would have the following errors accumulated from each output neuron individually, for example if I have 2 output neurons:  errorN1 = sum(...samplesErrorsN1), errorN2 = sum(...samplesErrorsN2), etc...   In minibatch, For each output neuron I calculate the sum of the errors of all samples, so I have the accumulated error of each output neuron individually? Am I correct?  
  
  ---
  
  