Answers.Body: <p><strong>tl;dr:</strong> A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check <a href="https://arxiv.org/pdf/1206.5533.pdf" rel="noreferrer">this article</a> for more details.</p>
  
  <hr>
  
  <h3>A more detailed explanation...</h3>
  
  <p>If you have a small enough number of samples, you can let the network see all of the samples before updating its weights; this is called <a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="noreferrer">Gradient Descent</a>. The benefit from this is that you guarantee that the weights will be updated in the direction that reduces the training loss for the whole dataset. The downside is that it is computationally expensive and in most cases infeasible for deep neural nets.</p>
  
  <p>What is done in practice is that the network sees only a batch of the training data, instead of the whole dataset, before updating its weights. However, this technique does not guarantee that the network updates its weights in a way that will reduce the dataset's training loss; instead it reduces the batch's training loss, which might not the same thing. This adds noise to the training process, which can in some cases be a good thing, but requires the network to take too many steps to converge (this isn't a problem since each step is much faster).</p>
  
  <p>What you're saying is essentially training the network each time on a single sample. This is formally called <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="noreferrer">Stochastic Gradient Descent</a>, however the term is used more broadly to include any case where the network is trained on a subset of the whole training set. The problem with this approach is that it adds too much noise to the training process, causing it to require a lot more steps to actually converge.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T06:02:41.203Z
Answers.Text: *"...and in most cases infeasible for deep neural nets"* Do you mean that it would take a long time to train (e.g. one year instead of one week) or is there some other infeasibility?
Answers.UserDisplayName: Mateen Ulhaq
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T12:41:52.767Z
Answers.Text: You're restricted in terms of hardware. Typically a neural network is trained on a GPU which has limited memory and computational units. The bigger the batch size, the more memory and computation the network requires. Therefore after a certain batch size, you cannot train anymore on the GPU.
Answers.UserDisplayName: spurra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T18:10:51.173Z
Answers.Text: Stochastic Gradient Descent feels like another method to back propagation. Maybe both use partial derivatives but is it true that backpropagation and Gradient Descent are totally different things?
Answers.UserDisplayName: MScott
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T23:05:49.583Z
Answers.Text: @MScott these two are often confused with one another. Backpropagation is simply an algorithm for efficiently **computing the gradient** of the loss function w.r.t the model's parameters. Gradient Descent is an algorithm for using these gradients to **update the parameters** of the model, in order to minimize this loss. Algorithms like this are called optimization algorithms. SGD is just an extension of Gradient Descent and there are many others out there (e.g. Adam, Adadelta, Adagrad, RMSProp).
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-01T23:01:52.563Z
Answers.LastActivityDate: 2020-03-01T23:01:52.563Z
Answers.UserDisplayName: Djib2011
Answers.Body: <p>We basically distinguish between 3 forms of batch training:
  <span class="math-container">$$Loss_{minibatch} = \sum_{m} l_m(\mathbf{W},t_m) \;\;\; with \;m \;\epsilon \; M$$</span>
  where <code>M</code> is a (random) subset of the whole dataset.</p>
  <p><span class="math-container">$$Loss_{batch} = \sum_{b} l_m(\mathbf{W},t_b) \;\;\; with \;b \;\epsilon \; B$$</span>
  where <code>B</code> is the whole dataset.</p>
  <p><span class="math-container">$$Loss_{stochastic} = l_i(\mathbf{W},t_i) $$</span>
  where <code>i</code> is a single sample from the whole dataset.</p>
  <p>Here <code>t</code> is the target/label of a sample <code>m,b,i</code> and <code>W</code> are the network weights. The most common case today is usually minibatch training.</p>
  <p>When we are training(updating the weights of the neural network to optimize towards a lower Loss) we take the derivative of this loss function with respect to the weights W. This will give us the gradient of the NN which tells us how much and in what direction we should update each weight.</p>
  <p><span class="math-container">$$ \nabla L = \frac{dLoss_{minibatch}}{d \mathbf{W}} = \frac{d\sum_m l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \frac{d l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \nabla l_m$$</span></p>
  <p>As you can see here in the example of the minibatch case: the total gradient is the sum of the gradient of each sample in the minibatch. So why do you think the first elemnts do not have an impact on the weight update? Or do I understand you wrong?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T12:50:02.960Z
Answers.LastActivityDate: 2021-03-23T12:58:52.587Z
Answers.LastEditDate: 2021-03-23T12:58:52.587Z
Answers.UserDisplayName: Marcel_marcel1991
Body: <p>Why is a batch size needed to update the weights of a neural network?</p>
  
  <p>According to that <a href="https://www.youtube.com/watch?v=tIeHLnjs5U8&amp;t=278s" rel="nofollow noreferrer">Youtube Video from 3B1B</a>, the weights are updated by calculating the error between expectation and outcome of the neural net. Based on that, the chain rule is applied to calculate the new weights. </p>
  
  <p>Following that logic, why would I pass a complete batch through the net? The first entries wouldn't have an impact on the weighting.</p>
  
  <p>Do I need to define a batch size when I use backpropagation?</p>
  
Tags: <neural-networks><deep-learning><training><backpropagation><batch-size>
Title: What is the purpose of the batch size in neural networks?
UserDisplayName: MScott
ThreadText: # What is the purpose of the batch size in neural networks?
  
  (Asked by: username_0 on 2020-03-01)
  
  Why is a batch size needed to update the weights of a neural network?
  
  According to that [Youtube Video from 3B1B](https://www.youtube.com/watch?v=tIeHLnjs5U8&t=278s), the weights are updated by calculating the error between expectation and outcome of the neural net. Based on that, the chain rule is applied to calculate the new weights.
  
  Following that logic, why would I pass a complete batch through the net? The first entries wouldn't have an impact on the weighting.
  
  Do I need to define a batch size when I use backpropagation?
  
  **Tags:** neural-networks, deep-learning, training, backpropagation, batch-size
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-03-01. Score: 5
  
  **tl;dr:** A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check [this article](https://arxiv.org/pdf/1206.5533.pdf) for more details.
  
  ---
  
  ### A more detailed explanation...
  
  If you have a small enough number of samples, you can let the network see all of the samples before updating its weights; this is called [Gradient Descent](https://en.wikipedia.org/wiki/Gradient_descent). The benefit from this is that you guarantee that the weights will be updated in the direction that reduces the training loss for the whole dataset. The downside is that it is computationally expensive and in most cases infeasible for deep neural nets.
  
  What is done in practice is that the network sees only a batch of the training data, instead of the whole dataset, before updating its weights. However, this technique does not guarantee that the network updates its weights in a way that will reduce the dataset's training loss; instead it reduces the batch's training loss, which might not the same thing. This adds noise to the training process, which can in some cases be a good thing, but requires the network to take too many steps to converge (this isn't a problem since each step is much faster).
  
  What you're saying is essentially training the network each time on a single sample. This is formally called [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent), however the term is used more broadly to include any case where the network is trained on a subset of the whole training set. The problem with this approach is that it adds too much noise to the training process, causing it to require a lot more steps to actually converge.
  
  > **Comments:**
  > **username_2**: *"...and in most cases infeasible for deep neural nets"* Do you mean that it would take a long time to train (e.g. one year instead of one week) or is there some other infeasibility?  
  > **username_3**: You're restricted in terms of hardware. Typically a neural network is trained on a GPU which has limited memory and computational units. The bigger the batch size, the more memory and computation the network requires. Therefore after a certain batch size, you cannot train anymore on the GPU.  
  > **username_0**: Stochastic Gradient Descent feels like another method to back propagation. Maybe both use partial derivatives but is it true that backpropagation and Gradient Descent are totally different things?  
  > **username_1**: @MScott these two are often confused with one another. Backpropagation is simply an algorithm for efficiently **computing the gradient** of the loss function w.r.t the model's parameters. Gradient Descent is an algorithm for using these gradients to **update the parameters** of the model, in order to minimize this loss. Algorithms like this are called optimization algorithms. SGD is just an extension of Gradient Descent and there are many others out there (e.g. Adam, Adadelta, Adagrad, RMSProp).  
  
  ---
  
  ### Answer by username_4 on 2021-03-23. Score: 0
  
  We basically distinguish between 3 forms of batch training:
  $$Loss\_{minibatch} = \sum\_{m} l\_m(\mathbf{W},t\_m) \;\;\; with \;m \;\epsilon \; M$$
  where `M` is a (random) subset of the whole dataset.
  
  $$Loss\_{batch} = \sum\_{b} l\_m(\mathbf{W},t\_b) \;\;\; with \;b \;\epsilon \; B$$
  where `B` is the whole dataset.
  
  $$Loss\_{stochastic} = l\_i(\mathbf{W},t\_i) $$
  where `i` is a single sample from the whole dataset.
  
  Here `t` is the target/label of a sample `m,b,i` and `W` are the network weights. The most common case today is usually minibatch training.
  
  When we are training(updating the weights of the neural network to optimize towards a lower Loss) we take the derivative of this loss function with respect to the weights W. This will give us the gradient of the NN which tells us how much and in what direction we should update each weight.
  
  $$ \nabla L = \frac{dLoss\_{minibatch}}{d \mathbf{W}} = \frac{d\sum\_m l\_m(\mathbf{W},t\_m)}{d \mathbf{W}} = \sum\_m \frac{d l\_m(\mathbf{W},t\_m)}{d \mathbf{W}} = \sum\_m \nabla l\_m$$
  
  As you can see here in the example of the minibatch case: the total gradient is the sum of the gradient of each sample in the minibatch. So why do you think the first elemnts do not have an impact on the weight update? Or do I understand you wrong?
  
  
  ---
  
  