Answers.Body: <p>In the usual scenario, case <strong>2</strong> occurs.
  In the deep learning frameworks, Tensors have a special dimension (usually corresponding to the 0 axes) which numerates the example in the batch.
  Look for example in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" rel="nofollow noreferrer">PyTorch documentation of Conv2d</a> or
  <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D" rel="nofollow noreferrer">Tensorflow documentation of Conv2d</a>. The same is true for any Layer - <code>Linear</code>, <code>MultiheadAttention</code>, <code>RNN</code>.</p>
  <p>All samples from the batch are processed at once, as the integral entity. Most operations process each sample from the batch independently, without a combination of features from <span class="math-container">$i^{th}$</span> and <span class="math-container">$j^{th}$</span> sample. Linear layer, Convolution doesn't construct linear combinations of inputs, corresponding to different samples.<br />
  However, there is an exception - the <code>BatchNormalization</code> Layer which subtracts the mean and divides by the standard deviation.</p>
  <p>However, one may want to work with large batches in order to have a less noisy and more precise estimate of the gradient at each step, but the memory usage increases linearly with the batch size since one has to allocate <code>batch size</code> times the memory, that is required for propagating a single sample through the network. In case, the computational resources do not allow for storing such a large batch in memory, one can pass a part of a large batch or even a single example, and only then aggregate the results.</p>
  <p>This situation corresponds to the case <strong>1</strong>.
  Such functionality is implemented in the <a href="https://pytorch-lightning.readthedocs.io/en/latest/advanced/training_tricks.html" rel="nofollow noreferrer">PyTorch-Lightning library </a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T09:55:10.773Z
Answers.LastActivityDate: 2022-05-25T00:48:24.287Z
Answers.LastEditDate: 2022-05-25T00:48:24.287Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed <a href="https://www.coursera.org/learn/deep-neural-network/lecture/qcogH/mini-batch-gradient-descent" rel="nofollow noreferrer">in this video</a> after 6:11.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T09:17:45.353Z
Answers.Text: Which one is correct #1 or #2?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T10:19:11.133Z
Answers.Text: I forgot about batch normalization layer, as mention by @spiridon_the_sun_rotator, which requires the process defined in option **2** given above to define a mean and variance at each layer, over each batch.
Answers.UserDisplayName: serali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T08:54:45.933Z
Answers.LastActivityDate: 2021-09-27T08:54:45.933Z
Answers.UserDisplayName: serali
Body: <p>In the stochastic gradient descent algorithm, the weight update happens for every training sample.</p>
  <p>In the mini-batch gradient descent algorithm, the weight update happens for every batch of training samples.</p>
  <p>In the batch gradient descent algorithm, the weight update happens for all samples in the training dataset.</p>
  <p>I am confused with the procedure of training that happens in the mini-batch gradient descent algorithm. I am guessing one of the following two must be correct</p>
  <ol>
  <li><p><strong>Passing each input individually at each layer</strong> and calculating the output. This happens for a number of training samples that are equal to batch size.</p>
  </li>
  <li><p><strong>Passing a batch of inputs at once at each layer</strong> and collecting the batch output at each layer.</p>
  </li>
  </ol>
  <p>Which of the above is true in general implementations of mini-batch gradient descent algorithms to train your neural networks?</p>
  
Tags: <neural-networks><deep-learning><gradient-descent><implementation><mini-batch-gradient-descent>
Title: In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?
UserDisplayName: hanugm
ThreadText: # In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?
  
  (Asked by: username_0 on 2021-09-27)
  
  In the stochastic gradient descent algorithm, the weight update happens for every training sample.
  
  In the mini-batch gradient descent algorithm, the weight update happens for every batch of training samples.
  
  In the batch gradient descent algorithm, the weight update happens for all samples in the training dataset.
  
  I am confused with the procedure of training that happens in the mini-batch gradient descent algorithm. I am guessing one of the following two must be correct
  
  1. **Passing each input individually at each layer** and calculating the output. This happens for a number of training samples that are equal to batch size.
  2. **Passing a batch of inputs at once at each layer** and collecting the batch output at each layer.
  
  Which of the above is true in general implementations of mini-batch gradient descent algorithms to train your neural networks?
  
  **Tags:** neural-networks, deep-learning, gradient-descent, implementation, mini-batch-gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-09-27. Score: 1 | Accepted answer
  
  In the usual scenario, case **2** occurs.
  In the deep learning frameworks, Tensors have a special dimension (usually corresponding to the 0 axes) which numerates the example in the batch.
  Look for example in the [PyTorch documentation of Conv2d](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html) or
  [Tensorflow documentation of Conv2d](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D). The same is true for any Layer - `Linear`, `MultiheadAttention`, `RNN`.
  
  All samples from the batch are processed at once, as the integral entity. Most operations process each sample from the batch independently, without a combination of features from $i^{th}$ and $j^{th}$ sample. Linear layer, Convolution doesn't construct linear combinations of inputs, corresponding to different samples.  
  However, there is an exception - the `BatchNormalization` Layer which subtracts the mean and divides by the standard deviation.
  
  However, one may want to work with large batches in order to have a less noisy and more precise estimate of the gradient at each step, but the memory usage increases linearly with the batch size since one has to allocate `batch size` times the memory, that is required for propagating a single sample through the network. In case, the computational resources do not allow for storing such a large batch in memory, one can pass a part of a large batch or even a single example, and only then aggregate the results.
  
  This situation corresponds to the case **1**.
  Such functionality is implemented in the [PyTorch-Lightning library](https://pytorch-lightning.readthedocs.io/en/latest/advanced/training_tricks.html) .
  
  
  ---
  
  ### Answer by username_2 on 2021-09-27. Score: 0
  
  What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed [in this video](https://www.coursera.org/learn/deep-neural-network/lecture/qcogH/mini-batch-gradient-descent) after 6:11.
  
  > **Comments:**
  > **username_0**: Which one is correct #1 or #2?  
  > **username_2**: I forgot about batch normalization layer, as mention by @spiridon_the_sun_rotator, which requires the process defined in option **2** given above to define a mean and variance at each layer, over each batch.  
  
  ---
  
  