Answers.Body: <p>You are correct, but requires final words:</p>
  <p>In <strong>Batch</strong> GD, we take the <strong>average</strong> of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error).</p>
  <p>On the other hand, in the <strong>Stochastic</strong> GD, we take <strong>one</strong> training sample to go <strong>one step</strong> towards the optimum, then repeat the latter for every training sample, hence updating the parameters once per sample sequentially in every epoch (no average here). As you can expect, the training will be noisy and the error will be fluctuating.</p>
  <p>Lastly, the <strong>mini-batch</strong> GD, is somehow in between the first two methods, that is: the average of a different portion of the data, every time. This method would take the benefits of the previous two, <strong>not so noisy</strong>, yet can deal with <strong>less</strong> smooth error manifold.</p>
  <hr />
  <p>Personally, I memorize them in my mind by creating the following map:</p>
  <ol>
  <li><strong>Batch GD</strong> ≡ Average of All per Step ≡ More suitable for Convex Problems at the Risk of Converging directly to Minima = Heavyweight.</li>
  <li><strong>Stochastic GD</strong> ≡ Fluctuating &amp; Noisy ≡ Converges on the Long Run especially for Large Dataset ≡ Lightweight but Slower ≡ No Vectorization Possible (because one sample per time).</li>
  <li><strong>Mini-Batch GD</strong> ≡ Portion of Data per Step ≡ Mixture of Stochastic and Batch GD ≡ Less Fluctuation + Would work for Less Smooth Error Manifolds ≡ Faster Computation.</li>
  </ol>
  <hr />
  <p>Regarding the naming convention, I would understand the word &quot;Batch&quot; as a &quot;Set&quot; or &quot;Collection&quot;, hence the whole &quot;Data<strong>set</strong>&quot;. Consequently, &quot;Mini&quot; would go with the flow, to mean a &quot;Part of the Set&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T19:57:30.327Z
Answers.Text: I think that the term "batch GD" is sometimes used to refer to "mini-batch GD" and that "stochastic GD" may actually refer to "mini-batch GD". I think people should take context into account because others may be using these terms inconsistently (this is my impression).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-31T02:47:06.063Z
Answers.Text: @Yahya, when you say for 'batch GD' that 'we take the average of all the training data...' you specifically mean that from the total aggregate loss on all independent training samples which is then backpropagated, you take the average across all the data to make that single update to the parameters from a single derivative of the model params. Is that correct? Can you also please include in your answer the analogous description for how this is similarly done in Mini-batch.
Answers.UserDisplayName: Vass
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-31T19:10:50.257Z
Answers.LastActivityDate: 2021-07-31T21:16:10.350Z
Answers.LastEditDate: 2021-07-31T21:16:10.350Z
Answers.UserDisplayName: Yahya
Body: <p>While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.</p>
  <p>For this question, assume that your dataset has <span class="math-container">$n$</span> training samples and we divided it into <span class="math-container">$k$</span> batches with <span class="math-container">$\dfrac{n}{k}$</span> samples in each batch. So, it can be easily understood the word &quot;batch&quot; is generally used to refer to a <strong>portion</strong> of the dataset rather than the whole dataset.</p>
  <p>In <strong>batch gradient descent</strong>, we pass all the <span class="math-container">$n$</span> available training samples to the network and then calculates the gradients (only once). We can repeat this process several times.</p>
  <p>In <strong>mini-batch gradient descent</strong>, we pass <span class="math-container">$\dfrac{n}{k}$</span> training samples to the network and calculates the gradient. That is, we calculate the gradient once for a batch. We repeat this process with all <span class="math-container">$k$</span> batches of samples to complete an epoch. And we can repeat this process several times.</p>
  <p>In <strong>stochastic gradient descent</strong>, we pass one training sample to the network and calculates the gradient. That is, we calculate the gradient once for iteration. We repeat this process with all <span class="math-container">$n$</span> times to complete an epoch. And we can repeat this process several times.</p>
  <p>Batch gradient descent can be viewed as a mini-batch gradient descent with <span class="math-container">$k = 1$</span> and stochastic gradient descent can be viewed as a mini-batch gradient descent with <span class="math-container">$k = n$</span>.</p>
  <p>Am I correct regarding the usage of terms in the context provided above? If wrong then where did I go wrong?</p>
  <p>If correct, I am confused about the usage of the word &quot;batch&quot; in &quot;batch gradient descent&quot;. In fact, we do not need the concept of <em>batch</em> in <em>batch gradient descent</em> since we pass all the training samples before calculating gradient. In fact, there is no need for batch gradient descent to partition the training dataset into batches. Then why do we use the word &quot;batch&quot; in batch gradient descent? Similarly, we are using the word &quot;mini-batch&quot; in &quot;mini-batch gradient descent&quot;. In fact, we are passing a batch of samples before calculating the gradient. Then why it is called &quot;mini-batch&quot; gradient descent instead of &quot;batch&quot; gradient descent?</p>
  
Tags: <terminology><gradient-descent><stochastic-gradient-descent><mini-batch-gradient-descent>
Title: Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
UserDisplayName: hanugm
ThreadText: # Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
  
  (Asked by: username_0 on 2021-07-30)
  
  While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.
  
  For this question, assume that your dataset has $n$ training samples and we divided it into $k$ batches with $\dfrac{n}{k}$ samples in each batch. So, it can be easily understood the word "batch" is generally used to refer to a **portion** of the dataset rather than the whole dataset.
  
  In **batch gradient descent**, we pass all the $n$ available training samples to the network and then calculates the gradients (only once). We can repeat this process several times.
  
  In **mini-batch gradient descent**, we pass $\dfrac{n}{k}$ training samples to the network and calculates the gradient. That is, we calculate the gradient once for a batch. We repeat this process with all $k$ batches of samples to complete an epoch. And we can repeat this process several times.
  
  In **stochastic gradient descent**, we pass one training sample to the network and calculates the gradient. That is, we calculate the gradient once for iteration. We repeat this process with all $n$ times to complete an epoch. And we can repeat this process several times.
  
  Batch gradient descent can be viewed as a mini-batch gradient descent with $k = 1$ and stochastic gradient descent can be viewed as a mini-batch gradient descent with $k = n$.
  
  Am I correct regarding the usage of terms in the context provided above? If wrong then where did I go wrong?
  
  If correct, I am confused about the usage of the word "batch" in "batch gradient descent". In fact, we do not need the concept of *batch* in *batch gradient descent* since we pass all the training samples before calculating gradient. In fact, there is no need for batch gradient descent to partition the training dataset into batches. Then why do we use the word "batch" in batch gradient descent? Similarly, we are using the word "mini-batch" in "mini-batch gradient descent". In fact, we are passing a batch of samples before calculating the gradient. Then why it is called "mini-batch" gradient descent instead of "batch" gradient descent?
  
  **Tags:** terminology, gradient-descent, stochastic-gradient-descent, mini-batch-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-31. Score: 4
  
  You are correct, but requires final words:
  
  In **Batch** GD, we take the **average** of all training data to update the parameters, hence, one step per epoch. That's very valid if you have a convex problem (i.e. smooth error).
  
  On the other hand, in the **Stochastic** GD, we take **one** training sample to go **one step** towards the optimum, then repeat the latter for every training sample, hence updating the parameters once per sample sequentially in every epoch (no average here). As you can expect, the training will be noisy and the error will be fluctuating.
  
  Lastly, the **mini-batch** GD, is somehow in between the first two methods, that is: the average of a different portion of the data, every time. This method would take the benefits of the previous two, **not so noisy**, yet can deal with **less** smooth error manifold.
  
  ---
  
  Personally, I memorize them in my mind by creating the following map:
  
  1. **Batch GD** ≡ Average of All per Step ≡ More suitable for Convex Problems at the Risk of Converging directly to Minima = Heavyweight.
  2. **Stochastic GD** ≡ Fluctuating & Noisy ≡ Converges on the Long Run especially for Large Dataset ≡ Lightweight but Slower ≡ No Vectorization Possible (because one sample per time).
  3. **Mini-Batch GD** ≡ Portion of Data per Step ≡ Mixture of Stochastic and Batch GD ≡ Less Fluctuation + Would work for Less Smooth Error Manifolds ≡ Faster Computation.
  
  ---
  
  Regarding the naming convention, I would understand the word "Batch" as a "Set" or "Collection", hence the whole "Data**set**". Consequently, "Mini" would go with the flow, to mean a "Part of the Set".
  
  > **Comments:**
  > **username_2**: I think that the term "batch GD" is sometimes used to refer to "mini-batch GD" and that "stochastic GD" may actually refer to "mini-batch GD". I think people should take context into account because others may be using these terms inconsistently (this is my impression).  
  > **username_3**: @username_1, when you say for 'batch GD' that 'we take the average of all the training data...' you specifically mean that from the total aggregate loss on all independent training samples which is then backpropagated, you take the average across all the data to make that single update to the parameters from a single derivative of the model params. Is that correct? Can you also please include in your answer the analogous description for how this is similarly done in Mini-batch.  
  
  ---
  
  