Answers.Body: <p>Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions. </p>
  
  <p>The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.</p>
  
  <p>You can check <a href="https://stats.stackexchange.com/questions/49528/batch-gradient-descent-versus-stochastic-gradient-descent">this</a> out for more on the differences between the three. Hope this helped!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-27T12:15:07.980Z
Answers.Text: Thanks, I am understanding this better. So, basically, BGD is not very used because I guess most loss functions have multiple minimas. Am I wrong?
  What other loss functions could I use in this particular example?
Answers.UserDisplayName: Pigna
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-27T14:32:43.343Z
Answers.Text: Yes. *What other loss functions could I use in this particular example* - You have used the difference as loss (*written as 'error' in your code*) if I'm not wrong? If so, you can also try out Magnitude of the difference, square of the difference, logarithmic cross-entropy - looking up common loss functions will give you an abundant number of potential ones.
Answers.UserDisplayName: ashenoy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-27T08:37:42.387Z
Answers.LastActivityDate: 2019-09-27T08:37:42.387Z
Answers.UserDisplayName: ashenoy
Answers.Body: <p>Because the computation of <strong>Batch Gradient Descent(BGD)</strong> is more stable(less fluctuated) than <strong>Mini-Batch Gradient Descent(MBGD)</strong> and <strong>Stochastic Gradient Descent(SGD)</strong>, so <strong>BGD</strong> less easily(more difficultly) escapes local minima than <strong>MBGD</strong> and <strong>SGD</strong>, resulting in poor learning even thouch <strong>BGD</strong> can get more accurate output than <strong>MBGD</strong> and <strong>SGD</strong>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-28T04:47:24.217Z
Answers.LastActivityDate: 2024-07-29T01:06:03.497Z
Answers.LastEditDate: 2024-07-29T01:06:03.497Z
Answers.UserDisplayName: Super Kai - Kazuya Ito
Body: <p>I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between Stochastic/Mini-Batch Gradient Descent and Batch Gradient Descent:</p>
  <p><a href="https://i.sstatic.net/Bkn4o.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Bkn4o.png" alt="A comparison between the prediction results" /></a></p>
  <p>The training data is a collection of point coordinates (x,y).
  The labels are 0s or 1s (below or above the parabola).
  <a href="https://i.sstatic.net/aAoqc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aAoqc.png" alt="enter image description here" /></a></p>
  <p>As a test, I am doing a classification task.
  My objective is to make the NN learn which points are above the parabola(yellow) and which points are below the parabola(purple).</p>
  <p>Here is the link to the notebook: <a href="https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb" rel="nofollow noreferrer">https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb</a></p>
  <ul>
  <li>Why is the Batch Gradient Descent performing so poorly with respect
  to the other two methods?</li>
  <li>Is it a bug? But how can it be since the code is almost identical to the<br />
  Mini-Batch Gradient Descent?</li>
  <li>I am using the same (randomly chosen with try and error) hyperparameters for all three<br />
  neural networks. Does Batch Gradient Descent need a more accurate<br />
  technique to find the correct hyperparameters? If yes, why so?</li>
  </ul>
  
Tags: <machine-learning><deep-learning><python><gradient-descent><batch-learning>
Title: Why is Batch Gradient Descent performing worse than Stochastic and Mini-Batch Gradient Descent?
UserDisplayName: Pigna
ThreadText: # Why is Batch Gradient Descent performing worse than Stochastic and Mini-Batch Gradient Descent?
  
  (Asked by: username_0 on 2019-09-26)
  
  I have implemented a neural network from scratch (only using numpy) and I am having problems understanding why the results are so different between Stochastic/Mini-Batch Gradient Descent and Batch Gradient Descent:
  
  [![A comparison between the prediction results](https://i.sstatic.net/Bkn4o.png)](https://i.sstatic.net/Bkn4o.png)
  
  The training data is a collection of point coordinates (x,y).
  The labels are 0s or 1s (below or above the parabola).
  [![enter image description here](https://i.sstatic.net/aAoqc.png)](https://i.sstatic.net/aAoqc.png)
  
  As a test, I am doing a classification task.
  My objective is to make the NN learn which points are above the parabola(yellow) and which points are below the parabola(purple).
  
  Here is the link to the notebook: <https://github.com/Pign4/ScratchML/blob/master/Neural%20Network.ipynb>
  
  * Why is the Batch Gradient Descent performing so poorly with respect
    to the other two methods?
  * Is it a bug? But how can it be since the code is almost identical to the  
    Mini-Batch Gradient Descent?
  * I am using the same (randomly chosen with try and error) hyperparameters for all three  
    neural networks. Does Batch Gradient Descent need a more accurate  
    technique to find the correct hyperparameters? If yes, why so?
  
  **Tags:** machine-learning, deep-learning, python, gradient-descent, batch-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-09-27. Score: 2 | Accepted answer
  
  Assuming the problem at hand is a classification (Above or Below parabola), this is probably because of the nature of Batch gradient descent. Since the gradient is being calculated on the whole batch, it tends to work well on only convex loss functions.
  
  The reason for why batch gradient descent is not working too great probably is because of the high number of minimas in the error manifold, ending up learning nothing relevant. You can change the loss function and observe the change in results, they might not be great (Batch GD usually isn't) but you'll be able to see differences.
  
  You can check [this](https://stats.stackexchange.com/questions/49528/batch-gradient-descent-versus-stochastic-gradient-descent) out for more on the differences between the three. Hope this helped!
  
  > **Comments:**
  > **username_0**: Thanks, I am understanding this better. So, basically, BGD is not very used because I guess most loss functions have multiple minimas. Am I wrong? What other loss functions could I use in this particular example?  
  > **username_1**: Yes. *What other loss functions could I use in this particular example* - You have used the difference as loss (*written as 'error' in your code*) if I'm not wrong? If so, you can also try out Magnitude of the difference, square of the difference, logarithmic cross-entropy - looking up common loss functions will give you an abundant number of potential ones.  
  
  ---
  
  ### Answer by username_2 on 2024-07-28. Score: 0
  
  Because the computation of **Batch Gradient Descent(BGD)** is more stable(less fluctuated) than **Mini-Batch Gradient Descent(MBGD)** and **Stochastic Gradient Descent(SGD)**, so **BGD** less easily(more difficultly) escapes local minima than **MBGD** and **SGD**, resulting in poor learning even thouch **BGD** can get more accurate output than **MBGD** and **SGD**.
  
  
  ---
  
  