Answers.Body: <p><em>Adam is generalizing much better</em> hard to say with plots with different y-scale</p>
  <p>Anyway, you should not rely on a single benchmark, and a single network performance to draw conclusions, and actually, there are works showing the opposites (&quot;Understanding the Generalization of Adam and SGD in Neural Networks&quot;, &quot;On the Generalization Benefit of Noise in Stochastic Gradient Descent&quot;)</p>
  <p>Generally speaking, they usually question the fact that SGD has larger noise in the gradients that act as a regularizer, thus forcing the network to be attracted to minima that are less sharp (and thus hypothetically more robust)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-09T15:31:06.203Z
Answers.Text: Thank you for your answer. I agree with your first point. My observations are indeed insufficient for drawing a general conclusion. You are also right about the comparison between Adam and SGD. However, here I am trying to compare Adam with MBGD, and not SGD. In this case, updates are done on mini-batches instead of a single data (as in SGD), so I wouldn't expect the gradients to be specifically noisier than Adam.
Answers.UserDisplayName: Negarina.c
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-09T16:39:46.230Z
Answers.Text: @Negarina.c Minibatch GD and stochastic GD are pretty much the same, apart from the independence of mini-batches. But still, you are considering a subset of data, thus the noise is still there, both for GD and for Adam
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-29T12:08:31.497Z
Answers.LastActivityDate: 2024-08-29T12:08:31.497Z
Answers.UserDisplayName: Alberto
Body: <p>I am currently training a 3D CNN classifier on preprocessed EEG data. I tried 3 optimization algorithms for comparison, namely Mini-Batch Gradient Descent with Momentum (MBGD-M), RMSprop, and Adam. Below are the accuracy and loss graphs for the training and validation datasets:
  <a href="https://i.sstatic.net/tXidDlyf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tXidDlyf.png" alt="enter image description here" /></a>
  I do understand the reason behind the differences between the convergence speeds of the algorithms and didn't expect otherwise. However, I am puzzled as to why Adam is generalizing much better than RMSprop and especially MBGD-M.
  Unfortunately, I couldn't find any resources regarding the generalizing ability of these algorithms (Some argued SGD generalizes better than Adam, however in this case, all the algorithms use the same mini-batch size).</p>
  <p>Any insights would be greatly appreciated.</p>
  
Tags: <optimization>
Title: Generalization comparison between MBGD-M, RMSprop and Adam
UserDisplayName: Negarina.c
ThreadText: # Generalization comparison between MBGD-M, RMSprop and Adam
  
  (Asked by: username_0 on 2024-08-26)
  
  I am currently training a 3D CNN classifier on preprocessed EEG data. I tried 3 optimization algorithms for comparison, namely Mini-Batch Gradient Descent with Momentum (MBGD-M), RMSprop, and Adam. Below are the accuracy and loss graphs for the training and validation datasets:
  [![enter image description here](https://i.sstatic.net/tXidDlyf.png)](https://i.sstatic.net/tXidDlyf.png)
  I do understand the reason behind the differences between the convergence speeds of the algorithms and didn't expect otherwise. However, I am puzzled as to why Adam is generalizing much better than RMSprop and especially MBGD-M.
  Unfortunately, I couldn't find any resources regarding the generalizing ability of these algorithms (Some argued SGD generalizes better than Adam, however in this case, all the algorithms use the same mini-batch size).
  
  Any insights would be greatly appreciated.
  
  **Tags:** optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-29. Score: 1
  
  *Adam is generalizing much better* hard to say with plots with different y-scale
  
  Anyway, you should not rely on a single benchmark, and a single network performance to draw conclusions, and actually, there are works showing the opposites ("Understanding the Generalization of Adam and SGD in Neural Networks", "On the Generalization Benefit of Noise in Stochastic Gradient Descent")
  
  Generally speaking, they usually question the fact that SGD has larger noise in the gradients that act as a regularizer, thus forcing the network to be attracted to minima that are less sharp (and thus hypothetically more robust)
  
  > **Comments:**
  > **username_0**: Thank you for your answer. I agree with your first point. My observations are indeed insufficient for drawing a general conclusion. You are also right about the comparison between Adam and SGD. However, here I am trying to compare Adam with MBGD, and not SGD. In this case, updates are done on mini-batches instead of a single data (as in SGD), so I wouldn't expect the gradients to be specifically noisier than Adam.  
  > **username_1**: @Negarina.c Minibatch GD and stochastic GD are pretty much the same, apart from the independence of mini-batches. But still, you are considering a subset of data, thus the noise is still there, both for GD and for Adam  
  
  ---
  
  