Answers.Body: <p>Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence.</p>
  <p>Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on previous iterations squared. <a href="https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c" rel="nofollow noreferrer">https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c</a>.</p>
  <p>The loss surfaces of the neural networks is a difficult and poorly understood topic in the present. I suppose, that the poor convergence of SGD is caused by the roughness of loss surface, where the gradient makes big leaps, and jumps over the mimima. The adaptive learing strategy of the <code>Adam</code>, on the other hand, allows to dive into the valleys.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T21:11:52.813Z
Answers.LastActivityDate: 2021-02-02T21:11:52.813Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I am currently experimenting with the U-Net. I am doing semantic segmentation on the <a href="https://www.kaggle.com/c/data-science-bowl-2018" rel="nofollow noreferrer">2018 Data Science Bowl</a> dataset from Kaggle without any data augmentation.</p>
  <p>In my experiments, I am trying different hyper-parameters, like using Adam, mini-batch GD (MBGD), and batch normalization. Interestingly, all models with BN and/or Adam improve, while models without BN and with MBGD do not.</p>
  <p>How could this be explained? If it is due to the internal covariate shift, the Adam models without BN should not improve either, right?</p>
  <p>In the image below is the binary CE (BCE) train loss of my three models where the basic U-Net is blue, the basic U-Net with BN after every convolution is green, and the basic U-Net with Adam instead of MBGD is orange. The learning rate used in all models is 0.0001. I have also used other learning rates with worse results.</p>
  <p><img src="https://i.sstatic.net/oDE0s.png" alt="training loss (BCE) of basic U-Net (blue), basic U-Net with BN (green) and basic U-Net with Adam(orange)" /></p>
  
Tags: <u-net><batch-normalization><adam><mini-batch-gradient-descent><semantic-segmentation>
Title: Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
UserDisplayName: Bert Gayus
ThreadText: # Why does my model not improve when training with mini-batch gradient descent, while it does with Adam?
  
  (Asked by: username_0 on 2021-02-01)
  
  I am currently experimenting with the U-Net. I am doing semantic segmentation on the [2018 Data Science Bowl](https://www.kaggle.com/c/data-science-bowl-2018) dataset from Kaggle without any data augmentation.
  
  In my experiments, I am trying different hyper-parameters, like using Adam, mini-batch GD (MBGD), and batch normalization. Interestingly, all models with BN and/or Adam improve, while models without BN and with MBGD do not.
  
  How could this be explained? If it is due to the internal covariate shift, the Adam models without BN should not improve either, right?
  
  In the image below is the binary CE (BCE) train loss of my three models where the basic U-Net is blue, the basic U-Net with BN after every convolution is green, and the basic U-Net with Adam instead of MBGD is orange. The learning rate used in all models is 0.0001. I have also used other learning rates with worse results.
  
  ![training loss (BCE) of basic U-Net (blue), basic U-Net with BN (green) and basic U-Net with Adam(orange)](https://i.sstatic.net/oDE0s.png)
  
  **Tags:** u-net, batch-normalization, adam, mini-batch-gradient-descent, semantic-segmentation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-02. Score: 1 | Accepted answer
  
  Well, some time ago I also faced the same issue in the semantic segmentation task. Batch normalization is expected to improve convergence, because the normalization of activations prevents the explosion of the gradients magnitude and leads to more steady convergence.
  
  Adam is an adaptive optimizer with momentum and division by the weighted sum of gradients on previous iterations squared. <https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c>.
  
  The loss surfaces of the neural networks is a difficult and poorly understood topic in the present. I suppose, that the poor convergence of SGD is caused by the roughness of loss surface, where the gradient makes big leaps, and jumps over the mimima. The adaptive learing strategy of the `Adam`, on the other hand, allows to dive into the valleys.
  
  
  ---
  
  