Answers.Body: <p>I have consistently found Adam to work very well but to tell you the truth I have not seen all that much difference in performance based on the optimizer. Other factor seem to have much more influence on the final model performance.In particular adjusting the learning rate during training can be very effective. Also saving the weights for the lowest validation loss and loading the model with those weights to make predictions works very well. Keras provides two callbacks that help you achieve this. Documentation is at 
  <a href="https://keras.io/callbacks/" rel="nofollow noreferrer">https://keras.io/callbacks/</a>. The ReduceLROnPlateau callback allows you to adjust the learning rate based on monitoring a metric. Typically validation loss is monitored. If the loss fails to reduce after N consecutive epochs(parameter patience) the learning rate is adjusted by a factor(parameter factor). You can think of training as descending into a valley which gets more and more narrow as you approach the bottom. If the learning rate does not adjust to this "narrowness" there is no way you will get to the very bottom.
  The other callback is ModelCheckpoint. This allows you to save the model( or just the weights) based on monitoring of a metric. Again usually validation loss is monitored and the parameter save_best_only is set to true. This saves the model with the lowest validation loss. That model can than be used to make predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T22:29:23.827Z
Answers.LastActivityDate: 2020-03-02T22:29:23.827Z
Answers.UserDisplayName: Gerry P
Answers.Body: <p>The fact that images are similar to each other or the fact that you are using binray classification, don't give you a particular choice of Optimizer, when an optimization algorithm is developped, those information are not taken into account. What is taken into account is the nature of the function we want to optimize (Is it smooth, convex, strongly convex, are stochastic gradient noisy...) The most used optimizer by far is ADAM, under some assumptions on the boundness of the gradient of the objective function, this <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="nofollow noreferrer">paper</a> gives the convergence rate of ADAM, they also provide experimental to validate that ADAM is better then some other optimizers. Some other  <a href="http://cs229.stanford.edu/proj2015/054_report.pdf" rel="nofollow noreferrer">works</a> propose to mix adam with nestrov mommentum acceleration.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-25T10:00:01.633Z
Answers.LastActivityDate: 2020-02-25T10:00:01.633Z
Answers.UserDisplayName: hola
Answers.Body: <p>If you are using a shallow neural network SGD would be better, ADAM optimizer will give you a soon overfitting.
  but be careful about choosing the learning rate. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T02:15:22.127Z
Answers.Text: I wish you could provide some sources, I could find what you wrote in no time
Answers.UserDisplayName: bit_scientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T10:23:07.293Z
Answers.Text: @bit_scientist you can check this link https://keras.io/optimizers/
Answers.UserDisplayName: SahaTib
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-25T10:59:56.550Z
Answers.LastActivityDate: 2020-02-25T10:59:56.550Z
Answers.UserDisplayName: SahaTib
Body: <p>I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are <strong>similar</strong> to one another. </p>
  
  <p>I'd like to read some theoretical proofs (if any) to convince myself that particular optimization has better results over the rest. </p>
  
  <p>And, similarly, what kind of optimizer is better for binary classification when images are <strong>very different</strong> from each other?</p>
  
Tags: <deep-learning><classification><optimization><hyperparameter-optimization><optimizers>
Title: What kind of optimizer is suggested to use for binary classification of similar images?
UserDisplayName: bit_scientist
ThreadText: # What kind of optimizer is suggested to use for binary classification of similar images?
  
  (Asked by: username_0 on 2020-02-24)
  
  I have spent some time searching Google and wasn't able to find out what kind of optimization algorithm is best for binary classification when images are **similar** to one another.
  
  I'd like to read some theoretical proofs (if any) to convince myself that particular optimization has better results over the rest.
  
  And, similarly, what kind of optimizer is better for binary classification when images are **very different** from each other?
  
  **Tags:** deep-learning, classification, optimization, hyperparameter-optimization, optimizers
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-03-02. Score: 3
  
  I have consistently found Adam to work very well but to tell you the truth I have not seen all that much difference in performance based on the optimizer. Other factor seem to have much more influence on the final model performance.In particular adjusting the learning rate during training can be very effective. Also saving the weights for the lowest validation loss and loading the model with those weights to make predictions works very well. Keras provides two callbacks that help you achieve this. Documentation is at
  <https://keras.io/callbacks/>. The ReduceLROnPlateau callback allows you to adjust the learning rate based on monitoring a metric. Typically validation loss is monitored. If the loss fails to reduce after N consecutive epochs(parameter patience) the learning rate is adjusted by a factor(parameter factor). You can think of training as descending into a valley which gets more and more narrow as you approach the bottom. If the learning rate does not adjust to this "narrowness" there is no way you will get to the very bottom.
  The other callback is ModelCheckpoint. This allows you to save the model( or just the weights) based on monitoring of a metric. Again usually validation loss is monitored and the parameter save\_best\_only is set to true. This saves the model with the lowest validation loss. That model can than be used to make predictions.
  
  
  ---
  
  ### Answer by username_2 on 2020-02-25. Score: 1
  
  The fact that images are similar to each other or the fact that you are using binray classification, don't give you a particular choice of Optimizer, when an optimization algorithm is developped, those information are not taken into account. What is taken into account is the nature of the function we want to optimize (Is it smooth, convex, strongly convex, are stochastic gradient noisy...) The most used optimizer by far is ADAM, under some assumptions on the boundness of the gradient of the objective function, this [paper](https://arxiv.org/pdf/1412.6980.pdf) gives the convergence rate of ADAM, they also provide experimental to validate that ADAM is better then some other optimizers. Some other [works](http://cs229.stanford.edu/proj2015/054_report.pdf) propose to mix adam with nestrov mommentum acceleration.
  
  
  ---
  
  ### Answer by username_3 on 2020-02-25. Score: 1
  
  If you are using a shallow neural network SGD would be better, ADAM optimizer will give you a soon overfitting.
  but be careful about choosing the learning rate.
  
  > **Comments:**
  > **username_0**: I wish you could provide some sources, I could find what you wrote in no time  
  > **username_3**: @bit_scientist you can check this link https://keras.io/optimizers/  
  
  ---
  
  