Answers.Body: <p>I have experimented with this to a small degree and have not noticed that much of an impact.</p>
  <p>To date, Adam appears to give the best results on a variety of image data sets. I have found that &quot;adjusting&quot; the learning rate during training is an effective means of improving model performance and has more impact than the selection of the optimizer.</p>
  <p>Keras has two callbacks that are useful for this purpose. Documentation is at <a href="https://keras.io/callbacks/" rel="nofollow noreferrer">https://keras.io/callbacks/</a>. The <code>ModelCheckpoint</code> callback enables you to save the full model or just the model weights based on monitoring a metric. Typically, you monitor validation loss and set the parameter <code>save_best_only=True</code> to save the results for the lowest validation loss. The other useful callback is <code>ReduceLROnPlateau</code>, which allows you to adjust the learning rate based on monitoring a metric. Again, the metric usually monitored is the validation loss. If the loss fails to reduce after a user-set number of epochs (parameter patience), the learning rate will be adjusted by a user-set factor (parameter factor). You can think of the training process as traveling down a valley. As you near the bottom of the valley, it becomes more and more narrow. If your learning rate does not adjust to the &quot;narrowness&quot; there is no way you will get to the bottom of the valley.</p>
  <p>You can also write a custom callback to adjust the learning rate. I have done this and created one which first adjusts the learning rate based on monitoring the training loss until the training accuracy reaches 95%. Then it switches to adjust the learning rate based on monitoring the validation loss. It saves the model weights for the lowest validation loss and loads the model with these weights to make predictions. I have found this approach leads to faster training and higher accuracy.</p>
  <p>The fact is you can't tell if your model has converged on a global minimum or a local minimum. This is evidenced by the fact that, unless you take special efforts to inhibit randomization, you can get different results each time you run your model. The loss can be envisioned as a surface in <span class="math-container">$N$</span> space, where <span class="math-container">$N$</span> is the number of trainable parameters. Lord knows what that surface is like and where your initial parameter weights put you on that surface, plus how other random processes cause you to traverse that surface.</p>
  <p>As an example, I ran a model at least 20 times and got resultant losses that were very close to each there. Then I ran it again and got far better results for exactly the same data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T21:12:56.553Z
Answers.Text: Interesting bio and project goal, but consider potential for abuse - deliberately diagnosing or stigmatizing appearance. An arguably fair medical asset, but unsure a public utility.
Answers.UserDisplayName: OverLordGoldDragon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-02T21:53:52.260Z
Answers.LastActivityDate: 2020-09-12T15:14:35.917Z
Answers.LastEditDate: 2020-09-12T15:14:35.917Z
Answers.UserDisplayName: Gerry P
Answers.Body: <blockquote>
  <p>Does the choice of optimizer really matter in terms of training the model?</p>
  </blockquote>
  <p>Yes.</p>
  <blockquote>
  <p>Would the Adam optimizer be better than the Adadelta optimizer?</p>
  </blockquote>
  <p>Yes. (But sometimes, adadelta gives better result. Depends upon the dataset and fine-tune mechanism)</p>
  <blockquote>
  <p>Would they all basically be the same?</p>
  </blockquote>
  <p>No. Here is the <a href="https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c" rel="nofollow noreferrer">explanation</a></p>
  <blockquote>
  <p>Would some optimizers be better because they allow most of the weights to achieve their &quot;global&quot; minima?</p>
  </blockquote>
  <p>It's not possible to check whether the model achieved global minima or not practically.</p>
  <p>We can evaluate the model by either over-fitting or under-fitting using the training and validation set, and the generalization of the model with the test set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T14:38:10.367Z
Answers.LastActivityDate: 2020-09-12T15:16:08.537Z
Answers.LastEditDate: 2020-09-12T15:16:08.537Z
Answers.UserDisplayName: Sudhakar17
Body: <p>Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with their annotations. We want to train a model using YOLOv3.</p>
  <p>Does the choice of optimizer really matter in terms of training the model? Would the Adam optimizer be better than the Adadelta optimizer? Or would they all basically be the same?</p>
  <p>Would some optimizers be better because they allow most of the weights to achieve their &quot;global&quot; minima?</p>
  
Tags: <object-detection><object-recognition><stochastic-gradient-descent><adam><adadelta>
Title: Is the choice of the optimiser relevant when doing object detection?
UserDisplayName: neeraj
ThreadText: # Is the choice of the optimiser relevant when doing object detection?
  
  (Asked by: username_0 on 2020-02-24)
  
  Suppose that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with their annotations. We want to train a model using YOLOv3.
  
  Does the choice of optimizer really matter in terms of training the model? Would the Adam optimizer be better than the Adadelta optimizer? Or would they all basically be the same?
  
  Would some optimizers be better because they allow most of the weights to achieve their "global" minima?
  
  **Tags:** object-detection, object-recognition, stochastic-gradient-descent, adam, adadelta
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-03-02. Score: 1
  
  I have experimented with this to a small degree and have not noticed that much of an impact.
  
  To date, Adam appears to give the best results on a variety of image data sets. I have found that "adjusting" the learning rate during training is an effective means of improving model performance and has more impact than the selection of the optimizer.
  
  Keras has two callbacks that are useful for this purpose. Documentation is at <https://keras.io/callbacks/>. The `ModelCheckpoint` callback enables you to save the full model or just the model weights based on monitoring a metric. Typically, you monitor validation loss and set the parameter `save_best_only=True` to save the results for the lowest validation loss. The other useful callback is `ReduceLROnPlateau`, which allows you to adjust the learning rate based on monitoring a metric. Again, the metric usually monitored is the validation loss. If the loss fails to reduce after a user-set number of epochs (parameter patience), the learning rate will be adjusted by a user-set factor (parameter factor). You can think of the training process as traveling down a valley. As you near the bottom of the valley, it becomes more and more narrow. If your learning rate does not adjust to the "narrowness" there is no way you will get to the bottom of the valley.
  
  You can also write a custom callback to adjust the learning rate. I have done this and created one which first adjusts the learning rate based on monitoring the training loss until the training accuracy reaches 95%. Then it switches to adjust the learning rate based on monitoring the validation loss. It saves the model weights for the lowest validation loss and loads the model with these weights to make predictions. I have found this approach leads to faster training and higher accuracy.
  
  The fact is you can't tell if your model has converged on a global minimum or a local minimum. This is evidenced by the fact that, unless you take special efforts to inhibit randomization, you can get different results each time you run your model. The loss can be envisioned as a surface in $N$ space, where $N$ is the number of trainable parameters. Lord knows what that surface is like and where your initial parameter weights put you on that surface, plus how other random processes cause you to traverse that surface.
  
  As an example, I ran a model at least 20 times and got resultant losses that were very close to each there. Then I ran it again and got far better results for exactly the same data.
  
  > **Comments:**
  > **username_2**: Interesting bio and project goal, but consider potential for abuse - deliberately diagnosing or stigmatizing appearance. An arguably fair medical asset, but unsure a public utility.  
  
  ---
  
  ### Answer by username_3 on 2020-02-28. Score: 0
  
  > Does the choice of optimizer really matter in terms of training the model?
  
  Yes.
  
  > Would the Adam optimizer be better than the Adadelta optimizer?
  
  Yes. (But sometimes, adadelta gives better result. Depends upon the dataset and fine-tune mechanism)
  
  > Would they all basically be the same?
  
  No. Here is the [explanation](https://towardsdatascience.com/adam-latest-trends-in-deep-learning-optimization-6be9a291375c)
  
  > Would some optimizers be better because they allow most of the weights to achieve their "global" minima?
  
  It's not possible to check whether the model achieved global minima or not practically.
  
  We can evaluate the model by either over-fitting or under-fitting using the training and validation set, and the generalization of the model with the test set.
  
  
  ---
  
  