Answers.Body: <p>Just sharing my thoughts on this: </p>
  
  <ul>
  <li><p>A local minimum in the loss function is just a synthetic measure expressing how well the model is performing on the target dataset (I'm assuming it's the test set)</p></li>
  <li><p>Letting aside the absolute values, you could also consider the steepness of that region as follows: </p>
  
  <ul>
  <li><p>the first LM is at the bottom of steep region which means the related parametrization is quite <em>unstable</em> as if you change it slightly the performance drop quickly (loss increases quickly) so you could interpret this 
  as a parametrization with not great generalization capability (it is not robust against small changes): always remember that what you are actually looking for is model able to generalize well when <em>in production</em> (i.e. after you have deployed it so when training, validation, test is finished) and test set is typically a small proxy for the data observed in production (but it really depends on the final application itself)</p></li>
  <li><p>the second LM is related to a much less steep region: even if you change the parametrization slightly you do not observe a big performance change which could be interpreted as a more <em>stable</em> parametrization hence with better possibility to generalize on unseen data </p></li>
  </ul></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T21:39:58.930Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/90495/discussion-on-answer-by-nicola-bernini-which-local-minima-to-choose-according-to).
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T07:36:35.337Z
Answers.LastActivityDate: 2019-03-02T09:06:44.530Z
Answers.LastEditDate: 2019-03-02T09:06:44.530Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>The following plot shows error function output based on system weights.
  Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.</p>
  
  <p>Considering the amount of convex in the local minima, is there any way to opt between these two local minima?</p>
  
  <p><a href="https://i.sstatic.net/7rMRC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7rMRC.png" alt="enter image description here"></a></p>
  
Tags: <neural-networks><backpropagation><optimization><gradient-descent><objective-functions>
Title: Which local minima to choose according to the shape of the error surface?
UserDisplayName: hmojtaba
ThreadText: # Which local minima to choose according to the shape of the error surface?
  
  (Asked by: username_0 on 2019-03-01)
  
  The following plot shows error function output based on system weights.
  Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.
  
  Considering the amount of convex in the local minima, is there any way to opt between these two local minima?
  
  [![enter image description here](https://i.sstatic.net/7rMRC.png)](https://i.sstatic.net/7rMRC.png)
  
  **Tags:** neural-networks, backpropagation, optimization, gradient-descent, objective-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-02. Score: 1 | Accepted answer
  
  Just sharing my thoughts on this:
  
  * A local minimum in the loss function is just a synthetic measure expressing how well the model is performing on the target dataset (I'm assuming it's the test set)
  * Letting aside the absolute values, you could also consider the steepness of that region as follows:
  
    + the first LM is at the bottom of steep region which means the related parametrization is quite *unstable* as if you change it slightly the performance drop quickly (loss increases quickly) so you could interpret this
      as a parametrization with not great generalization capability (it is not robust against small changes): always remember that what you are actually looking for is model able to generalize well when *in production* (i.e. after you have deployed it so when training, validation, test is finished) and test set is typically a small proxy for the data observed in production (but it really depends on the final application itself)
    + the second LM is related to a much less steep region: even if you change the parametrization slightly you do not observe a big performance change which could be interpreted as a more *stable* parametrization hence with better possibility to generalize on unseen data
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/90495/discussion-on-answer-by-username_1-username_1-which-local-minima-to-choose-according-to).  
  
  ---
  
  