Answers.Body: <p>Your interpretation of the graph is correct, in this simple example weight taken from the saddle point will perform better than the samples taken at the local minima. Said tat, this 1D example is a bit of an outlier compared to what happen in more realistic scenarios.</p>
  <p>If we keep in mind the <a href="https://en.wikipedia.org/wiki/Second_partial_derivative_test" rel="nofollow noreferrer">second partial derivative test</a>, used to check if a point of a function is a maximum, minimum or saddle point, then it's easy to see that especially in higher dimensions the probability of a point being a saddle one are higher than being a maximum or a minimum one.</p>
  <p>When analyzing a point we first compute the determinant of the Hessian matrix of our loss function (I'm using the 2 variables formulation for convenience):</p>
  <p><span class="math-container">$H(x, y)=f_{xx}(x, y)f_{yy}(x, y) - (f_{xy}(x,y))^{2}$</span></p>
  <p>if <span class="math-container">$H(x,y)&lt;0$</span> then the point is a saddle point. Otherwise if <span class="math-container">$H(x,y)&gt;0$</span> it could be a maximum or minimum. Notice that <span class="math-container">$f_{xx}(x, y)$</span> tells us the direction of concavity in the x direction, whereas <span class="math-container">$f_{yy}(x, y)$</span> tells us the direction of the concavity in the y direction. This makes sense, if the concavities points on different directions for different dimensions we can't have a maximum or a minimum (see picture below, positive direction for y axis, negative for x axis), and indeed the resulting score is negative, indicating a saddle point.</p>
  <p>The take home point is that a local minima compared to a saddle point respects the property of sharing the same negative direction across all dimensions of the function. This implies that, as the number of dimensions increases (and with real models we can easily start from thousands of them) the probability of a point being a saddle point increases as well, while the probability of being a local or global minima decreases.</p>
  <p>So in a more realistic case, just for a matter of probability, a solution associated with a random local minima has lot of chances to be better than a solution  associated to a random saddle point.</p>
   <img src="https://i.sstatic.net/UJIRj.png" width="200" height="300"> 
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-07T11:17:55.900Z
Answers.LastActivityDate: 2022-02-07T11:17:55.900Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p><strong>Can Saddle Points Provide &quot;Better Solutions&quot; to Machine Learning Models Compared to Local Minimums?</strong></p>
  <p>The solution to a Machine Learning model (i.e. the final model parameters) are selected by trying to optimize the Loss Function associated with that Machine Learning model. The &quot;best&quot; solution (i.e. &quot;best&quot; choice of model parameters) are those associated with the &quot;global minimum&quot; of this Loss Function (i.e. the smallest value of the Loss Function) - thus, &quot;relatively better&quot; solutions can be considered as solutions that are located closer to the &quot;Global Minimum&quot;. Optimization Algorithms (e.g. Gradient Descent) try to search for the &quot;Global Minimum&quot; of the Loss Function by repeatedly searching in the direction of the derivatives corresponding to this Loss Function.</p>
  <p>However, there are different obstacles than can occur during this search process. For instance:</p>
  <ul>
  <li>The Optimization Algorithm can get stuck in a &quot;Local Minimum&quot;</li>
  <li>The Optimization Algorithm can get stuck in a &quot;Saddle Point&quot;</li>
  </ul>
  <p>I have heard &quot;Saddle Points&quot; as being considered &quot;worse&quot; than &quot;Local Minimums&quot; - this is because &quot;Saddle Points&quot; aren't actually a minimum of any sort, whereas &quot;Local Minimums&quot; are at least minimums at the local level. <strong>Thus, this would imply that model parameters chosen from a &quot;Saddle Point&quot; should be worse than model parameters chosen from a &quot;Local Minimum&quot;.</strong> To further illustrate my question, I drew the following graph of a hypothetical Loss Function for some Machine Learning model:</p>
  <p><a href="https://i.sstatic.net/1JjcQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1JjcQ.png" alt="enter image description here" /></a></p>
  <p>In the above picture, we can see that Loss Function has a smaller loss at the &quot;Saddle Point&quot; compared to the loss at the &quot;Local Minimum&quot;. Thus, in this case - (Assuming that we could not reach &quot;P3&quot;) if we had to choose a selection of model parameters from &quot;P2&quot; (&quot;Saddle Point&quot;) and &quot;P1&quot; (&quot;Local Minimum&quot;) - it would clearly make more sense to pick model parameters from &quot;P2&quot;.</p>
  <p><strong>My Question:</strong> In general, do we know if solutions corresponding to &quot;Local Minimums&quot; points on a Loss Function are considered to be &quot;better&quot; corresponding to &quot;Saddle Points&quot; (e.g. perhaps solutions from &quot;Local Minimums&quot; might be more &quot;stable&quot;)? Or is this claim incorrect, and <strong>solutions corresponding to regions of the Loss Function with lower Loss values are generally &quot;better&quot; - regardless of whether they come from a &quot;Saddle Point&quot; or a &quot;Local Minimum&quot;?</strong></p>
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T09:32:05.867Z
Comments.Text: Rather than writing "Comparing Solutions from Saddle Points vs. Local Minimums", put your **specific** question in the title. Note "Comparing Solutions from Saddle Points vs. Local Minimums" is not a question and it's not specific.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-15T22:16:57.473Z
Comments.Text: Ok, can you do it then? See https://ai.meta.stackexchange.com/q/1654/2444 in case this request does not make much sense to you.
Comments.UserDisplayName: nbro
Tags: <neural-networks>
Title: Comparing Solutions from Saddle Points vs. Local Minimums
UserDisplayName: stats_noob
ThreadText: # Comparing Solutions from Saddle Points vs. Local Minimums
  
  (Asked by: username_0 on 2022-02-07)
  
  **Can Saddle Points Provide "Better Solutions" to Machine Learning Models Compared to Local Minimums?**
  
  The solution to a Machine Learning model (i.e. the final model parameters) are selected by trying to optimize the Loss Function associated with that Machine Learning model. The "best" solution (i.e. "best" choice of model parameters) are those associated with the "global minimum" of this Loss Function (i.e. the smallest value of the Loss Function) - thus, "relatively better" solutions can be considered as solutions that are located closer to the "Global Minimum". Optimization Algorithms (e.g. Gradient Descent) try to search for the "Global Minimum" of the Loss Function by repeatedly searching in the direction of the derivatives corresponding to this Loss Function.
  
  However, there are different obstacles than can occur during this search process. For instance:
  
  * The Optimization Algorithm can get stuck in a "Local Minimum"
  * The Optimization Algorithm can get stuck in a "Saddle Point"
  
  I have heard "Saddle Points" as being considered "worse" than "Local Minimums" - this is because "Saddle Points" aren't actually a minimum of any sort, whereas "Local Minimums" are at least minimums at the local level. **Thus, this would imply that model parameters chosen from a "Saddle Point" should be worse than model parameters chosen from a "Local Minimum".** To further illustrate my question, I drew the following graph of a hypothetical Loss Function for some Machine Learning model:
  
  [![enter image description here](https://i.sstatic.net/1JjcQ.png)](https://i.sstatic.net/1JjcQ.png)
  
  In the above picture, we can see that Loss Function has a smaller loss at the "Saddle Point" compared to the loss at the "Local Minimum". Thus, in this case - (Assuming that we could not reach "P3") if we had to choose a selection of model parameters from "P2" ("Saddle Point") and "P1" ("Local Minimum") - it would clearly make more sense to pick model parameters from "P2".
  
  **My Question:** In general, do we know if solutions corresponding to "Local Minimums" points on a Loss Function are considered to be "better" corresponding to "Saddle Points" (e.g. perhaps solutions from "Local Minimums" might be more "stable")? Or is this claim incorrect, and **solutions corresponding to regions of the Loss Function with lower Loss values are generally "better" - regardless of whether they come from a "Saddle Point" or a "Local Minimum"?**
  
  Thanks!
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: Rather than writing "Comparing Solutions from Saddle Points vs. Local Minimums", put your **specific** question in the title. Note "Comparing Solutions from Saddle Points vs. Local Minimums" is not a question and it's not specific.  
  > **username_1**: Ok, can you do it then? See https://ai.meta.stackexchange.com/q/1654/2444 in case this request does not make much sense to you.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-07. Score: 1
  
  Your interpretation of the graph is correct, in this simple example weight taken from the saddle point will perform better than the samples taken at the local minima. Said tat, this 1D example is a bit of an outlier compared to what happen in more realistic scenarios.
  
  If we keep in mind the [second partial derivative test](https://en.wikipedia.org/wiki/Second_partial_derivative_test), used to check if a point of a function is a maximum, minimum or saddle point, then it's easy to see that especially in higher dimensions the probability of a point being a saddle one are higher than being a maximum or a minimum one.
  
  When analyzing a point we first compute the determinant of the Hessian matrix of our loss function (I'm using the 2 variables formulation for convenience):
  
  $H(x, y)=f\_{xx}(x, y)f\_{yy}(x, y) - (f\_{xy}(x,y))^{2}$
  
  if $H(x,y)<0$ then the point is a saddle point. Otherwise if $H(x,y)>0$ it could be a maximum or minimum. Notice that $f\_{xx}(x, y)$ tells us the direction of concavity in the x direction, whereas $f\_{yy}(x, y)$ tells us the direction of the concavity in the y direction. This makes sense, if the concavities points on different directions for different dimensions we can't have a maximum or a minimum (see picture below, positive direction for y axis, negative for x axis), and indeed the resulting score is negative, indicating a saddle point.
  
  The take home point is that a local minima compared to a saddle point respects the property of sharing the same negative direction across all dimensions of the function. This implies that, as the number of dimensions increases (and with real models we can easily start from thousands of them) the probability of a point being a saddle point increases as well, while the probability of being a local or global minima decreases.
  
  So in a more realistic case, just for a matter of probability, a solution associated with a random local minima has lot of chances to be better than a solution associated to a random saddle point.
  
  ![](https://i.sstatic.net/UJIRj.png)
  
  
  ---
  
  