Answers.Body: <p>It's important to note that in a pragmatic instance of ML on a "real dataset", you likely wouldn't have a "strict" saddle point with precisely zero gradient. Your error surface won't be "smooth", so even though you would have something that resembles a saddle point, what you would obtain would in fact be a small region with near-zero gradient.</p>
  
  <p>So let's say you are in a region with near-zero gradient. Assuming that the gradient is this area is normalized at 0 with small Gaussian distributed noise (thus gradient =  small Gaussian noise). You can then see that the algorithm can't quite escape the region (or at least, will spend a lot of time here) since 1. Gaussian random walks will more-or-less stay in place (unless for a long time) 2. small gradients means there is no obvious direction to leave the region.</p>
  
  <p>In any case, SGD more or less solves this issue, and its usage is standard practice for reasons beyond this problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-04T17:42:51.353Z
Answers.LastActivityDate: 2020-03-04T17:42:51.353Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>I was reading a blog post that talked about the problem of the saddle point in training. </p>
  
  <blockquote>
    <p>In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this? </p>
    
    <p>Wouldn’t it continue down in the y direction and hence escape the saddle point?</p>
  </blockquote>
  
  <p><a href="https://i.sstatic.net/NyXTy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NyXTy.png" alt="Saddle point"></a></p>
  
  <p>Link to post:
  <a href="https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/" rel="nofollow noreferrer">https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/</a></p>
  
  <p>Please go to <em>Challenges with Gradient Descent #2: Saddle Points</em>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T13:48:19.253Z
Comments.Text: IIRC, the main issue would be that in high dimensions there would be no "obvious" direction for the gradient descent to move towards, as you would tend to believe in a 3d graph like above. (some analogies fail in high dimensions) im not super certain about this answer (though i think its right), so i'll leave it as a comment instead
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T14:23:41.180Z
Comments.Text: Thank you for the reply. If there is no obvious direction then is it to say the saddle points flat area is relatively large in all directions? For a smaller flat area, if the y direction is the local minima in this saddle point then it would make sense to me for it to oscillate in that direction as it continues to overshoot the minima. It never considers the x direction as the gradients are too small. However, if the y direction is the local maxima here, then it should not be stuck in the y direction but x?
Comments.UserDisplayName: Jack-P
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T17:18:20.877Z
Comments.Text: yeah tbh i was also wondering about this question when i was first introduced to the problem. i think the point being is that in a pragmatic instance, your error surface won't be strictly-by-definition a saddle point with 0 gradient, but will be a small region with near-0 gradient. because there is small-noise in the gradient, the algo will wander around but cannot escape since there is no clear/strong tangental direction to escape the region with low gradient. actually i think this is the answer, so i'll write it properly
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-04T17:26:48.100Z
Comments.Text: fun fact: a friend told me once that in pragmatic instances (datasets of the "real world"), error surfaces might have an "egg carton shape" ("might", because what even does a high-dim egg carton look like?). we say this because we observe that (obviously: using the same ML architecture and hyper-parameters), we tend to get more or less the same error rates despite learning vastly different weights. the implication is that the error surface has many local minimas but they are more-or-less of "the same depth"
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T17:32:13.287Z
Comments.Text: I agree with k c . From my experience in running the same model  many times the local minimum attained seem to result in about the same values for training accuracy with slightly more variance on validation accuracy. The saddle point issue is harder to get a handle on. As noted in other comments while mathematically a saddle point has a 0 value for the derivative in a real world model it might get very small but will not be zero. I would expect that the result would be oscillations in the flat area about the saddle point. Reducing the learning rate might help to converge to a smaller area
Comments.UserDisplayName: Gerry P
Tags: <optimization><gradient-descent>
Title: Oscillating around the saddle point in gradient descent?
UserDisplayName: Jack-P
ThreadText: # Oscillating around the saddle point in gradient descent?
  
  (Asked by: username_0 on 2020-03-04)
  
  I was reading a blog post that talked about the problem of the saddle point in training.
  
  > In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to y at the saddle point, gradient descent will oscillate to and from the y direction. This gives an illusion of converging to a minima. Why is this?
  >
  > Wouldn’t it continue down in the y direction and hence escape the saddle point?
  
  [![Saddle point](https://i.sstatic.net/NyXTy.png)](https://i.sstatic.net/NyXTy.png)
  
  Link to post:
  <https://blog.paperspace.com/intro-to-optimization-in-deep-learning-gradient-descent/>
  
  Please go to *Challenges with Gradient Descent #2: Saddle Points*.
  
  **Tags:** optimization, gradient-descent
  
  > **Comments:**
  > **username_1**: IIRC, the main issue would be that in high dimensions there would be no "obvious" direction for the gradient descent to move towards, as you would tend to believe in a 3d graph like above. (some analogies fail in high dimensions) im not super certain about this answer (though i think its right), so i'll leave it as a comment instead  
  > **username_0**: Thank you for the reply. If there is no obvious direction then is it to say the saddle points flat area is relatively large in all directions? For a smaller flat area, if the y direction is the local minima in this saddle point then it would make sense to me for it to oscillate in that direction as it continues to overshoot the minima. It never considers the x direction as the gradients are too small. However, if the y direction is the local maxima here, then it should not be stuck in the y direction but x?  
  > **username_1**: yeah tbh i was also wondering about this question when i was first introduced to the problem. i think the point being is that in a pragmatic instance, your error surface won't be strictly-by-definition a saddle point with 0 gradient, but will be a small region with near-0 gradient. because there is small-noise in the gradient, the algo will wander around but cannot escape since there is no clear/strong tangental direction to escape the region with low gradient. actually i think this is the answer, so i'll write it properly  
  > **username_1**: fun fact: a friend told me once that in pragmatic instances (datasets of the "real world"), error surfaces might have an "egg carton shape" ("might", because what even does a high-dim egg carton look like?). we say this because we observe that (obviously: using the same ML architecture and hyper-parameters), we tend to get more or less the same error rates despite learning vastly different weights. the implication is that the error surface has many local minimas but they are more-or-less of "the same depth"  
  > **username_2**: I agree with k c . From my experience in running the same model  many times the local minimum attained seem to result in about the same values for training accuracy with slightly more variance on validation accuracy. The saddle point issue is harder to get a handle on. As noted in other comments while mathematically a saddle point has a 0 value for the derivative in a real world model it might get very small but will not be zero. I would expect that the result would be oscillations in the flat area about the saddle point. Reducing the learning rate might help to converge to a smaller area  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-04. Score: 0 | Accepted answer
  
  It's important to note that in a pragmatic instance of ML on a "real dataset", you likely wouldn't have a "strict" saddle point with precisely zero gradient. Your error surface won't be "smooth", so even though you would have something that resembles a saddle point, what you would obtain would in fact be a small region with near-zero gradient.
  
  So let's say you are in a region with near-zero gradient. Assuming that the gradient is this area is normalized at 0 with small Gaussian distributed noise (thus gradient = small Gaussian noise). You can then see that the algorithm can't quite escape the region (or at least, will spend a lot of time here) since 1. Gaussian random walks will more-or-less stay in place (unless for a long time) 2. small gradients means there is no obvious direction to leave the region.
  
  In any case, SGD more or less solves this issue, and its usage is standard practice for reasons beyond this problem.
  
  
  ---
  
  