Answers.Body: <blockquote>
  <p>Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?</p>
  </blockquote>
  <p>Yes, with some minor caveats.</p>
  <p>All the points on the flat region are equivalent (and in your example, are all valid global minimum points). Gradients outside of the region will point correctly away from that region and gradient descent steps will therefore move parameters towards it.</p>
  <p>Provided the step size multiplied by the gradient near the flat region is not too large, then a step taken near it will end up with parameters inside the region. After that, then any further gradients will be zero, so it is not possible to use <em>basic</em> gradient steps to escape it.</p>
  <p>In the case of a global minimum, that's fine, you don't care <em>which</em> point in the global minimum you have converged to (otherwise your function to optimise would be different).</p>
  <p>In the case of local minima or saddle points, you might care to use optimisation methods that can escape flat areas. Minibatch or stochastic gradient descent can do this because gradient measurements are noisy, whilst <a href="https://distill.pub/2017/momentum/" rel="nofollow noreferrer">momentum algorithms</a> can continue making steps when the immediate gradient signal is zero.</p>
  <p>The example function you used is not something you would expect to come across when optimising a machine learning algorithm, although some loss functions do have components that have similar behaviour. For example, <a href="https://en.wikipedia.org/wiki/Triplet_loss" rel="nofollow noreferrer">triplet loss</a> uses a <span class="math-container">$\text{max}(d_1 - d_2 + \alpha, 0)$</span> where <span class="math-container">$d_1$</span> and <span class="math-container">$d_2$</span> are distances between an anchor image and desired class versus different class respectively, and <span class="math-container">$\alpha$</span> is a margin or minimum desirable distance between classes. The details of this are not important unless you want to create a face recogniser or similar - the important detail for your question is that <span class="math-container">$\text{max}(x, 0)$</span> is really used in ML as a loss function, and may have a similar shape to your example function. Once used in aggregate with many data examples though, and with regularisation, the shape would not be so simple, and proabably would not have any reachable flat minima regions like this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-22T19:02:10.337Z
Answers.LastActivityDate: 2021-08-23T23:13:50.707Z
Answers.LastEditDate: 2021-08-23T23:13:50.707Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am studying a chapter named <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> of a deep learning book. Afaik, it does not deal with flat regions with desired points.</p>
  <p>For example, let us consider a function whose local/global minimum or maximum values lies on flat regions. You can take <a href="https://www.desmos.com/calculator/5vbpayhztu" rel="nofollow noreferrer">this</a> graph (just) for example.</p>
  <p><a href="https://i.sstatic.net/M620x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/M620x.png" alt="Example graph with flat minimum" /></a></p>
  <p>Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?</p>
  
Tags: <optimization><gradient-descent>
Title: Do gradient-based algorithms deal with the flat regions with desired points?
UserDisplayName: hanugm
ThreadText: # Do gradient-based algorithms deal with the flat regions with desired points?
  
  (Asked by: username_0 on 2021-08-21)
  
  I am studying a chapter named [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) of a deep learning book. Afaik, it does not deal with flat regions with desired points.
  
  For example, let us consider a function whose local/global minimum or maximum values lies on flat regions. You can take [this](https://www.desmos.com/calculator/5vbpayhztu) graph (just) for example.
  
  [![Example graph with flat minimum](https://i.sstatic.net/M620x.png)](https://i.sstatic.net/M620x.png)
  
  Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?
  
  **Tags:** optimization, gradient-descent
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-22. Score: 1
  
  > Can gradient-based algorithms work on those curves with their local/global minima, or do maxima lie on flat regions?
  
  Yes, with some minor caveats.
  
  All the points on the flat region are equivalent (and in your example, are all valid global minimum points). Gradients outside of the region will point correctly away from that region and gradient descent steps will therefore move parameters towards it.
  
  Provided the step size multiplied by the gradient near the flat region is not too large, then a step taken near it will end up with parameters inside the region. After that, then any further gradients will be zero, so it is not possible to use *basic* gradient steps to escape it.
  
  In the case of a global minimum, that's fine, you don't care *which* point in the global minimum you have converged to (otherwise your function to optimise would be different).
  
  In the case of local minima or saddle points, you might care to use optimisation methods that can escape flat areas. Minibatch or stochastic gradient descent can do this because gradient measurements are noisy, whilst [momentum algorithms](https://distill.pub/2017/momentum/) can continue making steps when the immediate gradient signal is zero.
  
  The example function you used is not something you would expect to come across when optimising a machine learning algorithm, although some loss functions do have components that have similar behaviour. For example, [triplet loss](https://en.wikipedia.org/wiki/Triplet_loss) uses a $\text{max}(d\_1 - d\_2 + \alpha, 0)$ where $d\_1$ and $d\_2$ are distances between an anchor image and desired class versus different class respectively, and $\alpha$ is a margin or minimum desirable distance between classes. The details of this are not important unless you want to create a face recogniser or similar - the important detail for your question is that $\text{max}(x, 0)$ is really used in ML as a loss function, and may have a similar shape to your example function. Once used in aggregate with many data examples though, and with regularisation, the shape would not be so simple, and proabably would not have any reachable flat minima regions like this.
  
  
  ---
  
  