Answers.Body: <p>As mentioned in the abstract of on of these papers, bad local minima is a suboptimal local minimum which means a local minimum that is near to a global minimum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-16T07:46:43.317Z
Answers.LastActivityDate: 2019-01-16T07:46:43.317Z
Answers.UserDisplayName: OmG
Answers.Body: <p>The adjective <em>bad</em> isn't mathematically descriptive. A better term is sub-optimal, which implies the state of learning might appear optimal based on current information but the optimal solution from among all possibilities is not yet located.</p>
  
  <p>Consider a graph representing a loss function, one of the names to measure disparity between the current learning state and the optimal. Some papers use the term <em>error</em>. In all learning and adaptive control cases, it is the quantification of disparity between current and optimal states. This is a 3D plot, so it only visualizes the loss as a function of two real number parameters. There can be thousands, but the two will suffice to visualize the meaning of local versus global minima. Some may recall this phenomenon from pre-calculus or analytic geometry.</p>
  
  <p>If pure gradient descent is used with an initial value to the far left, the local minimum in the loss surface would be located first. Climbing the slope to test the loss value at the global minimum further to the right would not generally occur. The gradient will in most cases cause the next iteration in learning algorithms to travel down hill, thus the term <em>gradient descent</em>.</p>
  
  <p>This is a simple case beyond just the visualization of two parameters, since there can be thousands of local minima in the loss surface.</p>
  
  <p><a href="https://i.sstatic.net/2tS9r.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2tS9r.png" alt="Error Surface Showing How Global Optimum Can be Missed"></a></p>
  
  <p>There are many strategic approaches to improve the speed and reliability in the search for the global minimum loss. These are just a few.</p>
  
  <ul>
  <li>Descent using gradient, or more exactly, the Jacobian</li>
  <li>Possibly further terms of the multidimensional Taylor series, the next in line being related to curvature, the Hessian</li>
  <li>Injection of noise, which is at an intuitive (and somewhat oversimplified) level is like shaking the graph so a ball representing the current learning state might bounce over a ridge or peak and land (by essentially trial and error) the global minimum &mdash; Simulated annealing is a materials science analogy and involves simulating the injection of Brownian (thermal) motion</li>
  <li>Searches from more than one starting position</li>
  <li>Parallelism, with or without an intelligent control strategy, to try multiple initial learning states and hyper-parameter settings</li>
  <li>Models of the surface based on past learning or theoretically known principles so that the global minimum can be projected as trials are used to tune the parameters of the model</li>
  </ul>
  
  <p>Interestingly, the only way to prove that the optimal state is found is to try every possibility by checking each one, which is not nearly feasible in most cases, or by relying on a model from which the global optimal may be determined. Most theoretical frameworks target a particular accuracy, reliability, speed, and minimal input information quantity as part of an AI project, so that no such exhaustive search or model perfection is required.</p>
  
  <p>In practical terms, for example, an automated vehicle control system is adequate when the unit testing, alpha functional testing, and eventually beta functional testing all indicate a lower incidence of both injury and loss of property than when humans drive. It is a statistical quality assurance, as in the case of most service and manufacturing businesses.</p>
  
  <p>The graph above was developed for another answer, which has additional information for those interested.</p>
  
  <ul>
  <li><a href="https://ai.stackexchange.com/questions/7328/if-digital-values-are-mere-estimates-why-not-return-to-analog-for-ai/7982#7982">If digital values are mere estimates, why not return to analog for AI?</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-20T14:49:32.590Z
Answers.LastActivityDate: 2019-01-20T14:49:32.590Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>What is "bad local minima"? </p>
  
  <p>The following papers all mention this expression.</p>
  
  <ol>
  <li><a href="https://arxiv.org/abs/1901.03909" rel="nofollow noreferrer">Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit</a></li>
  <li><a href="https://arxiv.org/abs/1901.00279" rel="nofollow noreferrer">limination of All Bad Local Minima in Deep Learning</a></li>
  <li><a href="https://arxiv.org/abs/1805.08671" rel="nofollow noreferrer">Adding One Neuron Can Eliminate All Bad Local Minima</a></li>
  </ol>
  
Tags: <machine-learning><deep-learning><terminology><papers><calculus>
Title: What is a bad local minimum in machine learning?
UserDisplayName: Umang Gupta
ThreadText: # What is a bad local minimum in machine learning?
  
  (Asked by: username_0 on 2019-01-16)
  
  What is "bad local minima"?
  
  The following papers all mention this expression.
  
  1. [Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit](https://arxiv.org/abs/1901.03909)
  2. [limination of All Bad Local Minima in Deep Learning](https://arxiv.org/abs/1901.00279)
  3. [Adding One Neuron Can Eliminate All Bad Local Minima](https://arxiv.org/abs/1805.08671)
  
  **Tags:** machine-learning, deep-learning, terminology, papers, calculus
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-01-20. Score: 2 | Accepted answer
  
  The adjective *bad* isn't mathematically descriptive. A better term is sub-optimal, which implies the state of learning might appear optimal based on current information but the optimal solution from among all possibilities is not yet located.
  
  Consider a graph representing a loss function, one of the names to measure disparity between the current learning state and the optimal. Some papers use the term *error*. In all learning and adaptive control cases, it is the quantification of disparity between current and optimal states. This is a 3D plot, so it only visualizes the loss as a function of two real number parameters. There can be thousands, but the two will suffice to visualize the meaning of local versus global minima. Some may recall this phenomenon from pre-calculus or analytic geometry.
  
  If pure gradient descent is used with an initial value to the far left, the local minimum in the loss surface would be located first. Climbing the slope to test the loss value at the global minimum further to the right would not generally occur. The gradient will in most cases cause the next iteration in learning algorithms to travel down hill, thus the term *gradient descent*.
  
  This is a simple case beyond just the visualization of two parameters, since there can be thousands of local minima in the loss surface.
  
  [![Error Surface Showing How Global Optimum Can be Missed](https://i.sstatic.net/2tS9r.png)](https://i.sstatic.net/2tS9r.png)
  
  There are many strategic approaches to improve the speed and reliability in the search for the global minimum loss. These are just a few.
  
  * Descent using gradient, or more exactly, the Jacobian
  * Possibly further terms of the multidimensional Taylor series, the next in line being related to curvature, the Hessian
  * Injection of noise, which is at an intuitive (and somewhat oversimplified) level is like shaking the graph so a ball representing the current learning state might bounce over a ridge or peak and land (by essentially trial and error) the global minimum — Simulated annealing is a materials science analogy and involves simulating the injection of Brownian (thermal) motion
  * Searches from more than one starting position
  * Parallelism, with or without an intelligent control strategy, to try multiple initial learning states and hyper-parameter settings
  * Models of the surface based on past learning or theoretically known principles so that the global minimum can be projected as trials are used to tune the parameters of the model
  
  Interestingly, the only way to prove that the optimal state is found is to try every possibility by checking each one, which is not nearly feasible in most cases, or by relying on a model from which the global optimal may be determined. Most theoretical frameworks target a particular accuracy, reliability, speed, and minimal input information quantity as part of an AI project, so that no such exhaustive search or model perfection is required.
  
  In practical terms, for example, an automated vehicle control system is adequate when the unit testing, alpha functional testing, and eventually beta functional testing all indicate a lower incidence of both injury and loss of property than when humans drive. It is a statistical quality assurance, as in the case of most service and manufacturing businesses.
  
  The graph above was developed for another answer, which has additional information for those interested.
  
  * [If digital values are mere estimates, why not return to analog for AI?](https://ai.stackexchange.com/questions/7328/if-digital-values-are-mere-estimates-why-not-return-to-analog-for-ai/7982#7982)
  
  
  ---
  
  ### Answer by username_1 on 2019-01-16. Score: 0
  
  As mentioned in the abstract of on of these papers, bad local minima is a suboptimal local minimum which means a local minimum that is near to a global minimum.
  
  
  ---
  
  