Answers.Body: <p>I don't think that the concept of <a href="https://en.wikipedia.org/wiki/Generalization_error" rel="nofollow noreferrer">generalization</a> is (directly) related to the "shape" of the function close to the point where it attains a minimum.</p>
  
  <p>The concept of generalisation refers to when a trained model is able "<a href="https://stats.stackexchange.com/q/385312/82135">perform</a> well" on unseen data (that is, data not seen during the training phase). If a trained model does not generalise well, then it might have <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">"overfitted" or "underfitted"</a>. </p>
  
  <p>Overfitting means that the model performs well on the training data, but not on other data. In other words, in the case of overfitting, the model learns the structure of the training data, but not of the whole population (of interest) or it learns the "noise" (that is, the data which is not part of the population of interest) present in the training data. This also implies that the training data is not a "good" sample of the population, that is, it doesn't "summarise" well all characteristics of the population. In practice, this is often the case (for large populations).</p>
  
  <p>Underfitting occurs when the model is not even able to learn enough about the training data during the training phase. For example, underfitting occurs when you try to train a linear model but the data does not have a linear relationship.</p>
  
  <p>So, being able to generalise (or not) depends on the model (including the number of parameters), but also on the data you are given.</p>
  
  <blockquote>
    <p>Does the right one generalize better compared to the left one?</p>
  </blockquote>
  
  <p>The shape of the function will only be useful during the training phase. More specifically, in this case, you might reach one of the two minima faster than the other (depending also on the optimisation algorithms, model, etc., that you use). </p>
  
  <p>I would like to note that, in machine learning, we often minimise a function of functions (which in mathematics is called a "<a href="https://en.wikipedia.org/wiki/Functional_(mathematics)" rel="nofollow noreferrer">functional</a>"). Why is that? For "simplicity", consider a simple neural network (NN) model (e.g. a multi-layer perceptron). We usually train such NN using gradient descent (or one of its variants) by minimising a function (e.g. the mean squared error). Essentially, when we train such a NN, we want to find the function (which is represented by the parameters of the NN) that minimises e.g. the MSE. In this case, the mean squared error (MSE) is the functional we attempt to minimise. Note that the MSE is a function of the parameters of the current NN and that a NN is a model that represents a function.</p>
  
  <p>Let's go back to your question. If, during the training phase, you get one minimum rather than the other, you will get different NNs (say <span class="math-container">$A$</span> and <span class="math-container">$B$</span>). Note, again, that the point where our MSE function attains one its minima, in this context of training a NN (and often, in general, in machine learning) is a function (and not just a scalar). I know that, at the beginning (and if you're not a math guy), it might be difficult to think of functions as points (actually, more precisely, we should call them "vectors") that minimise other functions, but this concept exists and actually underlies a great amount of machine learning techniques (e.g. training NNs using back-propagation by minimising a cost function).</p>
  
  <p><em>So, why do I think that we can't say much about the generalisation ability of NNs <span class="math-container">$A$</span> and <span class="math-container">$B$</span>?</em></p>
  
  <p>Suppose that we have access, after the training phase, to both NNs <span class="math-container">$A$</span> and <span class="math-container">$B$</span> (that is, the ones that correspond to the "points" where your function attains the two (visible) minima. <span class="math-container">$A$</span> and <span class="math-container">$B$</span> will perform equally well on the training data (or even validation data). We might then conclude that both generalise in the same way. But this might be a wrong conclusion because both the training and the validation data (as I mentioned above) might not be good samples of the population. So, we can't really say which NN (or "minima"), <span class="math-container">$A$</span> or <span class="math-container">$B$</span>, generalises better. They might generalise more or less in the same way (according to their performance e.g. on the <a href="https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets" rel="nofollow noreferrer">validation dataset</a>) or not (because the validation dataset is not a good sample of the population).</p>
  
  <p>To conclude, the concept of generalisation is a little bit more complex than function minimisation, because it is also related to data. In machine learning, we often minimise a <strong>functional</strong> (a function whose parameters are functions). If your functional attains the same minimum at two different points (actually, functions), during training, one might be faster to reach than the other. However, if you have two NNs that represent these two minima, we can't say much about their generalisation ability. Essentially, we can just hope that the validation dataset is a good sample of the population.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T23:43:02.427Z
Answers.Text: thanks for the detailed answer, I will try to understand your answer and come back here soon.
Answers.UserDisplayName: hmojtaba
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T09:08:19.423Z
Answers.Text: the details provided in this answer convinced me that there could be more intuitions and science related to my question that I have to consider. Although it does not satisfy me.
Answers.UserDisplayName: hmojtaba
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T23:28:35.510Z
Answers.LastActivityDate: 2019-03-02T10:37:56.403Z
Answers.LastEditDate: 2019-03-02T10:37:56.403Z
Answers.UserDisplayName: nbro
Answers.Body: <p>In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: </p>
  
  <ul>
  <li><p>what the author of the question provides us with is the Loss Function Shape so I'll try to use the full information here to compare the 2 minima </p></li>
  <li><p>looking at the LF steepness we observe the Left LM is in a steeper region than the Right LM: how can this be interpreted? </p></li>
  <li><p>I'd interpret the LF Steepness as a Measure of Parametrization Stability: in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one </p></li>
  <li><p>when the NN is running "in production" typically the parametrization is fixed (in most typical application, weights are not changed out of the training phase) so you should not be concerned about parametrization stability, 
  however I'm one of those believing the <a href="http://www.bioinf.jku.at/publications/older/3304.pdf" rel="nofollow noreferrer">Flat Minima - Schmidhuber 1997</a> idea that Local Minima Flatness is connected to generalization property </p></li>
  </ul>
  
  <p>However it is important to observe this is still a very open and interesting question as in <a href="https://arxiv.org/abs/1703.04933" rel="nofollow noreferrer">Sharp Minima Can Generalize For Deep Nets</a> Dinh 2017 et al. demonstrated 
  it is not just about flatness, because reparametrization of the NN model, though preserving minima locations, change its shape so basically sharp minima might be transformed into flat ones without affecting network performance </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T08:48:10.463Z
Answers.Text: Even though this "_in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one_" was already the assumption of the OP, I appreciate more this answer because you cite some research work.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T09:09:13.057Z
Answers.Text: Overviewing the reference kindly provided by @NicolaBernini, I am actually happy that, as a beginner in AI, this question jumped in my head. Thanks for the answer.
Answers.UserDisplayName: hmojtaba
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T08:25:17.603Z
Answers.LastActivityDate: 2019-03-02T08:25:17.603Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>The following plot shows error function output based on system weights.
  Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.</p>
  
  <p>Does the right one generalize better compared to the left one? </p>
  
  <p>My assumption is that for the right minimum if the weights change, the overall error increases less compared to the left minimum point. Would this somehow mean the system does better generalization if the right one is chosen as the optimum minimum?</p>
  
  <p><a href="https://i.sstatic.net/pDEFF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pDEFF.png" alt="enter image description here"></a></p>
  
Tags: <neural-networks><backpropagation><optimization><gradient-descent><objective-functions>
Title: Could error surface shape be useful to detect which local minima is better for generalization?
UserDisplayName: hmojtaba
ThreadText: # Could error surface shape be useful to detect which local minima is better for generalization?
  
  (Asked by: username_0 on 2019-03-01)
  
  The following plot shows error function output based on system weights.
  Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.
  
  Does the right one generalize better compared to the left one?
  
  My assumption is that for the right minimum if the weights change, the overall error increases less compared to the left minimum point. Would this somehow mean the system does better generalization if the right one is chosen as the optimum minimum?
  
  [![enter image description here](https://i.sstatic.net/pDEFF.png)](https://i.sstatic.net/pDEFF.png)
  
  **Tags:** neural-networks, backpropagation, optimization, gradient-descent, objective-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-03-02. Score: 4 | Accepted answer
  
  In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations:
  
  * what the author of the question provides us with is the Loss Function Shape so I'll try to use the full information here to compare the 2 minima
  * looking at the LF steepness we observe the Left LM is in a steeper region than the Right LM: how can this be interpreted?
  * I'd interpret the LF Steepness as a Measure of Parametrization Stability: in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one
  * when the NN is running "in production" typically the parametrization is fixed (in most typical application, weights are not changed out of the training phase) so you should not be concerned about parametrization stability,
    however I'm one of those believing the [Flat Minima - Schmidhuber 1997](http://www.bioinf.jku.at/publications/older/3304.pdf) idea that Local Minima Flatness is connected to generalization property
  
  However it is important to observe this is still a very open and interesting question as in [Sharp Minima Can Generalize For Deep Nets](https://arxiv.org/abs/1703.04933) Dinh 2017 et al. demonstrated
  it is not just about flatness, because reparametrization of the NN model, though preserving minima locations, change its shape so basically sharp minima might be transformed into flat ones without affecting network performance
  
  > **Comments:**
  > **username_1**: Even though this "_in fact perturbating the Parametrization slightly has a bigger effect on Left LM observed performance rather than the Right LM one_" was already the assumption of the OP, I appreciate more this answer because you cite some research work.  
  > **username_0**: Overviewing the reference kindly provided by @NicolaBernini, I am actually happy that, as a beginner in AI, this question jumped in my head. Thanks for the answer.  
  
  ---
  
  ### Answer by username_1 on 2019-03-01. Score: 2
  
  I don't think that the concept of [generalization](https://en.wikipedia.org/wiki/Generalization_error) is (directly) related to the "shape" of the function close to the point where it attains a minimum.
  
  The concept of generalisation refers to when a trained model is able "[perform](https://stats.stackexchange.com/q/385312/82135) well" on unseen data (that is, data not seen during the training phase). If a trained model does not generalise well, then it might have ["overfitted" or "underfitted"](https://en.wikipedia.org/wiki/Overfitting).
  
  Overfitting means that the model performs well on the training data, but not on other data. In other words, in the case of overfitting, the model learns the structure of the training data, but not of the whole population (of interest) or it learns the "noise" (that is, the data which is not part of the population of interest) present in the training data. This also implies that the training data is not a "good" sample of the population, that is, it doesn't "summarise" well all characteristics of the population. In practice, this is often the case (for large populations).
  
  Underfitting occurs when the model is not even able to learn enough about the training data during the training phase. For example, underfitting occurs when you try to train a linear model but the data does not have a linear relationship.
  
  So, being able to generalise (or not) depends on the model (including the number of parameters), but also on the data you are given.
  
  > Does the right one generalize better compared to the left one?
  
  The shape of the function will only be useful during the training phase. More specifically, in this case, you might reach one of the two minima faster than the other (depending also on the optimisation algorithms, model, etc., that you use).
  
  I would like to note that, in machine learning, we often minimise a function of functions (which in mathematics is called a "[functional](https://en.wikipedia.org/wiki/Functional_(mathematics))"). Why is that? For "simplicity", consider a simple neural network (NN) model (e.g. a multi-layer perceptron). We usually train such NN using gradient descent (or one of its variants) by minimising a function (e.g. the mean squared error). Essentially, when we train such a NN, we want to find the function (which is represented by the parameters of the NN) that minimises e.g. the MSE. In this case, the mean squared error (MSE) is the functional we attempt to minimise. Note that the MSE is a function of the parameters of the current NN and that a NN is a model that represents a function.
  
  Let's go back to your question. If, during the training phase, you get one minimum rather than the other, you will get different NNs (say $A$ and $B$). Note, again, that the point where our MSE function attains one its minima, in this context of training a NN (and often, in general, in machine learning) is a function (and not just a scalar). I know that, at the beginning (and if you're not a math guy), it might be difficult to think of functions as points (actually, more precisely, we should call them "vectors") that minimise other functions, but this concept exists and actually underlies a great amount of machine learning techniques (e.g. training NNs using back-propagation by minimising a cost function).
  
  *So, why do I think that we can't say much about the generalisation ability of NNs $A$ and $B$?*
  
  Suppose that we have access, after the training phase, to both NNs $A$ and $B$ (that is, the ones that correspond to the "points" where your function attains the two (visible) minima. $A$ and $B$ will perform equally well on the training data (or even validation data). We might then conclude that both generalise in the same way. But this might be a wrong conclusion because both the training and the validation data (as I mentioned above) might not be good samples of the population. So, we can't really say which NN (or "minima"), $A$ or $B$, generalises better. They might generalise more or less in the same way (according to their performance e.g. on the [validation dataset](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets)) or not (because the validation dataset is not a good sample of the population).
  
  To conclude, the concept of generalisation is a little bit more complex than function minimisation, because it is also related to data. In machine learning, we often minimise a **functional** (a function whose parameters are functions). If your functional attains the same minimum at two different points (actually, functions), during training, one might be faster to reach than the other. However, if you have two NNs that represent these two minima, we can't say much about their generalisation ability. Essentially, we can just hope that the validation dataset is a good sample of the population.
  
  > **Comments:**
  > **username_0**: thanks for the detailed answer, I will try to understand your answer and come back here soon.  
  > **username_0**: the details provided in this answer convinced me that there could be more intuitions and science related to my question that I have to consider. Although it does not satisfy me.  
  
  ---
  
  