Answers.Body: <p>I'm going to take the fitness landscape to be the <a href="https://en.wikipedia.org/wiki/Graph_of_a_function#Definition" rel="nofollow noreferrer">graph</a> of the loss function, <span class="math-container">$\mathcal{G} = \{\left(\theta, L(\theta)\right) : \theta \in \mathbb{R}^n\}$</span>, where <span class="math-container">$\theta$</span> parameterises the network (i.e. it is the weights and biases) and <span class="math-container">$L$</span> is a given loss function; in other words, the surface you would get by plotting the loss function against its parameters.</p>
  <p>We always assume the loss function is differentiable in order to do backpropagation, which means at the very least the loss function is smooth enough to be continuous, but in principle it may not be infinitely differentiable<sup>1</sup>.</p>
  <p>You talk about using gradient descent to find the global minimiser. In general this is not possible: many functions have local minimisers which are not global minimisers. For an example, you could plot <span class="math-container">$y = x^2 \sin(1/x^2)$</span>: of course the situation is similar, if harder to visualise, in higher dimensions. A certain class of functions known as <strong>convex functions</strong> satisfy the property that any local minimiser is a global minimiser. Unfortunately, the loss function of a neural network is rarely convex.</p>
  <p>For some interesting pictures, see <a href="https://papers.nips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf" rel="nofollow noreferrer"><em>Visualizing the Loss Landscape of Neural Nets</em></a> by Li et al.</p>
  <hr />
  <p><sup>1</sup> For a more detailed discussion on continuity and differentiability, any good text on mathematical analysis will do, for example Rudin's <em>Principles of Mathematical Analysis</em>. In general, any function <span class="math-container">$f$</span> that is differentiable on some interval is also continuous, but it need not be twice differentiable, i.e. <span class="math-container">$f''$</span> need not exist.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:19:55.183Z
Answers.Text: This answer seems reasonable to me, i.e. you need differentiability to apply back-propagation, but note that not all neural networks are trained with back-prop (although this may be an assumption, when you use the term "deep learning"). But why do you say that $G$ is a graph? Maybe you mean a space (https://en.wikipedia.org/wiki/Space_(mathematics))? In any case, it seems that you're using the term "graph" not as defined in graph theory.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T18:15:28.770Z
Answers.Text: @nbro Thanks for the question, I mean graph as in the literal sense of plotting a graph of a function (you can see an example of this usage [here](https://en.wikipedia.org/wiki/Graph_of_a_function#Definition), for example). My choice of notation has clearly added to the confusion as it is common to denote a graph (in the network sense) by $G$, but I don't mean to say $G$ is a network. Hopefully the latest revision clarifies this.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T08:52:12.993Z
Answers.LastActivityDate: 2021-04-09T18:14:29.867Z
Answers.LastEditDate: 2021-04-09T18:14:29.867Z
Answers.UserDisplayName: htl
Answers.Body: <h3>Main answer</h3>
  <p>To answer your question as directly as possible: No, deep learning does not make that &quot;assumption&quot;.</p>
  <p>But you're close. Just swap the word &quot;assumption&quot; with &quot;imposition&quot;.</p>
  <p>Deep learning <strong>sets things up</strong> such that the landscape is (mostly) smooth and always continuous*, and therefore it is possible to do some sort of optimization via gradient descent.</p>
  <p>* quick footnotes on that bit:</p>
  <ul>
  <li><a href="https://math.stackexchange.com/questions/472148/smooth-functions-or-continuous#:%7E:text=Smooth%20implies%20continuous%2C%20but%20not,continuous%20everywhere%2C%20yet%20nowhere%20differentiable.&amp;text=A%20smooth%20function%20is%20a,desired%20order%20over%20some%20domain.">Smoothness is a stronger condition than continuity</a>, that's why I mention them both.</li>
  <li>My statement is not authoritative, so take it with a grain of salt, especially the &quot;always&quot; bit. Maybe someone will debunk this in the comments.</li>
  <li>The reason that I say &quot;(mostly) smooth&quot; is because I can think of a counter example to smoothness, which is the ReLU activation function. ReLU is still continuous though.</li>
  </ul>
  <h3>Further elaboration</h3>
  <p>In deep learning we have linear layers which we know are differentiable. We also have non-linear activations, and a loss function which for the intents of this discussion can be bundled with non-linear activations. If you look at papers which focus specifically on crafting new types of non-linear activations and loss functions you will usually find a discussion section that goes something like &quot;and we designed it this way such that it's differentiable. Here's how you differentiate it. Here are the properties of the derivative&quot;. For instance, just check out this <a href="https://arxiv.org/pdf/1511.07289.pdf" rel="nofollow noreferrer">paper on ELU, a refinement on ReLU</a>.</p>
  <p>We don't need to &quot;assume&quot; anything really, as we are the ones who designate the building blocks of the deep learning network. And the building blocks are not all that complicated in themselves, so we can <strong>know</strong> that they are differentiable (or piecewise differentiable like ReLU). And for rigor, I should also remind you that <a href="https://math.stackexchange.com/questions/142084/prove-that-the-composition-of-differentiable-functions-is-differentiable">the composition of multiple differentiable functions is also differentiable</a>.</p>
  <p>So hopefully that helps you see what I mean when I say deep learning architects &quot;impose&quot; differentiability, rather than &quot;assume&quot; it. After all, we are the architects!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T09:19:08.203Z
Answers.LastActivityDate: 2021-04-09T09:44:14.480Z
Answers.LastEditDate: 2021-04-09T09:44:14.480Z
Answers.UserDisplayName: Alexander Soare
Answers.Body: <p><strong>Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one?</strong></p>
  <p>One can interpret this question from a formal-mathematical standpoint and from a more &quot;intuitively-practical&quot; standpoint.</p>
  <p>From the formal point of view, <a href="https://en.wikipedia.org/wiki/Smoothness" rel="nofollow noreferrer">smoothness</a> is the requirement that the function is continuous with continuous first derivatives. And this assumption is quite often not true in lots of applications - mostly because of the widespread use of ReLU activation function - <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Potential_problems" rel="nofollow noreferrer">it is not differentiable at zero</a>.</p>
  <p>From the practical point of view, though, by &quot;smoothness&quot; we mean that the function's &quot;landscape&quot; does not have a lot of sharp jumps and edges like that:</p>
  <p><a href="https://i.sstatic.net/E4i3F.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/E4i3F.png" alt="enter image description here" /></a></p>
  <p>Practically, there's not much difference between having a discontinuous derivative and having derivatives making very sharp jumps.</p>
  <p>And again, the answer is no - the loss function landscape is extremely spiky with lots of sharp edges - the picture above is an example of an actual loss function landscape.</p>
  <p><strong>But... why the gradient descent works then?</strong></p>
  <p>As far as I know, this is a subject of an ongoing discussion in the community. There are different takes and some conflicting viewpoints that are still subject of a debate.</p>
  <p>My opinion is that, fundamentally, the idea that we need it to converge to the global optimum is a flawed one. Neural networks was shown to have enough capacity to <a href="https://arxiv.org/pdf/1611.03530.pdf" rel="nofollow noreferrer">completely remember the training dataset</a>. A neural network, that completely remembered the training data. has reached the global optimization minimum (given only the training data). We are not interested in such overtrained models - we want models that generalize well.</p>
  <p>As far as I know, there is no conclusive results on which properties of the minimum are linked to ability to generalize. People argued that these should be the <a href="https://arxiv.org/pdf/1609.04836.pdf" rel="nofollow noreferrer">&quot;flat&quot;</a> minima, but then it <a href="https://arxiv.org/pdf/1703.04933.pdf" rel="nofollow noreferrer">was refuted</a>. After that a <a href="https://arxiv.org/pdf/1803.05407.pdf" rel="nofollow noreferrer">&quot;wide optimium&quot;</a> term was introduced and gave rise of an interesting technique of Stochastic Weight Averaging.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T15:41:45.793Z
Answers.Text: Why would we be looking for the "widest" minimum? Moreover, although I think what you mean, you should explain what you mean by "widest minimum" (maybe a more common term is "flat minimum"). I really don't understand why we would not be looking for the **global** minimum. The global minimum/maximum is the optimal solution. Not sure what you're talking about here and where your conclusions come from.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T16:53:05.707Z
Answers.Text: @nbro Would you agree that a neural network, that completely remembered the training data, has reached the global optimization minimum?
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:01:20.800Z
Answers.Text: Mathematics is not about agreeing on something or not. It's about making assumptions or starting from premises and follow the rules to reach some conclusions. In any case, that would be the global optimum but for the training dataset. In reality, as you know, we're not trying to overfit the training data but to find a function that generalizes well to unseen data too. The thing is: I don't know where your conclusion that we're trying to find the "widest minimum" comes from and why would that mean "better generalization", which is what you seem to mean/imply now.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:03:47.930Z
Answers.Text: To be honest, the question is not even clear. The author is talking about the "fitness landscape", but they didn't even clarify what that would mean in the context of deep learning. They are probably talking about "loss landscape", but it's also not clear. I'm not an expert on the topic, but I don't think that your conclusion (last part) is well motivated. The first paragraphs seem reasonable to me, though, although, again, I am not a mathematician and not an expert on this specific topic of "loss landscapes", whatever the definition is.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:09:11.517Z
Answers.Text: @nbro I thought that my three last sentences unambiguously stated that I'm talking about generalization... If you have a suggestion on how to rephrase that - I'd happily do that. I didn't want to go into formal details of it because it was my slightly opinionated aside on the subject. Also, I do agree that the question is not well-phrased, but demanding mathematically precise formulation from beginners is not very productive, wouldn't you say?
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:13:53.817Z
Answers.Text: In your last paragraph, you indeed say "we want generalization". That's correct. But where does the "widest minimum" come from and what does it even mean? It's not clear. That's what I'm questioning.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:14:22.137Z
Answers.Text: Furthermore, I think that giving an answer with many mathematical details to such a poorly formulated question may not be easy to follow for the beginner, but it definitely would clarify certain doubts, if you understand the math. What I think, generally, is that "no information is better than misleading information", so opinions are typically not very useful and this site is not about opinions, but more about facts. If the question is "Do you think that X is Y?", then, ok, you could say your opinion, but these questions are not suitable for our site.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:23:50.497Z
Answers.Text: @nbro I just used "wide minimum" as an intuitive analog of "minimum that generalizes well". I can remove "wide" if you insist. But I don't feel like using "flat" in that context, because there's some research that shown that minimum curvature is not necessarily linked to its generalization performance (as well lts "globalness").
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T17:25:07.213Z
Answers.Text: I think you should just define what you mean by that. What kind of minimum are we exactly looking for and, more importantly, why do you say that we're not looking for the global minimum and clarify that you mean the global minimum given only the training data.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T18:00:37.263Z
Answers.Text: @nbro Done. Actually I did a refresher on literature and "wide" is a pretty widely (heh) used term in publications on the subject...
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T18:36:36.907Z
Answers.Text: @Kostya, if the loss function is not smooth, how can we compute derivatives?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T18:38:14.270Z
Answers.Text: @Kostya, yes RELU is not differentiable at $x=0$, but we assume the derivate is zero at $x=0$.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T19:04:46.937Z
Answers.Text: @Kostya, I don't understand how you create link between gradient descent on the function that is not smooth and finding the local minima or better minima(may be your saying it as wider minima). Since we are standing on a function that has many local minima it is natural to stuck in a local minima using gradient descent. I don't see how non smooth fuction and getting local minima linked together.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T09:55:30.653Z
Answers.LastActivityDate: 2021-04-09T17:58:14.040Z
Answers.LastEditDate: 2021-04-09T17:58:14.040Z
Answers.UserDisplayName: Kostya
Body: <p>I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental movements or iterations across the landscape.</p>
  <p>My question is: <strong>Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one? If so, is it a valid assumption?</strong></p>
  <p>Most of the graphical representations I have seen of gradient descent show a smooth landscape.</p>
  <p>This <a href="https://en.m.wikipedia.org/wiki/Fitness_landscape" rel="nofollow noreferrer">Wikipedia page</a> describes the fitness landscape.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T07:54:37.310Z
Comments.Text: What is the fitness landscape? Is it the surface where gradient descent occurs? To compute the derivatives we need a smooth manifold. So, I think the landscape is smooth.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T07:57:29.953Z
Comments.Text: I am not sure about what fitness landscape mean but if it is the landscape where gradient descent occur. Then I think smooth means there is not discontinuty or sudden jump in the surface, so we can find out the derivatives.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T09:42:38.557Z
Comments.Text: i dont believe it's smooth landscape theoretically, since gradient descent in neural net does partial derivation thus other weights may get to wrong values when a certain weight is optimised; this results oscillation sometimes
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T12:57:43.183Z
Comments.Text: You say "I've come across the concept of fitness landscape before". Can you please provide the link to the research papers or books that use this term "fitness landscape". I think this term occurs more in the context of evolutionary algorithms and not deep learning.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:08:03.190Z
Comments.Text: @nbro: You are right. It does occur mainly in the evolutionary context. I just thought the concept might still be applicable.
Comments.UserDisplayName: Joebevo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:09:38.633Z
Comments.Text: Please, provide the link to the papers where you found this term. People are giving answers but maybe based on wrong assumptions, and that makes those answers basically useless, because you didn't provide enough context.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:16:04.920Z
Comments.Text: https://en.m.wikipedia.org/wiki/Fitness_landscape
Comments.UserDisplayName: Joebevo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:16:30.090Z
Comments.Text: Just edit your post to include the relevant links and context, and maybe explain that you though that "fitness landscape" would also be applicable in the context of DL.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T13:16:31.183Z
Comments.Text: The Wikipedia page gives a good overview
Comments.UserDisplayName: Joebevo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T15:39:07.043Z
Comments.Text: So, what do you mean by "fitness landscape" in the context of deep learning? Maybe it would be better to change "fitness landscape" to a more appropriate term. Maybe "loss landscape"?
Comments.UserDisplayName: nbro
Tags: <gradient-descent><fitness-functions>
Title: Does gradient descent in deep learning assume a smooth fitness landscape?
UserDisplayName: Joebevo
ThreadText: # Does gradient descent in deep learning assume a smooth fitness landscape?
  
  (Asked by: username_0 on 2021-04-09)
  
  I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental movements or iterations across the landscape.
  
  My question is: **Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one? If so, is it a valid assumption?**
  
  Most of the graphical representations I have seen of gradient descent show a smooth landscape.
  
  This [Wikipedia page](https://en.m.wikipedia.org/wiki/Fitness_landscape) describes the fitness landscape.
  
  **Tags:** gradient-descent, fitness-functions
  > **Comments:**
  > **username_1**: What is the fitness landscape? Is it the surface where gradient descent occurs? To compute the derivatives we need a smooth manifold. So, I think the landscape is smooth.  
  > **username_1**: I am not sure about what fitness landscape mean but if it is the landscape where gradient descent occur. Then I think smooth means there is not discontinuty or sudden jump in the surface, so we can find out the derivatives.  
  > **username_2**: i dont believe it's smooth landscape theoretically, since gradient descent in neural net does partial derivation thus other weights may get to wrong values when a certain weight is optimised; this results oscillation sometimes  
  > **username_3**: You say "I've come across the concept of fitness landscape before". Can you please provide the link to the research papers or books that use this term "fitness landscape". I think this term occurs more in the context of evolutionary algorithms and not deep learning.  
  > **username_0**: @nbro: You are right. It does occur mainly in the evolutionary context. I just thought the concept might still be applicable.  
  > **username_3**: Please, provide the link to the papers where you found this term. People are giving answers but maybe based on wrong assumptions, and that makes those answers basically useless, because you didn't provide enough context.  
  > **username_0**: https://en.m.wikipedia.org/wiki/Fitness_landscape  
  > **username_3**: Just edit your post to include the relevant links and context, and maybe explain that you though that "fitness landscape" would also be applicable in the context of DL.  
  > **username_0**: The Wikipedia page gives a good overview  
  > **username_3**: So, what do you mean by "fitness landscape" in the context of deep learning? Maybe it would be better to change "fitness landscape" to a more appropriate term. Maybe "loss landscape"?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2021-04-09. Score: 1
  
  I'm going to take the fitness landscape to be the [graph](https://en.wikipedia.org/wiki/Graph_of_a_function#Definition) of the loss function, $\mathcal{G} = \{\left(\theta, L(\theta)\right) : \theta \in \mathbb{R}^n\}$, where $\theta$ parameterises the network (i.e. it is the weights and biases) and $L$ is a given loss function; in other words, the surface you would get by plotting the loss function against its parameters.
  
  We always assume the loss function is differentiable in order to do backpropagation, which means at the very least the loss function is smooth enough to be continuous, but in principle it may not be infinitely differentiable1.
  
  You talk about using gradient descent to find the global minimiser. In general this is not possible: many functions have local minimisers which are not global minimisers. For an example, you could plot $y = x^2 \sin(1/x^2)$: of course the situation is similar, if harder to visualise, in higher dimensions. A certain class of functions known as **convex functions** satisfy the property that any local minimiser is a global minimiser. Unfortunately, the loss function of a neural network is rarely convex.
  
  For some interesting pictures, see [*Visualizing the Loss Landscape of Neural Nets*](https://papers.nips.cc/paper/2018/file/a41b3bb3e6b050b6c9067c67f663b915-Paper.pdf) by Li et al.
  
  ---
  
  1 For a more detailed discussion on continuity and differentiability, any good text on mathematical analysis will do, for example Rudin's *Principles of Mathematical Analysis*. In general, any function $f$ that is differentiable on some interval is also continuous, but it need not be twice differentiable, i.e. $f''$ need not exist.
  
  > **Comments:**
  > **username_3**: This answer seems reasonable to me, i.e. you need differentiability to apply back-propagation, but note that not all neural networks are trained with back-prop (although this may be an assumption, when you use the term "deep learning"). But why do you say that $G$ is a graph? Maybe you mean a space (https://en.wikipedia.org/wiki/Space_(mathematics))? In any case, it seems that you're using the term "graph" not as defined in graph theory.  
  > **username_4**: @nbro Thanks for the question, I mean graph as in the literal sense of plotting a graph of a function (you can see an example of this usage [here](https://en.wikipedia.org/wiki/Graph_of_a_function#Definition), for example). My choice of notation has clearly added to the confusion as it is common to denote a graph (in the network sense) by $G$, but I don't mean to say $G$ is a network. Hopefully the latest revision clarifies this.  
  
  ---
  
  ### Answer by username_5 on 2021-04-09. Score: 1
  
  ### Main answer
  
  To answer your question as directly as possible: No, deep learning does not make that "assumption".
  
  But you're close. Just swap the word "assumption" with "imposition".
  
  Deep learning **sets things up** such that the landscape is (mostly) smooth and always continuous\*, and therefore it is possible to do some sort of optimization via gradient descent.
  
  \* quick footnotes on that bit:
  
  * [Smoothness is a stronger condition than continuity](https://math.stackexchange.com/questions/472148/smooth-functions-or-continuous#:%7E:text=Smooth%20implies%20continuous%2C%20but%20not,continuous%20everywhere%2C%20yet%20nowhere%20differentiable.&text=A%20smooth%20function%20is%20a,desired%20order%20over%20some%20domain.), that's why I mention them both.
  * My statement is not authoritative, so take it with a grain of salt, especially the "always" bit. Maybe someone will debunk this in the comments.
  * The reason that I say "(mostly) smooth" is because I can think of a counter example to smoothness, which is the ReLU activation function. ReLU is still continuous though.
  
  ### Further elaboration
  
  In deep learning we have linear layers which we know are differentiable. We also have non-linear activations, and a loss function which for the intents of this discussion can be bundled with non-linear activations. If you look at papers which focus specifically on crafting new types of non-linear activations and loss functions you will usually find a discussion section that goes something like "and we designed it this way such that it's differentiable. Here's how you differentiate it. Here are the properties of the derivative". For instance, just check out this [paper on ELU, a refinement on ReLU](https://arxiv.org/pdf/1511.07289.pdf).
  
  We don't need to "assume" anything really, as we are the ones who designate the building blocks of the deep learning network. And the building blocks are not all that complicated in themselves, so we can **know** that they are differentiable (or piecewise differentiable like ReLU). And for rigor, I should also remind you that [the composition of multiple differentiable functions is also differentiable](https://math.stackexchange.com/questions/142084/prove-that-the-composition-of-differentiable-functions-is-differentiable).
  
  So hopefully that helps you see what I mean when I say deep learning architects "impose" differentiability, rather than "assume" it. After all, we are the architects!
  
  
  ---
  
  ### Answer by username_6 on 2021-04-09. Score: -2
  
  **Does deep learning assume that the fitness landscape on which the gradient descent occurs is a smooth one?**
  
  One can interpret this question from a formal-mathematical standpoint and from a more "intuitively-practical" standpoint.
  
  From the formal point of view, [smoothness](https://en.wikipedia.org/wiki/Smoothness) is the requirement that the function is continuous with continuous first derivatives. And this assumption is quite often not true in lots of applications - mostly because of the widespread use of ReLU activation function - [it is not differentiable at zero](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Potential_problems).
  
  From the practical point of view, though, by "smoothness" we mean that the function's "landscape" does not have a lot of sharp jumps and edges like that:
  
  [![enter image description here](https://i.sstatic.net/E4i3F.png)](https://i.sstatic.net/E4i3F.png)
  
  Practically, there's not much difference between having a discontinuous derivative and having derivatives making very sharp jumps.
  
  And again, the answer is no - the loss function landscape is extremely spiky with lots of sharp edges - the picture above is an example of an actual loss function landscape.
  
  **But... why the gradient descent works then?**
  
  As far as I know, this is a subject of an ongoing discussion in the community. There are different takes and some conflicting viewpoints that are still subject of a debate.
  
  My opinion is that, fundamentally, the idea that we need it to converge to the global optimum is a flawed one. Neural networks was shown to have enough capacity to [completely remember the training dataset](https://arxiv.org/pdf/1611.03530.pdf). A neural network, that completely remembered the training data. has reached the global optimization minimum (given only the training data). We are not interested in such overtrained models - we want models that generalize well.
  
  As far as I know, there is no conclusive results on which properties of the minimum are linked to ability to generalize. People argued that these should be the ["flat"](https://arxiv.org/pdf/1609.04836.pdf) minima, but then it [was refuted](https://arxiv.org/pdf/1703.04933.pdf). After that a ["wide optimium"](https://arxiv.org/pdf/1803.05407.pdf) term was introduced and gave rise of an interesting technique of Stochastic Weight Averaging.
  
  > **Comments:**
  > **username_3**: Why would we be looking for the "widest" minimum? Moreover, although I think what you mean, you should explain what you mean by "widest minimum" (maybe a more common term is "flat minimum"). I really don't understand why we would not be looking for the **global** minimum. The global minimum/maximum is the optimal solution. Not sure what you're talking about here and where your conclusions come from.  
  > **username_6**: @nbro Would you agree that a neural network, that completely remembered the training data, has reached the global optimization minimum?  
  > **username_3**: Mathematics is not about agreeing on something or not. It's about making assumptions or starting from premises and follow the rules to reach some conclusions. In any case, that would be the global optimum but for the training dataset. In reality, as you know, we're not trying to overfit the training data but to find a function that generalizes well to unseen data too. The thing is: I don't know where your conclusion that we're trying to find the "widest minimum" comes from and why would that mean "better generalization", which is what you seem to mean/imply now.  
  > **username_3**: To be honest, the question is not even clear. The author is talking about the "fitness landscape", but they didn't even clarify what that would mean in the context of deep learning. They are probably talking about "loss landscape", but it's also not clear. I'm not an expert on the topic, but I don't think that your conclusion (last part) is well motivated. The first paragraphs seem reasonable to me, though, although, again, I am not a mathematician and not an expert on this specific topic of "loss landscapes", whatever the definition is.  
  > **username_6**: @nbro I thought that my three last sentences unambiguously stated that I'm talking about generalization... If you have a suggestion on how to rephrase that - I'd happily do that. I didn't want to go into formal details of it because it was my slightly opinionated aside on the subject. Also, I do agree that the question is not well-phrased, but demanding mathematically precise formulation from beginners is not very productive, wouldn't you say?  
  > **username_3**: In your last paragraph, you indeed say "we want generalization". That's correct. But where does the "widest minimum" come from and what does it even mean? It's not clear. That's what I'm questioning.  
  > **username_3**: Furthermore, I think that giving an answer with many mathematical details to such a poorly formulated question may not be easy to follow for the beginner, but it definitely would clarify certain doubts, if you understand the math. What I think, generally, is that "no information is better than misleading information", so opinions are typically not very useful and this site is not about opinions, but more about facts. If the question is "Do you think that X is Y?", then, ok, you could say your opinion, but these questions are not suitable for our site.  
  > **username_6**: @nbro I just used "wide minimum" as an intuitive analog of "minimum that generalizes well". I can remove "wide" if you insist. But I don't feel like using "flat" in that context, because there's some research that shown that minimum curvature is not necessarily linked to its generalization performance (as well lts "globalness").  
  > **username_3**: I think you should just define what you mean by that. What kind of minimum are we exactly looking for and, more importantly, why do you say that we're not looking for the global minimum and clarify that you mean the global minimum given only the training data.  
  > **username_6**: @nbro Done. Actually I did a refresher on literature and "wide" is a pretty widely (heh) used term in publications on the subject...  
  > **username_1**: @Kostya, if the loss function is not smooth, how can we compute derivatives?  
  > **username_1**: @Kostya, yes RELU is not differentiable at $x=0$, but we assume the derivate is zero at $x=0$.  
  > **username_1**: @Kostya, I don't understand how you create link between gradient descent on the function that is not smooth and finding the local minima or better minima(may be your saying it as wider minima). Since we are standing on a function that has many local minima it is natural to stuck in a local minima using gradient descent. I don't see how non smooth fuction and getting local minima linked together.  
  
  ---
  
  