Answers.Body: <p>It's worth noting that the Fourier series analogy was used in early explorations of universal approximation theorems <a href="https://ieeexplore.ieee.org/document/23903" rel="nofollow noreferrer">https://ieeexplore.ieee.org/document/23903</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T18:36:25.403Z
Answers.LastActivityDate: 2023-03-23T18:36:25.403Z
Answers.UserDisplayName: Luke
Answers.Body: <blockquote>
  <p>People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data.</p>
  </blockquote>
  <p>There is an opinion that this is completely missing the point. It doesn't really matter how well you can approximate your data - a lookup table that completely remembers your data can perform with 100% accuracy. What matters is the generalization capabilities of your functional family: how well does it interpolates and extrapolates beyond its training observations.</p>
  <p>Speaking about the Fourier transform - I think one can even say that the <a href="https://en.wikipedia.org/wiki/Gibbs_phenomenon" rel="nofollow noreferrer">Gibbs phenomenon</a> can be considered as very early example of people noticing such &quot;bad generalization&quot; behavior.</p>
  <p>Why deep neural networks are so (comparatively) good at generalization? There's still a discussion about it.  There's <a href="https://en.wikipedia.org/wiki/Manifold_hypothesis" rel="nofollow noreferrer">Manifold hypothesis</a>, there's <a href="https://arxiv.org/abs/2012.00152" rel="nofollow noreferrer">kernel machines</a> interpretation, there are studies of <a href="https://www.sciencedirect.com/science/article/pii/S0370157321001290" rel="nofollow noreferrer">training landscapes</a> of deep neural nets. There are even takes that employ <a href="https://arxiv.org/abs/2106.10165" rel="nofollow noreferrer">criticality analysis</a> from physics. To me it looks like all these approaches are parts of a bigger picture that we are yet to see.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T13:12:57.597Z
Answers.LastActivityDate: 2023-03-23T13:12:57.597Z
Answers.UserDisplayName: Kostya
Body: <p>People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable to me, because something like Fourier series are also able to approximate almost any function between compact domains of Euclidean spaces.</p>
  <p>So my question is, what makes neural networks different from something like Fourier analysis where we can approximate any sufficiently nice function we like as well?</p>
  <p>Am I not understanding the universal approximation theorem, or are there justifications for the power of neural networks that go deeper than talk about approximation?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-24T03:06:39.380Z
Comments.Text: [Related post on Cross Validated (Statistics Stack)](https://stats.stackexchange.com/a/41290/247274)
Comments.UserDisplayName: Dave
Tags: <neural-networks><deep-learning><universal-approximation-theorems>
Title: What makes the approximation capabilities of neural networks different than something like, say, Fourier series?
UserDisplayName: MaximusIdeal
ThreadText: # What makes the approximation capabilities of neural networks different than something like, say, Fourier series?
  
  (Asked by: username_0 on 2023-03-23)
  
  People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data. However, this seems unremarkable to me, because something like Fourier series are also able to approximate almost any function between compact domains of Euclidean spaces.
  
  So my question is, what makes neural networks different from something like Fourier analysis where we can approximate any sufficiently nice function we like as well?
  
  Am I not understanding the universal approximation theorem, or are there justifications for the power of neural networks that go deeper than talk about approximation?
  
  **Tags:** neural-networks, deep-learning, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: [Related post on Cross Validated (Statistics Stack)](https://stats.stackexchange.com/a/41290/247274)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-03-23. Score: 4 | Accepted answer
  
  > People often cite the universal approximation theorem as a reason for why neutral networks are so effective at capturing patterns or features of various training data.
  
  There is an opinion that this is completely missing the point. It doesn't really matter how well you can approximate your data - a lookup table that completely remembers your data can perform with 100% accuracy. What matters is the generalization capabilities of your functional family: how well does it interpolates and extrapolates beyond its training observations.
  
  Speaking about the Fourier transform - I think one can even say that the [Gibbs phenomenon](https://en.wikipedia.org/wiki/Gibbs_phenomenon) can be considered as very early example of people noticing such "bad generalization" behavior.
  
  Why deep neural networks are so (comparatively) good at generalization? There's still a discussion about it. There's [Manifold hypothesis](https://en.wikipedia.org/wiki/Manifold_hypothesis), there's [kernel machines](https://arxiv.org/abs/2012.00152) interpretation, there are studies of [training landscapes](https://www.sciencedirect.com/science/article/pii/S0370157321001290) of deep neural nets. There are even takes that employ [criticality analysis](https://arxiv.org/abs/2106.10165) from physics. To me it looks like all these approaches are parts of a bigger picture that we are yet to see.
  
  
  ---
  
  ### Answer by username_2 on 2023-03-23. Score: 3
  
  It's worth noting that the Fourier series analogy was used in early explorations of universal approximation theorems <https://ieeexplore.ieee.org/document/23903>.
  
  
  ---
  
  