Answers.Body: <p>There is stuff like the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="noreferrer">Universal Approximation Theorem</a>. </p>
  
  <p>There are also <a href="https://arxiv.org/pdf/1703.09833v1.pdf" rel="noreferrer">investigations into the loss surface</a> of neural networks.</p>
  
  <p>And classics like <a href="http://www.bioinf.jku.at/publications/older/2304.pdf" rel="noreferrer">this explanation</a> of the vanishing gradient problem.</p>
  
  <p>But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions can currently only be answered empirically. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-16T23:31:02.497Z
Answers.Text: I know that the original question was unclear, but I don't think that the OP was asking about the UAT. The UAT is about the general approximation capabilities of neural networks. It doesn't tell you how to optimally approximate _specific_ functions with neural networks. I edited the original post to clarify the question that I think the OP was asking.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-05T06:58:07.060Z
Answers.LastActivityDate: 2017-04-05T06:58:07.060Z
Answers.UserDisplayName: BlindKungFuMaster
Answers.Body: <p><a href="http://www-isl.stanford.edu/~cover/papers/paper2.pdf" rel="nofollow noreferrer">Thomas Cover</a> and <a href="http://www.inference.org.uk/itila/book.html" rel="nofollow noreferrer">David MacKay</a> proofed the capacity of a perceptron. This proof was <a href="https://arxiv.org/abs/1810.02328" rel="nofollow noreferrer">recently extended</a> to Neural Networks. All of them provide upper bounds for the number of parameters needed to learn something.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-22T08:30:39.993Z
Answers.LastActivityDate: 2018-10-22T08:30:39.993Z
Answers.UserDisplayName: guest
Answers.Body: <p>Not really, I mean at it's core machine learning from an application perspective often seeks produce human level results, but there isn't any theorem describing human understanding of reality.</p>
  
  <p>Like proving computer vision works well is essentially like proving you have a correct understanding of human perception.</p>
  
  <p>It becomes somewhat circular, and while there exists proofs for certain qualities of data, none of them are true. I mean think about trying to describing reality, it exists on a lower dimensional manifold but analytically describing it? Don't think so.</p>
  
  <p>Even proving robustness ends up being somewhat futile since even if you correctly eliminate advasarial examples this doesn't mean you CV application will produce correct results in general, only that the classification is robust(robust and correct are two different things).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-17T04:11:58.370Z
Answers.LastActivityDate: 2020-05-17T04:11:58.370Z
Answers.UserDisplayName: FourierFlux
Body: <p>The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a <strong>mathematical proof</strong> that shows that certain parameters work "better" than others for a certain task?</p>
  
Tags: <neural-networks><machine-learning><reference-request><proofs><universal-approximation-theorems>
Title: Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?
UserDisplayName: Wizard Programming
ThreadText: # Is there a mathematical proof that shows that certain parameters work "better" than others for a certain task?
  
  (Asked by: username_0 on 2017-04-04)
  
  The machine learning community often only provides empirical results, but I am also interested in theoretical results and proofs. Specifically, is there a **mathematical proof** that shows that certain parameters work "better" than others for a certain task?
  
  **Tags:** neural-networks, machine-learning, reference-request, proofs, universal-approximation-theorems
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2017-04-05. Score: 7
  
  There is stuff like the [Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem).
  
  There are also [investigations into the loss surface](https://arxiv.org/pdf/1703.09833v1.pdf) of neural networks.
  
  And classics like [this explanation](http://www.bioinf.jku.at/publications/older/2304.pdf) of the vanishing gradient problem.
  
  But I'm afraid the mathematical theory of neural networks only exists in bits and pieces in many different papers. And many of the most important questions can currently only be answered empirically.
  
  > **Comments:**
  > **username_2**: I know that the original question was unclear, but I don't think that the OP was asking about the UAT. The UAT is about the general approximation capabilities of neural networks. It doesn't tell you how to optimally approximate _specific_ functions with neural networks. I edited the original post to clarify the question that I think the OP was asking.  
  
  ---
  
  ### Answer by username_3 on 2018-10-22. Score: 0
  
  [Thomas Cover](http://www-isl.stanford.edu/~cover/papers/paper2.pdf) and [David MacKay](http://www.inference.org.uk/itila/book.html) proofed the capacity of a perceptron. This proof was [recently extended](https://arxiv.org/abs/1810.02328) to Neural Networks. All of them provide upper bounds for the number of parameters needed to learn something.
  
  
  ---
  
  ### Answer by username_4 on 2020-05-17. Score: -1
  
  Not really, I mean at it's core machine learning from an application perspective often seeks produce human level results, but there isn't any theorem describing human understanding of reality.
  
  Like proving computer vision works well is essentially like proving you have a correct understanding of human perception.
  
  It becomes somewhat circular, and while there exists proofs for certain qualities of data, none of them are true. I mean think about trying to describing reality, it exists on a lower dimensional manifold but analytically describing it? Don't think so.
  
  Even proving robustness ends up being somewhat futile since even if you correctly eliminate advasarial examples this doesn't mean you CV application will produce correct results in general, only that the classification is robust(robust and correct are two different things).
  
  
  ---
  
  