Answers.Body: <p>The approximation theorem says you can approximate anything. But this is kind of meaningless in so far as you can do KNN and get an arbitrary approximation of your training data also.</p>
  
  <p>Proving CNN correctly extract features is, I don't think possible. Or if it is, something involving VC theory is probably the best you can do.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-15T12:15:31.427Z
Answers.Text: The statement "The approximation theorem says you can approximate anything." is false. Which approximation are you talking about? The approximation theorems are usually about approximating continuous functions, which is far from being anything.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-16T01:09:57.893Z
Answers.Text: @nbro OTOH we are only talking about approximation. Obviously continuous functions cannot be discontinuous, but they can approximate many discontinuous functions — probably something like any function with a finite number of discontinuities. That seems to be in agreement with [this](https://arxiv.org/pdf/1709.05289.pdf), for example.
Answers.UserDisplayName: senderle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-15T03:37:00.683Z
Answers.LastActivityDate: 2020-02-15T03:37:00.683Z
Answers.UserDisplayName: FourierFlux
Body: <p>I'm trying to really understand how multi-layer perceptrons work. I want to prove mathematically that MLP's can classify handwritten digits. The only thing I really have is that each perceptron can operate exactly like a logical operand, which obviously can classify things, and, with backpropagation and linear classification, it's obvious that, if a certain pattern exists, it'll activate the correct gates in order to classify correctly, but that is not a mathematical proof.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-14T23:51:27.673Z
Comments.Text: This is probably a special case of the universal approximation theorem. [Here](https://en.m.wikipedia.org/wiki/Universal_approximation_theorem) is a Wikipedia page about the theorem.
Comments.UserDisplayName: senderle
Tags: <classification><proofs><multilayer-perceptrons><perceptron>
Title: Is there a mathematical theory behind why MLP can classify handwritten digits?
UserDisplayName: user8714896
ThreadText: # Is there a mathematical theory behind why MLP can classify handwritten digits?
  
  (Asked by: username_0 on 2020-02-14)
  
  I'm trying to really understand how multi-layer perceptrons work. I want to prove mathematically that MLP's can classify handwritten digits. The only thing I really have is that each perceptron can operate exactly like a logical operand, which obviously can classify things, and, with backpropagation and linear classification, it's obvious that, if a certain pattern exists, it'll activate the correct gates in order to classify correctly, but that is not a mathematical proof.
  
  **Tags:** classification, proofs, multilayer-perceptrons, perceptron
  > **Comments:**
  > **username_1**: This is probably a special case of the universal approximation theorem. [Here](https://en.m.wikipedia.org/wiki/Universal_approximation_theorem) is a Wikipedia page about the theorem.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-02-15. Score: -1
  
  The approximation theorem says you can approximate anything. But this is kind of meaningless in so far as you can do KNN and get an arbitrary approximation of your training data also.
  
  Proving CNN correctly extract features is, I don't think possible. Or if it is, something involving VC theory is probably the best you can do.
  
  > **Comments:**
  > **username_3**: The statement "The approximation theorem says you can approximate anything." is false. Which approximation are you talking about? The approximation theorems are usually about approximating continuous functions, which is far from being anything.  
  > **username_1**: @nbro OTOH we are only talking about approximation. Obviously continuous functions cannot be discontinuous, but they can approximate many discontinuous functions — probably something like any function with a finite number of discontinuities. That seems to be in agreement with [this](https://arxiv.org/pdf/1709.05289.pdf), for example.  
  
  ---
  
  