Answers.Body: <p>I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point.</p>
  <p>I don't think there is a relation between this argument and universal approximation theorems, which state that certain classes of neural networks can approximate any <em>function</em> in certain domains, rather than any <em>learning algorithm</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T20:01:45.023Z
Answers.Text: Perhaps I am indeed mixing up two different things. Nonetheless, if a neural network is able to approximate any function, then, given sufficient data (however much "sufficient" may be), it should be able to accomplish anything any other learning algorithm, including the more "conventional" machine learning algorithms, can, right?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T22:14:44.350Z
Answers.Text: "should be able to accomplish anything any other learning algorithm [...] can": there is a subtlety though. Universal approximation only state *existence* of neural networks that approximate functions, but say nothing about algorithms for finding them.
Answers.UserDisplayName: Dani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T22:20:41.110Z
Answers.Text: But doesn't this comment agree with what I said? Since neural networks are able to approximate any function, then it should (assuming sufficient data) be able to approximate the same functions that any other learning algorithms do? Or am I still misunderstanding?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T22:27:34.983Z
Answers.Text: Let me put this another way: the functions defined by neural networks with specific weights and architectures set are different from the algorithms used to find them. Universal approximation theorems say: given a target function, a neural network that does not differ from it to a given degree of tolerance exists. However, a classical algorithm might find an acceptable function, whereas you might not be able to find such neural network. About the "sufficient data" question: UA theorems are valid for certain classes of NNs with infinite VC dimension, so infinite sample complexity.
Answers.UserDisplayName: Dani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T22:33:31.857Z
Answers.Text: Oh, I see what you're saying: The UA theorems state that, for any target function, there exists a neural network **architecture** that can approximate it; but that isn't necessarily the architecture that you're working with, and you might not be able to find a neural network architecture that can approximate said target function. Is that correct?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T22:35:09.243Z
Answers.Text: An architecture and specific weights for that architecture. Even if your architecture was correct, you still need to find the specific weights.
Answers.UserDisplayName: Dani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T19:52:38.910Z
Answers.LastActivityDate: 2020-11-07T19:52:38.910Z
Answers.UserDisplayName: Dani
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. In chapter <strong>1.2.1 Single Computational Layer: The Perceptron</strong>, the author says the following:</p>
  <blockquote>
  <p>Different choices of activation functions can be used to simulate different types of models used in machine learning, like <em>least-squares regression with numeric targets</em>, the <em>support vector machine</em>, or a <em>logistic regression classifier</em>. Most of the basic machine learning models can be easily represented as simple neural network architectures.</p>
  </blockquote>
  <p>I remember reading something about it being mathematically proven that neural networks can approximate any function, and therefore any machine learning method, or something along these lines. Am I remembering this correctly? Would someone please clarify my thoughts?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T19:16:47.230Z
Comments.Text: See https://en.wikipedia.org/wiki/Universal_approximation_theorem
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T19:56:19.243Z
Comments.Text: @pasabaporaqui Thanks for that. Chapter **1.5 The Secrets to the Power of Function Composition** seems to say the following: "This basic idea is the essence of the universal approximation theorem of neural networks. In fact, the proof of the ability of squashing functions to approximate any function is conceptually similar to that of kernels at least at an intuitive level. However, the number of base functions required to reach a high level of approximation can be extremely large in both cases, potentially increasing the data-centric requirements to an unmanageable level. [...]
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-07T19:56:25.600Z
Comments.Text: [...] For this reason, shallow networks face the persistent problem of overfitting. The universal approximation theorem asserts the ability to well-approximate the function implicit in the training data, but makes no guarantee about whether the function can be generalized to unseen test data."
Comments.UserDisplayName: The Pointer
Tags: <neural-networks><machine-learning><activation-functions><universal-approximation-theorems>
Title: Can most of the basic machine learning models be easily represented as simple neural network architectures?
UserDisplayName: The Pointer
ThreadText: # Can most of the basic machine learning models be easily represented as simple neural network architectures?
  
  (Asked by: username_0 on 2020-11-07)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. In chapter **1.2.1 Single Computational Layer: The Perceptron**, the author says the following:
  
  > Different choices of activation functions can be used to simulate different types of models used in machine learning, like *least-squares regression with numeric targets*, the *support vector machine*, or a *logistic regression classifier*. Most of the basic machine learning models can be easily represented as simple neural network architectures.
  
  I remember reading something about it being mathematically proven that neural networks can approximate any function, and therefore any machine learning method, or something along these lines. Am I remembering this correctly? Would someone please clarify my thoughts?
  
  **Tags:** neural-networks, machine-learning, activation-functions, universal-approximation-theorems
  > **Comments:**
  > **username_1**: See https://en.wikipedia.org/wiki/Universal_approximation_theorem  
  > **username_0**: @pasabaporaqui Thanks for that. Chapter **1.5 The Secrets to the Power of Function Composition** seems to say the following: "This basic idea is the essence of the universal approximation theorem of neural networks. In fact, the proof of the ability of squashing functions to approximate any function is conceptually similar to that of kernels at least at an intuitive level. However, the number of base functions required to reach a high level of approximation can be extremely large in both cases, potentially increasing the data-centric requirements to an unmanageable level. [...]  
  > **username_0**: [...] For this reason, shallow networks face the persistent problem of overfitting. The universal approximation theorem asserts the ability to well-approximate the function implicit in the training data, but makes no guarantee about whether the function can be generalized to unseen test data."  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-07. Score: 1 | Accepted answer
  
  I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point.
  
  I don't think there is a relation between this argument and universal approximation theorems, which state that certain classes of neural networks can approximate any *function* in certain domains, rather than any *learning algorithm*.
  
  > **Comments:**
  > **username_0**: Perhaps I am indeed mixing up two different things. Nonetheless, if a neural network is able to approximate any function, then, given sufficient data (however much "sufficient" may be), it should be able to accomplish anything any other learning algorithm, including the more "conventional" machine learning algorithms, can, right?  
  > **username_2**: "should be able to accomplish anything any other learning algorithm [...] can": there is a subtlety though. Universal approximation only state *existence* of neural networks that approximate functions, but say nothing about algorithms for finding them.  
  > **username_0**: But doesn't this comment agree with what I said? Since neural networks are able to approximate any function, then it should (assuming sufficient data) be able to approximate the same functions that any other learning algorithms do? Or am I still misunderstanding?  
  > **username_2**: Let me put this another way: the functions defined by neural networks with specific weights and architectures set are different from the algorithms used to find them. Universal approximation theorems say: given a target function, a neural network that does not differ from it to a given degree of tolerance exists. However, a classical algorithm might find an acceptable function, whereas you might not be able to find such neural network. About the "sufficient data" question: UA theorems are valid for certain classes of NNs with infinite VC dimension, so infinite sample complexity.  
  > **username_0**: Oh, I see what you're saying: The UA theorems state that, for any target function, there exists a neural network **architecture** that can approximate it; but that isn't necessarily the architecture that you're working with, and you might not be able to find a neural network architecture that can approximate said target function. Is that correct?  
  > **username_2**: An architecture and specific weights for that architecture. Even if your architecture was correct, you still need to find the specific weights.  
  
  ---
  
  