Answers.Body: <h3>Support vector machines</h3>
  <p>In the paper <a href="https://www.researchgate.net/publication/2531186_A_Note_on_the_Universal_Approximation_Capability_of_Support_Vector_Machines" rel="nofollow noreferrer">A Note on the Universal Approximation Capability of Support Vector Machines</a> (2002) B. Hammer and K. Gersmann investigate the universal function approximation capabilities of SVMs. More specifically, the authors show that SVMs with standard kernels (including Gaussian, polynomial, and several dot product kernels) can approximate any measurable or continuous function up to any desired accuracy. Therefore, SVMs are universal function approximators.</p>
  <h3>Polynomials</h3>
  <p>It is also widely known that we can approximate any continuous function with polynomials (see the <a href="https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem" rel="nofollow noreferrer">Stone-Weierstrass theorem</a>). You can use <a href="https://en.wikipedia.org/wiki/Polynomial_regression" rel="nofollow noreferrer">polynomial regression</a> to fit polynomials to your labelled data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T13:49:03.203Z
Answers.LastActivityDate: 2020-08-30T17:28:01.803Z
Answers.LastEditDate: 2020-08-30T17:28:01.803Z
Answers.UserDisplayName: nbro
Body: <p>The <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="noreferrer">universal approximation theorem</a> states that a <a href="https://en.wikipedia.org/wiki/Feedforward_neural_network" rel="noreferrer">feed-forward neural network </a> with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some assumptions on the activation function are met).</p>
  
  <p>Is there any other machine learning model (apart from any neural network model) that has been <strong>proved</strong> to be an <em>universal</em> function approximator (and that is potentially comparable to neural networks, in terms of usefulness and applicability)? If yes, can you provide a link to a research paper or book that shows the proof?</p>
  
  <p>Similar questions have been asked in the past in other places (e.g. <a href="https://qr.ae/TWI6KL" rel="noreferrer">here</a>, <a href="https://qr.ae/TWI62P" rel="noreferrer">here</a> and <a href="https://cstheory.stackexchange.com/q/7894/34637">here</a>), but they do not provide links to papers or books that show the proofs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-23T11:31:42.920Z
Comments.Text: 3rd link has a link two few books.
  https://cstheory.stackexchange.com/q/7894/34637
Comments.UserDisplayName: Michael D
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-30T18:59:40.247Z
Comments.Text: not a ML algo, but Fourier decomposition can achieve "universal approximation"....
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Tags: <neural-networks><machine-learning><reference-request><function-approximation><universal-approximation-theorems>
Title: Which machine learning models are universal function approximators?
UserDisplayName: nbro
ThreadText: # Which machine learning models are universal function approximators?
  
  (Asked by: username_0 on 2019-05-10)
  
  The [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) states that a [feed-forward neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network)  with a single hidden layer containing a finite number of neurons can approximate any continuous function (provided some assumptions on the activation function are met).
  
  Is there any other machine learning model (apart from any neural network model) that has been **proved** to be an *universal* function approximator (and that is potentially comparable to neural networks, in terms of usefulness and applicability)? If yes, can you provide a link to a research paper or book that shows the proof?
  
  Similar questions have been asked in the past in other places (e.g. [here](https://qr.ae/TWI6KL), [here](https://qr.ae/TWI62P) and [here](https://cstheory.stackexchange.com/q/7894/34637)), but they do not provide links to papers or books that show the proofs.
  
  **Tags:** neural-networks, machine-learning, reference-request, function-approximation, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: 3rd link has a link two few books. https://cstheory.stackexchange.com/q/7894/34637  
  > **username_2**: not a ML algo, but Fourier decomposition can achieve "universal approximation"....  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-05-10. Score: 4 | Accepted answer
  
  ### Support vector machines
  
  In the paper [A Note on the Universal Approximation Capability of Support Vector Machines](https://www.researchgate.net/publication/2531186_A_Note_on_the_Universal_Approximation_Capability_of_Support_Vector_Machines) (2002) B. Hammer and K. Gersmann investigate the universal function approximation capabilities of SVMs. More specifically, the authors show that SVMs with standard kernels (including Gaussian, polynomial, and several dot product kernels) can approximate any measurable or continuous function up to any desired accuracy. Therefore, SVMs are universal function approximators.
  
  ### Polynomials
  
  It is also widely known that we can approximate any continuous function with polynomials (see the [Stone-Weierstrass theorem](https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem)). You can use [polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression) to fit polynomials to your labelled data.
  
  
  ---
  
  