Answers.Body: <p>Just set all the next layers to map the identity.</p>
  <hr />
  <p>Either construct them  saturating the weights, or consider that each layer is a 1-hidden-layer MLP which is a universal approximator, so it can learn the identity</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-10T21:17:32.790Z
Answers.LastActivityDate: 2024-11-10T21:17:32.790Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>To extend Cybenko's famous result from a single hidden layer to multiple hidden layers, you essentially need to show that any continuous function on a compact support is approximately <em>decomposable</em> into &quot;smaller&quot; continuous functions (not trivially the identity function) on a compact support recursively up to any fixed <span class="math-container">$k$</span> as you specified, each of which is already proved to be approximated universally with a single hidden layer.</p>
  <p>The <a href="https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem" rel="nofollow noreferrer">Stone–Weierstrass theorem</a> can be helpful here.</p>
  <blockquote>
  <p>In mathematical analysis, the Weierstrass approximation theorem states that every continuous function defined on a closed interval [a, b] can be uniformly approximated as closely as desired by a polynomial function... The Stone–Weierstrass theorem generalizes the Weierstrass approximation theorem in two directions: instead of the real interval [a, b], an arbitrary compact Hausdorff space X is considered, and instead of the algebra of polynomial functions, a variety of other families of continuous functions on <span class="math-container">${\displaystyle X}$</span> are shown to suffice</p>
  </blockquote>
  <p>Since continuous functions with a compact support can be universally approximated by the algebra of polynomials or similar families which are Lipschitz continuous primitive recursive functions, they could be further approximately represented by composition of primitive recursive functions. But this may not be always possible and there might be some limit for <span class="math-container">$k$</span> as such recursive non-trivial decomposition may not be always able to continue indefinitely, and bear in mind that many practical advantages of MLP over a fat single hidden layer are empirical findings without rigorous proof so far from deep learning of feature-rich highly nonlinear functions which are intuitively approximately decomposable into many layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-09T08:10:33.603Z
Answers.LastActivityDate: 2024-11-09T08:19:45.207Z
Answers.LastEditDate: 2024-11-09T08:19:45.207Z
Answers.UserDisplayName: cinch
Body: <p>Cybenko showed that if <span class="math-container">$\sigma$</span> is a sigmoidal, continuous function, then for any <span class="math-container">$\varepsilon &gt; 0$</span>, for any continuous function <span class="math-container">$f: [0, 1]^d \to \mathbb{R}$</span>, there exists a function of the form <span class="math-container">$g:x \mapsto \sum\limits_{i = 1}^n a_i\sigma\left( \langle w_i, x \rangle + b_i \right) + b$</span> such that <span class="math-container">$\forall x \in [0, 1]^d, \left| f(x) - g(x) \right| \leq \varepsilon$</span>.</p>
  <p>So each continuous function on a compact set can be approximated by neural networks with <span class="math-container">$1$</span> hidden layer, with activation function <span class="math-container">$\sigma$</span> for this layer and identity for the output layer.</p>
  <p>My question is: how can we deduce from this result that for any fixed <span class="math-container">$k \geq 1$</span>, any continuous function on <span class="math-container">$[0, 1]^d$</span> can be approximated by neural networks with <span class="math-container">$k$</span> hidden layers, using <span class="math-container">$\sigma$</span> (and identity at the end) ?</p>
  <p>Basically, how to go from one hidden layer to <span class="math-container">$k$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-11T14:18:44.133Z
Comments.Text: Welcome to this community! Nice question.
Comments.UserDisplayName: nbro
Tags: <neural-networks><sigmoid><universal-approximation-theorems>
Title: How would you go from 1 to k hidden layers in Cybenko's result that neural networks are universal approximators?
UserDisplayName: JackEight
ThreadText: # How would you go from 1 to k hidden layers in Cybenko's result that neural networks are universal approximators?
  
  (Asked by: username_0 on 2024-11-08)
  
  Cybenko showed that if $\sigma$ is a sigmoidal, continuous function, then for any $\varepsilon > 0$, for any continuous function $f: [0, 1]^d \to \mathbb{R}$, there exists a function of the form $g:x \mapsto \sum\limits\_{i = 1}^n a\_i\sigma\left( \langle w\_i, x \rangle + b\_i \right) + b$ such that $\forall x \in [0, 1]^d, \left| f(x) - g(x) \right| \leq \varepsilon$.
  
  So each continuous function on a compact set can be approximated by neural networks with $1$ hidden layer, with activation function $\sigma$ for this layer and identity for the output layer.
  
  My question is: how can we deduce from this result that for any fixed $k \geq 1$, any continuous function on $[0, 1]^d$ can be approximated by neural networks with $k$ hidden layers, using $\sigma$ (and identity at the end) ?
  
  Basically, how to go from one hidden layer to $k$
  
  **Tags:** neural-networks, sigmoid, universal-approximation-theorems
  
  > **Comments:**
  > **username_1**: Welcome to this community! Nice question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-11-10. Score: 1 | Accepted answer
  
  Just set all the next layers to map the identity.
  
  ---
  
  Either construct them saturating the weights, or consider that each layer is a 1-hidden-layer MLP which is a universal approximator, so it can learn the identity
  
  
  ---
  
  ### Answer by username_3 on 2024-11-09. Score: 1
  
  To extend Cybenko's famous result from a single hidden layer to multiple hidden layers, you essentially need to show that any continuous function on a compact support is approximately *decomposable* into "smaller" continuous functions (not trivially the identity function) on a compact support recursively up to any fixed $k$ as you specified, each of which is already proved to be approximated universally with a single hidden layer.
  
  The [Stone–Weierstrass theorem](https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem) can be helpful here.
  
  > In mathematical analysis, the Weierstrass approximation theorem states that every continuous function defined on a closed interval [a, b] can be uniformly approximated as closely as desired by a polynomial function... The Stone–Weierstrass theorem generalizes the Weierstrass approximation theorem in two directions: instead of the real interval [a, b], an arbitrary compact Hausdorff space X is considered, and instead of the algebra of polynomial functions, a variety of other families of continuous functions on ${\displaystyle X}$ are shown to suffice
  
  Since continuous functions with a compact support can be universally approximated by the algebra of polynomials or similar families which are Lipschitz continuous primitive recursive functions, they could be further approximately represented by composition of primitive recursive functions. But this may not be always possible and there might be some limit for $k$ as such recursive non-trivial decomposition may not be always able to continue indefinitely, and bear in mind that many practical advantages of MLP over a fat single hidden layer are empirical findings without rigorous proof so far from deep learning of feature-rich highly nonlinear functions which are intuitively approximately decomposable into many layers.
  
  
  ---
  
  