Answers.Body: <p>This says that if you can approximate a function with one layer, you can also approximate it with multiple layers because you can make the extra layers do nothing.</p>
  <p>The universal approximation theorem tells you there is at least one way to approximate any function - which is the way that puts exponentially many nodes into one layer. It doesn't tell you whether there's a more efficient way using more layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-24T21:50:54.390Z
Answers.LastActivityDate: 2023-04-24T22:02:20.333Z
Answers.LastEditDate: 2023-04-24T22:02:20.333Z
Answers.UserDisplayName: Criticize SE actions means ban
Body: <p>I have a question about the explanation of universal approximation theorem provided by wikipedia.
  <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29</a></p>
  <p>It says, after a universal approximation of one layer that approximates <span class="math-container">$f$</span> with <span class="math-container">$g$</span> with
  <span class="math-container">\begin{align}
  \sup_{x\in K}  ||f(x) - g(x)|| &lt; \epsilon\\
  \text{where} \quad 
  g(x) = C \cdot [\sigma \circ (A \cdot x + b)],\\
  \end{align}</span><br />
  that</p>
  <blockquote>
  <p>an <span class="math-container">$f$</span> can also be approximated by a network of greater depth by using the same construction for the first layer and approximating the identity function with later layers.</p>
  </blockquote>
  <p>I cannot understand how I should interpret this. After the first layer, it outputs <span class="math-container">$g$</span> which is an approximation of <span class="math-container">$f$</span> right? Why is identity function following here? Doesn’t it keep outputting the same <span class="math-container">$g$</span> without its improving quality?</p>
  
Tags: <deep-learning><universal-approximation-theorems>
Title: Why do we need the identify function when approximating a function with a neural network with multiple layers?
UserDisplayName: user1168149
ThreadText: # Why do we need the identify function when approximating a function with a neural network with multiple layers?
  
  (Asked by: username_0 on 2023-04-23)
  
  I have a question about the explanation of universal approximation theorem provided by wikipedia.
  <https://en.wikipedia.org/wiki/Universal_approximation_theorem#cite_note-:0-29>
  
  It says, after a universal approximation of one layer that approximates $f$ with $g$ with
  \begin{align}
  \sup\_{x\in K} ||f(x) - g(x)|| < \epsilon\\
  \text{where} \quad
  g(x) = C \cdot [\sigma \circ (A \cdot x + b)],\\
  \end{align}  
  that
  
  > an $f$ can also be approximated by a network of greater depth by using the same construction for the first layer and approximating the identity function with later layers.
  
  I cannot understand how I should interpret this. After the first layer, it outputs $g$ which is an approximation of $f$ right? Why is identity function following here? Doesn’t it keep outputting the same $g$ without its improving quality?
  
  **Tags:** deep-learning, universal-approximation-theorems
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-24. Score: 4 | Accepted answer
  
  This says that if you can approximate a function with one layer, you can also approximate it with multiple layers because you can make the extra layers do nothing.
  
  The universal approximation theorem tells you there is at least one way to approximate any function - which is the way that puts exponentially many nodes into one layer. It doesn't tell you whether there's a more efficient way using more layers.
  
  
  ---
  
  