Answers.Body: <p>The classical version of the universal approximation theorem states that, roughly, given a continuous function <span class="math-container">$f \colon [0, 1]^n \to [0, 1]^n$</span>, <em>there exists</em> a single layer neural network and a set of weights and biases such that this network approximates the given function <span class="math-container">$f$</span> arbitrarily well.</p>
  <p>It doesn't say anything about how you obtain such weights: the result is entirely independent of the way you train your network, and what it says is that the set of single-layer neural networks has enough capacity, in principle, to approximate any continuous function arbitrarily well.</p>
  <p>This kind of result is fairly common in mathematics: it is shown that a suitable object must exist, but the proof is <em>non-constructive</em>, i.e. it doesn't actually show you how to get that object.</p>
  <p>You're indeed correct that the backpropagation algorithm might not be able to find suitable weights. In fact, if you fix an architecture in advance, there might not even be any weights that would lead to a neural network that is a good approximation to <span class="math-container">$f$</span>.</p>
  <h3>Why do we care?</h3>
  <p>Universality tells us, at least in principle, that we should be able to approximate the function we want, if we pick an appropriate architecture. The <a href="https://computing.dcu.ie/%7Ehumphrys/Notes/Neural/single.neural.html" rel="nofollow noreferrer">XOR Problem</a>, for example, affects a type of model called a <em>perceptron</em> (essentially a single neuron neural network), and says that the XOR function cannot be approximated by perceptrons. It is always useful to know the kinds of functions that can and cannot be expressed by a certain type of model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-03T21:21:48.357Z
Answers.Text: What was the usefulness of this article then? It sounds like he's saying functions can be approximated with sigmoid "towers" and "steps", but the neural network would have to have specific weights to force towers and steps, which is not even happening at all in the first place?
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T08:21:12.280Z
Answers.Text: The point is that, if universal approximation wasn't true, there might be types of functions that you can never hope to represent with a neural network, because there simply isn't any set of weights which would do this. This would have practical implications because it would mean that no matter how much you train or look for a good architecture, you might still fail to find a good neural net for your problem. The whole section where he explicitly constructs weights is just to help you understand why the result is true.
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T16:37:09.493Z
Answers.Text: So he gave an example of a *possible* way sigmoid functions can represent any function, but it isn't necessarily *the way* the neural network will *use* the sigmiods (by using stochastic gradient descent to get the best weights and biases), just *a way* that shows proof of concept.
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T17:04:37.087Z
Answers.Text: That's correct, @sangstar. The theorem doesn't care what your algorithm is for finding the weights (and, conversely, makes no promises that backpropagation will find very good weights). It just says that there are some weights which would be a good fit: it doesn't tell you how to find them, and it doesn't say that backpropagation will converge to these weights: backprop isn't even mentioned once in the paper proving the theorem!
Answers.UserDisplayName: htl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-04T17:15:18.220Z
Answers.Text: I suppose I find it counterintuitive, because the way the offer represents the “building blocks” it makes me feel like those “steps” and “towers” ought to be the orthonormal basis functions (because of how “fundamental” they look” that should form any functions, and that they should not be arbitrarily chosen but instead be *the* functions to use. Just an impression
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-03T08:59:23.203Z
Answers.LastActivityDate: 2021-04-03T08:59:23.203Z
Answers.UserDisplayName: htl
Body: <p><a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">This</a> article attempts to provide a graphical justification of the universal approximation theorem.</p>
  <p>It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded constant function or step function, and thus can therefore to a reasonable degree of approximation produce any function by essentially splitting up any function into a cluster (linear combination?) of these towers or steps.</p>
  <p>However, he produced the steps and towers using specific weight parametrizations.</p>
  <p>However, since when are we allowed to specify weights and biases? Isn't this all out of our hands and in the hands of cost function minimization?</p>
  <p>I don't understand why he was dealing with setting weights to this, biases to that, when in my experience that is all done by &quot;the machine&quot; to minimize the cost function. I doubt the weights to minimize the cost function are arranged in the ways specified in order to form the towers and steps that were formed in this tutorial, so I kind of don't understand what all the hub-ub is all about.</p>
  
Tags: <feedforward-neural-networks><stochastic-gradient-descent><universal-approximation-theorems>
Title: Issue with graphical interpretation of the universal approximation theorem
UserDisplayName: sangstar
ThreadText: # Issue with graphical interpretation of the universal approximation theorem
  
  (Asked by: username_0 on 2021-04-03)
  
  [This](http://neuralnetworksanddeeplearning.com/chap4.html) article attempts to provide a graphical justification of the universal approximation theorem.
  
  It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded constant function or step function, and thus can therefore to a reasonable degree of approximation produce any function by essentially splitting up any function into a cluster (linear combination?) of these towers or steps.
  
  However, he produced the steps and towers using specific weight parametrizations.
  
  However, since when are we allowed to specify weights and biases? Isn't this all out of our hands and in the hands of cost function minimization?
  
  I don't understand why he was dealing with setting weights to this, biases to that, when in my experience that is all done by "the machine" to minimize the cost function. I doubt the weights to minimize the cost function are arranged in the ways specified in order to form the towers and steps that were formed in this tutorial, so I kind of don't understand what all the hub-ub is all about.
  
  **Tags:** feedforward-neural-networks, stochastic-gradient-descent, universal-approximation-theorems
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-03. Score: 1
  
  The classical version of the universal approximation theorem states that, roughly, given a continuous function $f \colon [0, 1]^n \to [0, 1]^n$, *there exists* a single layer neural network and a set of weights and biases such that this network approximates the given function $f$ arbitrarily well.
  
  It doesn't say anything about how you obtain such weights: the result is entirely independent of the way you train your network, and what it says is that the set of single-layer neural networks has enough capacity, in principle, to approximate any continuous function arbitrarily well.
  
  This kind of result is fairly common in mathematics: it is shown that a suitable object must exist, but the proof is *non-constructive*, i.e. it doesn't actually show you how to get that object.
  
  You're indeed correct that the backpropagation algorithm might not be able to find suitable weights. In fact, if you fix an architecture in advance, there might not even be any weights that would lead to a neural network that is a good approximation to $f$.
  
  ### Why do we care?
  
  Universality tells us, at least in principle, that we should be able to approximate the function we want, if we pick an appropriate architecture. The [XOR Problem](https://computing.dcu.ie/%7Ehumphrys/Notes/Neural/single.neural.html), for example, affects a type of model called a *perceptron* (essentially a single neuron neural network), and says that the XOR function cannot be approximated by perceptrons. It is always useful to know the kinds of functions that can and cannot be expressed by a certain type of model.
  
  > **Comments:**
  > **username_0**: What was the usefulness of this article then? It sounds like he's saying functions can be approximated with sigmoid "towers" and "steps", but the neural network would have to have specific weights to force towers and steps, which is not even happening at all in the first place?  
  > **username_1**: The point is that, if universal approximation wasn't true, there might be types of functions that you can never hope to represent with a neural network, because there simply isn't any set of weights which would do this. This would have practical implications because it would mean that no matter how much you train or look for a good architecture, you might still fail to find a good neural net for your problem. The whole section where he explicitly constructs weights is just to help you understand why the result is true.  
  > **username_0**: So he gave an example of a *possible* way sigmoid functions can represent any function, but it isn't necessarily *the way* the neural network will *use* the sigmiods (by using stochastic gradient descent to get the best weights and biases), just *a way* that shows proof of concept.  
  > **username_1**: That's correct, @username_0. The theorem doesn't care what your algorithm is for finding the weights (and, conversely, makes no promises that backpropagation will find very good weights). It just says that there are some weights which would be a good fit: it doesn't tell you how to find them, and it doesn't say that backpropagation will converge to these weights: backprop isn't even mentioned once in the paper proving the theorem!  
  > **username_0**: I suppose I find it counterintuitive, because the way the offer represents the “building blocks” it makes me feel like those “steps” and “towers” ought to be the orthonormal basis functions (because of how “fundamental” they look” that should form any functions, and that they should not be arbitrarily chosen but instead be *the* functions to use. Just an impression  
  
  ---
  
  