Answers.Body: <p>As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a <em>convenient</em> way of producing an output <span class="math-container">$p \in [0, 1]$</span>, which can be interpreted as a probability, although <a href="https://stats.stackexchange.com/a/437769/82135">that can be misleading or even wrong (if you interpret it as an uncertainty too)</a>.</p>
  <p>You may require the output of the neural network to be a probability, for example, if you use a cross-entropy loss function, although you could in principle produce only <span class="math-container">$0$</span>s or <span class="math-container">$1$</span>s. The probability <span class="math-container">$p$</span> can then be used to decide the class (or label) of the input. For example, if <span class="math-container">$p &gt; \alpha$</span>, then you purposedly decide that the input belongs to class <span class="math-container">$1$</span>, otherwise, it belongs to class <span class="math-container">$0$</span>. The parameter <span class="math-container">$\alpha$</span> is called the <a href="https://developers.google.com/machine-learning/crash-course/classification/thresholding" rel="nofollow noreferrer">classification (or decision) threshold</a>. The choice of this threshold can actually depend on the problem and it is one of the reasons people use the <a href="https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc" rel="nofollow noreferrer">AUC metric</a>, i.e. to avoid choosing this classification threshold.</p>
  <blockquote>
  <p>Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?</p>
  </blockquote>
  <p>The <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">most famous universal approximation theorem for neural networks</a> assumes that the activation functions of the units of the only hidden layer are sigmoids, but it does <strong>not</strong> assume that the output of the network will be squashed to the range <span class="math-container">$[0, 1]$</span>. To be more precise, the UAT (theorem 2 of <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Approximation by Superpositions of a Sigmoidal Function</a>, 1989, by G. Cybenko) states</p>
  <blockquote>
  <p>Let <span class="math-container">$\sigma$</span> be any continuous sigmoidal function. Then finite sums of the</p>
  <p><span class="math-container">$$G(x) = \sum_{j=1}^N \alpha_j \sigma (y_j^T x + \theta_j)$$</span></p>
  <p>are dense in <span class="math-container">$C(I_n)$</span>.</p>
  <p>In other words, given any <span class="math-container">$f \in C(I_n)$</span> and <span class="math-container">$\epsilon &gt; 0$</span>, there is a sum, <span class="math-container">$G(x)$</span>, of the above form, for which</p>
  <p><span class="math-container">$$|G(x) - f(x)| &lt; \epsilon $$</span></p>
  </blockquote>
  <p>Here, <span class="math-container">$f$</span> is the continuous function that you want to approximate, <span class="math-container">$G(x)$</span> is a linear combination of the outputs of <span class="math-container">$N$</span> (which should be arbitrarily big) units of the only hidden layer, <span class="math-container">$I_n$</span> denotes the <span class="math-container">$n$</span>-dimensional unit cube, <span class="math-container">$[0, 1]^n$</span>, <span class="math-container">$C(I_n)$</span> denotes the space of continuous functions on <span class="math-container">$I_n$</span>, <span class="math-container">$x \in I_n$</span> (so the assumption is that the input to the neural network is an element of <span class="math-container">$[0, 1]^n$</span>, i.e. a vector <span class="math-container">$x \in \mathbb{R}^n$</span>, whose entries are between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>) and <span class="math-container">$y_j$</span> and <span class="math-container">$\theta_j$</span> are respectively the weights and bias of the <span class="math-container">$j$</span> unit. The assumption that <span class="math-container">$f$</span> is a real-valued function means that <span class="math-container">$f$</span> can take any value on <span class="math-container">$\mathbb{R}$</span> (i.e. <span class="math-container">$f: [0, 1]^n \rightarrow \mathbb{R}$</span>). You should note that <span class="math-container">$G(x)$</span> is the output of the neural network, which is a combination (where the coefficients are <span class="math-container">$\alpha_j$</span>) of the outputs of the units in the only hidden layer, so there's no restriction on the output of <span class="math-container">$G(x)$</span>, unless you restrict <span class="math-container">$\alpha_i$</span> (but, in this theorem, there's no restriction on the values <span class="math-container">$\alpha_j$</span> can take).</p>
  <p>Of course, if you restrict the output of the neural networks to the range <span class="math-container">$[0, 1]$</span>, you cannot approximate all continuous functions of the form <span class="math-container">$f: [0, 1]^n \rightarrow \mathbb{R}$</span> (because not all of these functions will have the codomain <span class="math-container">$[0, 1]$</span>)! However, the sigmoid has an inverse function, i.e. the <a href="https://developers.google.com/machine-learning/crash-course/classification/thresholding" rel="nofollow noreferrer">logit</a>, so you can reverse the output of such a neural network. So, in this sense (i.e. by reversing the output of the sigmoid), a neural network with a sigmoid as the activation function of the output layer can potentially approximate any continuous function too.</p>
  <p>The UAT above only states the existence of <span class="math-container">$G(x)$</span> (i.e. it's an <a href="https://en.wikipedia.org/wiki/Existence_theorem" rel="nofollow noreferrer">existence theorem</a>). It doesn't tell you how you can find <span class="math-container">$G(x)$</span>. So, if you use a sigmoid as the activation function of the output layer or not is a little bit orthogonal to the universality of neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-23T22:45:07.557Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105895/discussion-on-answer-by-nbro-is-the-use-of-the-sigmoid-as-the-activation-of-the).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-28T08:00:04.003Z
Answers.Text: So basically, people infer it from the universal approximation theorem but there is no actual paper...
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-28T12:34:26.177Z
Answers.Text: @AIM_BLB I don't know if there's a paper or not that formally shows that, if the output of the neural network is a sigmoid, then the NN is able to approximate a class X of functions, but if you have $\sigma(G(x))$, you can retrieve $G(x)$ by applying the inverse of $\sigma$, i.e. $\sigma^{-1}(\sigma(G(x))) = G(x)$. This is more what I meant.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:32:20.467Z
Answers.Text: @nbro have you ever come across a book or paper vagely stating this argument.  I've been looking but I cant find anything...
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:38:08.743Z
Answers.Text: @AnnieTheKatsu Stating what exactly?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:42:13.133Z
Answers.Text: I came across this post and saw your above comment.  It seems like your argument is "common knowledge" but I cant find it written down anywhere...
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:45:21.937Z
Answers.Text: @AnnieTheKatsu If you are talking about the comment above, the point of that comment is that you can reverse the output of a sigmoid, so, **in this sense**, a net with a sigmoid at the output still follows the UAT.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:49:13.837Z
Answers.Text: @AnnieTheKatsu Maybe it helps if you think about it in this other way. Suppose you have a net without a sigmoid at the output. Now, provided the conditions to achieve the universality (e.g. sufficiently number of neurons, etc.), then this network can approximate any continuous function. Now, you have this network that can approximate any continuous function, then you apply the sigmoid to it. This will squash the original output to $[0, 1]$, but you can reverse the output of the sigmoid again to obtain the original output.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:50:27.223Z
Answers.Text: I know, but the issue is I'm looking for a source I can site for this (to show traction in the paper Im writing)
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:50:37.950Z
Answers.Text: @AnnieTheKatsu I think I recently came across a paper that goes into the details of networks with probabilities as output, but I don't remember which paper was that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:51:20.550Z
Answers.Text: If you could find that it would be fantastic... It lookd to me that this is "folklore" which makes it hard to pin down...
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T23:53:30.257Z
Answers.Text: @AnnieTheKatsu Actually, I think that the paper that I am referring to is actually the paper by Cybenko that I am citing in my answer, i.e. "Approximation by Superpositions of a Sigmoidal Function". Although I didn't read that part fully, if I recall correctly, it seemed to me that Cybenko goes into the direction of "networks that classify". Have a look at it. I may be wrong, but the part "We now demonstrate the implications of these results in the context of decision regions" could be useful. Let me know.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-17T00:54:38.157Z
Answers.Text: I looked, the closest is Theorem 3 but still it doesnt use a soft-max/logistic output layer only appeals to Lusin's theorem...
Answers.UserDisplayName: ABIM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-17T01:07:29.533Z
Answers.Text: @AnnieTheKatsu I would need to think about it and read that part more carefully. But I think we are not asking the right question! What do we want to show? Again, note that I am not saying in my answer that a neural network with a sigmoid as output can approximate any continuous function. In fact, I am more saying the opposite. It cannot because not all functions have the range $[0, 1]$. But, as I said, this range can be converted to the reals and vice-versa.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-17T02:49:15.483Z
Answers.Text: Maybe you can extend the UAT to this case by assuming that both $G(x)$ and $f(x)$ can be first converted to a sigmoid, and then maybe you can show that the UAT still applies with this _change of variable_.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-23T14:27:12.540Z
Answers.LastActivityDate: 2020-03-28T01:38:23.713Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: nbro
Body: <p>Neural networks are commonly used for classification tasks, in fact from this <a href="https://ai.stackexchange.com/q/18576/2444">post</a> it seems like that's where they shine brightest.  </p>
  
  <p>However, when we want to classify using neural networks, we often have the <strong>output</strong> layer to take values in <span class="math-container">$[0,1]$</span>; typically, by taking the last layer to be the <em>sigmoid</em> function <span class="math-container">$x \mapsto \frac{e^x}{e^x +1}$</span>. </p>
  
  <p>Can neural networks with a sigmoid as the activation function of the <strong>output</strong> layer approximate continuous functions? Is there an analogue to the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a> for this case? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-23T16:21:31.937Z
Comments.Text: @DuttaA That question has some interesting answers. But note that this question is more focused and specific. The other question is about the _general_ use of sigmoids for neural networks and logistic regression.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-29T01:16:32.633Z
Comments.Text: Why not polyfit?
Comments.UserDisplayName: Serg Kryvonos
Tags: <neural-networks><classification><definitions><computational-learning-theory><sigmoid>
Title: Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?
UserDisplayName: ABIM
ThreadText: # Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?
  
  (Asked by: username_0 on 2020-03-23)
  
  Neural networks are commonly used for classification tasks, in fact from this [post](https://ai.stackexchange.com/q/18576/2444) it seems like that's where they shine brightest.
  
  However, when we want to classify using neural networks, we often have the **output** layer to take values in $[0,1]$; typically, by taking the last layer to be the *sigmoid* function $x \mapsto \frac{e^x}{e^x +1}$.
  
  Can neural networks with a sigmoid as the activation function of the **output** layer approximate continuous functions? Is there an analogue to the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) for this case?
  
  **Tags:** neural-networks, classification, definitions, computational-learning-theory, sigmoid
  > **Comments:**
  > **username_1**: @DuttaA That question has some interesting answers. But note that this question is more focused and specific. The other question is about the _general_ use of sigmoids for neural networks and logistic regression.  
  > **username_2**: Why not polyfit?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-23. Score: 3 | Accepted answer
  
  As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a *convenient* way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although [that can be misleading or even wrong (if you interpret it as an uncertainty too)](https://stats.stackexchange.com/a/437769/82135).
  
  You may require the output of the neural network to be a probability, for example, if you use a cross-entropy loss function, although you could in principle produce only $0$s or $1$s. The probability $p$ can then be used to decide the class (or label) of the input. For example, if $p > \alpha$, then you purposedly decide that the input belongs to class $1$, otherwise, it belongs to class $0$. The parameter $\alpha$ is called the [classification (or decision) threshold](https://developers.google.com/machine-learning/crash-course/classification/thresholding). The choice of this threshold can actually depend on the problem and it is one of the reasons people use the [AUC metric](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc), i.e. to avoid choosing this classification threshold.
  
  > Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?
  
  The [most famous universal approximation theorem for neural networks](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf) assumes that the activation functions of the units of the only hidden layer are sigmoids, but it does **not** assume that the output of the network will be squashed to the range $[0, 1]$. To be more precise, the UAT (theorem 2 of [Approximation by Superpositions of a Sigmoidal Function](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf), 1989, by G. Cybenko) states
  
  > Let $\sigma$ be any continuous sigmoidal function. Then finite sums of the
  >
  > $$G(x) = \sum\_{j=1}^N \alpha\_j \sigma (y\_j^T x + \theta\_j)$$
  >
  > are dense in $C(I\_n)$.
  >
  > In other words, given any $f \in C(I\_n)$ and $\epsilon > 0$, there is a sum, $G(x)$, of the above form, for which
  >
  > $$|G(x) - f(x)| < \epsilon $$
  
  Here, $f$ is the continuous function that you want to approximate, $G(x)$ is a linear combination of the outputs of $N$ (which should be arbitrarily big) units of the only hidden layer, $I\_n$ denotes the $n$-dimensional unit cube, $[0, 1]^n$, $C(I\_n)$ denotes the space of continuous functions on $I\_n$, $x \in I\_n$ (so the assumption is that the input to the neural network is an element of $[0, 1]^n$, i.e. a vector $x \in \mathbb{R}^n$, whose entries are between $0$ and $1$) and $y\_j$ and $\theta\_j$ are respectively the weights and bias of the $j$ unit. The assumption that $f$ is a real-valued function means that $f$ can take any value on $\mathbb{R}$ (i.e. $f: [0, 1]^n \rightarrow \mathbb{R}$). You should note that $G(x)$ is the output of the neural network, which is a combination (where the coefficients are $\alpha\_j$) of the outputs of the units in the only hidden layer, so there's no restriction on the output of $G(x)$, unless you restrict $\alpha\_i$ (but, in this theorem, there's no restriction on the values $\alpha\_j$ can take).
  
  Of course, if you restrict the output of the neural networks to the range $[0, 1]$, you cannot approximate all continuous functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (because not all of these functions will have the codomain $[0, 1]$)! However, the sigmoid has an inverse function, i.e. the [logit](https://developers.google.com/machine-learning/crash-course/classification/thresholding), so you can reverse the output of such a neural network. So, in this sense (i.e. by reversing the output of the sigmoid), a neural network with a sigmoid as the activation function of the output layer can potentially approximate any continuous function too.
  
  The UAT above only states the existence of $G(x)$ (i.e. it's an [existence theorem](https://en.wikipedia.org/wiki/Existence_theorem)). It doesn't tell you how you can find $G(x)$. So, if you use a sigmoid as the activation function of the output layer or not is a little bit orthogonal to the universality of neural networks.
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105895/discussion-on-answer-by-nbro-is-the-use-of-the-sigmoid-as-the-activation-of-the).  
  > **username_0**: So basically, people infer it from the universal approximation theorem but there is no actual paper...  
  > **username_1**: @AIM_BLB I don't know if there's a paper or not that formally shows that, if the output of the neural network is a sigmoid, then the NN is able to approximate a class X of functions, but if you have $\sigma(G(x))$, you can retrieve $G(x)$ by applying the inverse of $\sigma$, i.e. $\sigma^{-1}(\sigma(G(x))) = G(x)$. This is more what I meant.  
  > **username_0**: @nbro have you ever come across a book or paper vagely stating this argument.  I've been looking but I cant find anything...  
  > **username_1**: @AnnieTheKatsu Stating what exactly?  
  > **username_0**: I came across this post and saw your above comment.  It seems like your argument is "common knowledge" but I cant find it written down anywhere...  
  > **username_1**: @AnnieTheKatsu If you are talking about the comment above, the point of that comment is that you can reverse the output of a sigmoid, so, **in this sense**, a net with a sigmoid at the output still follows the UAT.  
  > **username_1**: @AnnieTheKatsu Maybe it helps if you think about it in this other way. Suppose you have a net without a sigmoid at the output. Now, provided the conditions to achieve the universality (e.g. sufficiently number of neurons, etc.), then this network can approximate any continuous function. Now, you have this network that can approximate any continuous function, then you apply the sigmoid to it. This will squash the original output to $[0, 1]$, but you can reverse the output of the sigmoid again to obtain the original output.  
  > **username_0**: I know, but the issue is I'm looking for a source I can site for this (to show traction in the paper Im writing)  
  > **username_1**: @AnnieTheKatsu I think I recently came across a paper that goes into the details of networks with probabilities as output, but I don't remember which paper was that.  
  > **username_0**: If you could find that it would be fantastic... It lookd to me that this is "folklore" which makes it hard to pin down...  
  > **username_1**: @AnnieTheKatsu Actually, I think that the paper that I am referring to is actually the paper by Cybenko that I am citing in my answer, i.e. "Approximation by Superpositions of a Sigmoidal Function". Although I didn't read that part fully, if I recall correctly, it seemed to me that Cybenko goes into the direction of "networks that classify". Have a look at it. I may be wrong, but the part "We now demonstrate the implications of these results in the context of decision regions" could be useful. Let me know.  
  > **username_0**: I looked, the closest is Theorem 3 but still it doesnt use a soft-max/logistic output layer only appeals to Lusin's theorem...  
  > **username_1**: @AnnieTheKatsu I would need to think about it and read that part more carefully. But I think we are not asking the right question! What do we want to show? Again, note that I am not saying in my answer that a neural network with a sigmoid as output can approximate any continuous function. In fact, I am more saying the opposite. It cannot because not all functions have the range $[0, 1]$. But, as I said, this range can be converted to the reals and vice-versa.  
  > **username_1**: Maybe you can extend the UAT to this case by assuming that both $G(x)$ and $f(x)$ can be first converted to a sigmoid, and then maybe you can show that the UAT still applies with this _change of variable_.  
  
  ---
  
  