Answers.Body: <p>I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to apply a scaling.</p>
  <p>In any case I would consider what the target distribution is before using a sigmoid and applying a scaling. The thing about a sigmoid is that the model has to be very confident in order to predict a value close to the boundaries. So in your case, the model would have to be really confident about something for it to output a -5, and it would have to be really confident of the opposite to output a +20. I would be surprised if this is what you are really looking for in this case.</p>
  <p>Let's say your target distribution is uniform. Then your best bet is to use a linear activation, and maybe use max/min to clip it. Then it would behave a lot like a ReLU.</p>
  <p>If I haven't quite nailed it with this answer, can you give more info on what you are predicting?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T21:04:19.920Z
Answers.LastActivityDate: 2020-11-20T21:04:19.920Z
Answers.UserDisplayName: Alexander Soare
Answers.Body: <p>There are several functions that can be denoted as <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer"><em>sigmoid functions</em></a>, such as the <a href="https://en.wikipedia.org/wiki/Logistic_function" rel="nofollow noreferrer"><strong>logistic function</strong></a> and the <strong>hyperbolic tangent</strong>, given that they have an <span class="math-container">$S$</span>-shaped curve. You can find more info about them in <a href="https://en.wikipedia.org/wiki/Sigmoid_function" rel="nofollow noreferrer">the related Wikipedia article</a>.</p>
  <p>However, when people use the term <em>sigmoid function</em>, they typically refer to the <a href="https://en.wikipedia.org/wiki/Logistic_function" rel="nofollow noreferrer"><em>logistic function</em></a>, which is a function of the form <span class="math-container">$$\sigma: \mathbb{R} \rightarrow [0, 1],$$</span> So, it accepts any real number and produces a number between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>, so you can pass any real number (as input) to the sigmoid.</p>
  <p>The <em>standard</em> logistic function <span class="math-container">$\sigma$</span> is mathematically defined as follows</p>
  <p><span class="math-container">$$\sigma(x)=\frac{1}{1+e^{-x}}\in [0, 1]$$</span></p>
  <p>Here is the graph of the logistic function. The <span class="math-container">$y$</span>-axis contains the output values, i.e. <span class="math-container">$\sigma(x)$</span> for any <span class="math-container">$x \in \mathbb{R}$</span>, while the <span class="math-container">$x$</span>-axis contains the input values, i.e. <span class="math-container">$x \in \mathbb{R}$</span>.</p>
  <p><a href="https://i.sstatic.net/ApvJ3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ApvJ3.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>That function can be between 0 and 1 but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?</p>
  </blockquote>
  <p>No, in principle, you do <strong>not</strong> have to scale your inputs, for the reasons I have just explained. However, this does not mean that, in practice, it may not be a good idea to <a href="https://stats.stackexchange.com/q/10289/82135">standardize or normalize</a> your inputs before feeding them into the neural network. See the <strong>vanishing gradient problem (VGP)</strong> <a href="https://machinelearningmastery.com/how-to-fix-vanishing-gradients-using-the-rectified-linear-activation-function/" rel="nofollow noreferrer">in this article</a> (by J. Brownlee), <a href="http://neuralnetworksanddeeplearning.com/chap5.html" rel="nofollow noreferrer">or this one</a> (by M. Nielsen), or in <a href="https://www.youtube.com/watch?v=qhXZsFVxGKo&amp;ab_channel=Deeplearning.ai" rel="nofollow noreferrer">this video</a> (by A. Ng). Moreover, you should probably actually use another activation function, such as the ReLU, which does not suffer from the VGP (which, nevertheless, should not occur, if your neural network does not contain more than a couple of hidden layers).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T12:26:13.997Z
Answers.LastActivityDate: 2020-11-20T13:17:13.600Z
Answers.LastEditDate: 2020-11-20T13:17:13.600Z
Answers.UserDisplayName: nbro
Body: <p>I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T09:21:29.273Z
Comments.Text: I believe it's not true because the scaling is done by the weights, which are learned by the ANN during the optimization. Hence, it's the ANN job, not ours.
Comments.UserDisplayName: Javier TG
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T22:28:56.377Z
Comments.Text: The sigmoid function can take any value but it is recommended that inputs are scaled to be lower. The most common sigmoid function can only output values between 0 and 1. If you mean that the values supposed to be outputted by then network are between -5 and 20 then you could add 5 and divide the values by 25 and reverse that after training. Whoever said that the input values needed to be between -5 and 20 was wrong or not understanding your question.
Comments.UserDisplayName: S2673
Tags: <neural-networks><activation-functions><sigmoid>
Title: How to use sigmoid as transfer function when input is not (0,1) range in ANN?
UserDisplayName: J. Doe
ThreadText: # How to use sigmoid as transfer function when input is not (0,1) range in ANN?
  
  (Asked by: username_0 on 2020-11-20)
  
  I am building my first ANN from scratch. I know that I need a transfer function and I want to use the sigmoid function as my teacher recommended that. That function can be between 0 and 1, but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?
  
  **Tags:** neural-networks, activation-functions, sigmoid
  > **Comments:**
  > **username_1**: I believe it's not true because the scaling is done by the weights, which are learned by the ANN during the optimization. Hence, it's the ANN job, not ours.  
  > **username_2**: The sigmoid function can take any value but it is recommended that inputs are scaled to be lower. The most common sigmoid function can only output values between 0 and 1. If you mean that the values supposed to be outputted by then network are between -5 and 20 then you could add 5 and divide the values by 25 and reverse that after training. Whoever said that the input values needed to be between -5 and 20 was wrong or not understanding your question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-11-20. Score: 1
  
  I presume when you say input you may be referring to the target values (the things you are trying to predict). If not, then some parts of your question might not make sense, like your proposal to apply a scaling.
  
  In any case I would consider what the target distribution is before using a sigmoid and applying a scaling. The thing about a sigmoid is that the model has to be very confident in order to predict a value close to the boundaries. So in your case, the model would have to be really confident about something for it to output a -5, and it would have to be really confident of the opposite to output a +20. I would be surprised if this is what you are really looking for in this case.
  
  Let's say your target distribution is uniform. Then your best bet is to use a linear activation, and maybe use max/min to clip it. Then it would behave a lot like a ReLU.
  
  If I haven't quite nailed it with this answer, can you give more info on what you are predicting?
  
  
  ---
  
  ### Answer by username_4 on 2020-11-20. Score: 1
  
  There are several functions that can be denoted as [*sigmoid functions*](https://en.wikipedia.org/wiki/Sigmoid_function), such as the [**logistic function**](https://en.wikipedia.org/wiki/Logistic_function) and the **hyperbolic tangent**, given that they have an $S$-shaped curve. You can find more info about them in [the related Wikipedia article](https://en.wikipedia.org/wiki/Sigmoid_function).
  
  However, when people use the term *sigmoid function*, they typically refer to the [*logistic function*](https://en.wikipedia.org/wiki/Logistic_function), which is a function of the form $$\sigma: \mathbb{R} \rightarrow [0, 1],$$ So, it accepts any real number and produces a number between $0$ and $1$, so you can pass any real number (as input) to the sigmoid.
  
  The *standard* logistic function $\sigma$ is mathematically defined as follows
  
  $$\sigma(x)=\frac{1}{1+e^{-x}}\in [0, 1]$$
  
  Here is the graph of the logistic function. The $y$-axis contains the output values, i.e. $\sigma(x)$ for any $x \in \mathbb{R}$, while the $x$-axis contains the input values, i.e. $x \in \mathbb{R}$.
  
  [![enter image description here](https://i.sstatic.net/ApvJ3.png)](https://i.sstatic.net/ApvJ3.png)
  
  > That function can be between 0 and 1 but my input values for the network are between -5 and 20. Someone told me that I need to scale the function so that it is in the range of -5 and 20 instead of 0 and 1. Is this true? Why?
  
  No, in principle, you do **not** have to scale your inputs, for the reasons I have just explained. However, this does not mean that, in practice, it may not be a good idea to [standardize or normalize](https://stats.stackexchange.com/q/10289/82135) your inputs before feeding them into the neural network. See the **vanishing gradient problem (VGP)** [in this article](https://machinelearningmastery.com/how-to-fix-vanishing-gradients-using-the-rectified-linear-activation-function/) (by J. Brownlee), [or this one](http://neuralnetworksanddeeplearning.com/chap5.html) (by M. Nielsen), or in [this video](https://www.youtube.com/watch?v=qhXZsFVxGKo&ab_channel=Deeplearning.ai) (by A. Ng). Moreover, you should probably actually use another activation function, such as the ReLU, which does not suffer from the VGP (which, nevertheless, should not occur, if your neural network does not contain more than a couple of hidden layers).
  
  
  ---
  
  