Answers.Body: <p>In regression, the goal is to approximate a function <span class="math-container">$f: \mathcal{I} \rightarrow \mathbb{R}$</span>, so <span class="math-container">$f(x) \in \mathbb{R}$</span>. In other words, in regression, you want to learn a function whose outputs can be any number, so not necessarily just a number in the range <span class="math-container">$[0, 1]$</span>.</p>
  <p>You use the sigmoid as the activation function of the output layer of a neural network, for example, when you want to interpret it as a probability. This is typically done when you are using the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy" rel="nofollow noreferrer">binary</a> <a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="nofollow noreferrer">cross-entropy</a> loss function, i.e. you are solving a <strong>binary classification</strong> problem (i.e. the output can either be one of two classes/labels).</p>
  <p>By default, <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense" rel="nofollow noreferrer"><code>tf.keras.layers.Dense</code></a> does not use any activation function, which means that the output of your neural network is indeed just a linear combination of the inputs from the previous layer. This should be fine for a regression problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T11:03:07.753Z
Answers.Text: Thanks for the explanation. But if the weights in a neural network can be any number, won't the output also vary wildly? Miles per gallon should be a value of around 20 to 100 for example, but if the random weight is 2.000.000, won't you get a crazy output?
Answers.UserDisplayName: Kokodoko
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T11:09:19.167Z
Answers.Text: @Kokodoko Yes, this can happen, but the weights should converge to some reasonable values, once you optimize the objective function. Moreover, you can also limit the weights  and activations e.g. by using specific activation functions that squash the inputs to the neurons to certain ranges, which is your case (in that example, they are using the sigmoid in the hidden layer, which squashes the inputs to the range [0, 1]). You can also have regularization of the weights, to make them small.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-11T10:20:33.813Z
Answers.LastActivityDate: 2021-03-11T10:20:33.813Z
Answers.UserDisplayName: nbro
Body: <p>I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses &quot;sigmoid&quot; to bring the results back to a nice number between 0 and 1.</p>
  <p>But in this beginner example on the TensorFlow website, the output layer has no activation function at all? Is this allowed? Wouldn't the result be a crazy number that's all over the place? Or maybe TensorFlow has a hidden default activation?</p>
  <p>This code is from the <a href="https://codelabs.developers.google.com/codelabs/tfjs-training-regression/index.html#3" rel="nofollow noreferrer">example</a> where you predict miles per gallon based on horsepower of a car.</p>
  <pre><code>// input layer
  model.add(tf.layers.dense({inputShape: [1], units: 1}));
  
  // hidden layer
  model.add(tf.layers.dense({units: 50, activation: 'sigmoid'}));
  
  // output layer - no activation needed ???
  model.add(tf.layers.dense({units: 1}));
  </code></pre>
  
Tags: <neural-networks><tensorflow><activation-functions><regression>
Title: Why is no activation function needed for the output layer of a neural network for regression?
UserDisplayName: Kokodoko
ThreadText: # Why is no activation function needed for the output layer of a neural network for regression?
  
  (Asked by: username_0 on 2021-03-11)
  
  I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back to a nice number between 0 and 1.
  
  But in this beginner example on the TensorFlow website, the output layer has no activation function at all? Is this allowed? Wouldn't the result be a crazy number that's all over the place? Or maybe TensorFlow has a hidden default activation?
  
  This code is from the [example](https://codelabs.developers.google.com/codelabs/tfjs-training-regression/index.html#3) where you predict miles per gallon based on horsepower of a car.
  
  ```
  // input layer
  model.add(tf.layers.dense({inputShape: [1], units: 1}));
  
  // hidden layer
  model.add(tf.layers.dense({units: 50, activation: 'sigmoid'}));
  
  // output layer - no activation needed ???
  model.add(tf.layers.dense({units: 1}));
  
  ```
  
  **Tags:** neural-networks, tensorflow, activation-functions, regression
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-11. Score: 3 | Accepted answer
  
  In regression, the goal is to approximate a function $f: \mathcal{I} \rightarrow \mathbb{R}$, so $f(x) \in \mathbb{R}$. In other words, in regression, you want to learn a function whose outputs can be any number, so not necessarily just a number in the range $[0, 1]$.
  
  You use the sigmoid as the activation function of the output layer of a neural network, for example, when you want to interpret it as a probability. This is typically done when you are using the [binary](https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy) [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) loss function, i.e. you are solving a **binary classification** problem (i.e. the output can either be one of two classes/labels).
  
  By default, [`tf.keras.layers.Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) does not use any activation function, which means that the output of your neural network is indeed just a linear combination of the inputs from the previous layer. This should be fine for a regression problem.
  
  > **Comments:**
  > **username_0**: Thanks for the explanation. But if the weights in a neural network can be any number, won't the output also vary wildly? Miles per gallon should be a value of around 20 to 100 for example, but if the random weight is 2.000.000, won't you get a crazy output?  
  > **username_1**: @username_0 Yes, this can happen, but the weights should converge to some reasonable values, once you optimize the objective function. Moreover, you can also limit the weights  and activations e.g. by using specific activation functions that squash the inputs to the neurons to certain ranges, which is your case (in that example, they are using the sigmoid in the hidden layer, which squashes the inputs to the range [0, 1]). You can also have regularization of the weights, to make them small.  
  
  ---
  
  