Answers.Body: <p>ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, <a href="https://datascience.stackexchange.com/q/47336/10640">e.g. regression tasks where the output needs to be positive</a>). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.</p>
  
  <p>The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>, given that the sigmoid function does exactly this, i.e. it squashes its input to the range <span class="math-container">$[0, 1]$</span>, i.e. <span class="math-container">$\text{sigmoid}(x) = p \in [0, 1]$</span>. When do you need the output of the network to be a probability?  For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.</p>
  
  <p>You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.</p>
  
  <p>See this question <a href="https://stackoverflow.com/q/47034888/3924118">How to choose cross-entropy loss in TensorFlow?</a> on Stack Overflow for more info about different cross-entropy functions.</p>
  
  <p>By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the <em>mean squared error</em>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-11T02:45:02.240Z
Answers.LastActivityDate: 2020-03-11T02:50:11.023Z
Answers.LastEditDate: 2020-03-11T02:50:11.023Z
Answers.UserDisplayName: nbro
Answers.Body: <p>You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good.</p>
  
  <p>Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network. 
  One neuron network means it's a linear regression or logistic regression if it passes through a sigmoid activation.</p>
  
  <p>You have to look at NN like below.
  The output of a neural networks is the final layer output.
  There are two possible situations in the model. </p>
  
  <ol>
  <li><p>Classification model: For classification people generally use softmax and it has more than one output and the maximum value always below 1(It's a probability distribution).</p></li>
  <li><p>Regression model: A regression model has a continuous output like the output you mentioned in your problem statement. It has only one output channel. In the regression model, you can make the output as a linear combination of the previous layers (without defining the output range). You can use ReLU if you are sure your prediction is always positive.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-11T03:38:37.697Z
Answers.LastActivityDate: 2020-03-11T03:52:24.983Z
Answers.LastEditDate: 2020-03-11T03:52:24.983Z
Answers.UserDisplayName: Ta_Req
Body: <p>Consider the following simple neural network with only one neuron.</p>
  
  <ul>
  <li>The input is <span class="math-container">$x_1$</span> and <span class="math-container">$y_2$</span>, where <span class="math-container">$-250 &lt; x &lt; 250$</span> and <span class="math-container">$-250 &lt; y &lt; 250$</span></li>
  <li>The weights of the only neuron are <span class="math-container">$w_1$</span> and <span class="math-container">$w_1$</span></li>
  <li>The output of the neuron is given by <span class="math-container">$o = \sigma(x_1w_1 + x_2w_2 + b)$</span>, where <span class="math-container">$\sigma$</span> is the ReLU activation function and <span class="math-container">$b$</span> the bias.</li>
  <li>Thus the cost should be <span class="math-container">$(o - y)^2$</span>.</li>
  </ul>
  
  <p>When using the sigmoid activation function, the target for each point is usually <span class="math-container">$0$</span> or <span class="math-container">$1$</span>.</p>
  
  <p>But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-10T21:01:45.150Z
Comments.Text: Hi and welcome to AI SE! Why do you want to use the ReLU as the activation function of the last layer of your neural network?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-11T01:05:16.170Z
Comments.Text: Thanks for editing! I'm a beginner with neural network. Actually, I read that ReLU is better than sigmoid because it have a non-linear behavior. Should I use sigmoid at the output layer?
Comments.UserDisplayName: Alan Vinícius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-11T01:08:28.413Z
Comments.Text: ReLU and sigmoid have different properties, as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but I don't exclude people to use it for some reason, I just never saw it). A sigmoid is often used because the sigmoid squashes its inputs to the range $[0, 1]$, so, by doing this, the network will produce numbers that can be interpreted as probabilities, which can be useful in certain cases. In general, the targets don't necessarily need to be restricted to be 0 or 1. Do you want me to provide this information as a formal answer below?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-11T01:31:00.627Z
Comments.Text: Got it! You were very clear.
Comments.UserDisplayName: Alan Vinícius
Tags: <neural-networks><backpropagation><relu><mean-squared-error>
Title: How to determine the target value when using ReLU as activation function?
UserDisplayName: Alan Vinícius
ThreadText: # How to determine the target value when using ReLU as activation function?
  
  (Asked by: username_0 on 2020-03-10)
  
  Consider the following simple neural network with only one neuron.
  
  * The input is $x\_1$ and $y\_2$, where $-250 < x < 250$ and $-250 < y < 250$
  * The weights of the only neuron are $w\_1$ and $w\_1$
  * The output of the neuron is given by $o = \sigma(x\_1w\_1 + x\_2w\_2 + b)$, where $\sigma$ is the ReLU activation function and $b$ the bias.
  * Thus the cost should be $(o - y)^2$.
  
  When using the sigmoid activation function, the target for each point is usually $0$ or $1$.
  
  But I'm a little confused witch target to use when the activation function is the ReLU, given that it can output numbers greater than 1.
  
  **Tags:** neural-networks, backpropagation, relu, mean-squared-error
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! Why do you want to use the ReLU as the activation function of the last layer of your neural network?  
  > **username_0**: Thanks for editing! I'm a beginner with neural network. Actually, I read that ReLU is better than sigmoid because it have a non-linear behavior. Should I use sigmoid at the output layer?  
  > **username_1**: ReLU and sigmoid have different properties, as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but I don't exclude people to use it for some reason, I just never saw it). A sigmoid is often used because the sigmoid squashes its inputs to the range $[0, 1]$, so, by doing this, the network will produce numbers that can be interpreted as probabilities, which can be useful in certain cases. In general, the targets don't necessarily need to be restricted to be 0 or 1. Do you want me to provide this information as a formal answer below?  
  > **username_0**: Got it! You were very clear.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-03-11. Score: 1
  
  ReLU and sigmoid have different properties (i.e. range), as you already noticed. I've never seen the ReLU being used as the activation function of the output layer (but some people may use it for some reason, [e.g. regression tasks where the output needs to be positive](https://datascience.stackexchange.com/q/47336/10640)). ReLU is usually used as the activation function of a hidden layer. However, in your case, you don't have hidden layers.
  
  The sigmoid function is used as the activation function of the output layer when you need to interpret the output of the neural network as a probability, i.e. a number between $0$ and $1$, given that the sigmoid function does exactly this, i.e. it squashes its input to the range $[0, 1]$, i.e. $\text{sigmoid}(x) = p \in [0, 1]$. When do you need the output of the network to be a probability? For example, if you decide to use the cross-entropy loss function (which is equivalent to the negative log-likelihood), then the output of your network should be a probability. For example, if you need to solve a binary classification task, then the combination of a sigmoid as the activation function of the output layer and the binary cross-entropy as the loss function is probably what you need.
  
  You could also have a classification problem with more than 2 classes (multi-class classification problem). In that case, you probably need to use a softmax as the activation function of your network combined with a cross-entropy loss function.
  
  See this question [How to choose cross-entropy loss in TensorFlow?](https://stackoverflow.com/q/47034888/3924118) on Stack Overflow for more info about different cross-entropy functions.
  
  By the way, in general, the targets don't necessarily need to be restricted to be 0 or 1. For example, if you are solving a regression task, your target may just be any number. However, in that case, you may need another loss function (which is often the *mean squared error*).
  
  
  ---
  
  ### Answer by username_2 on 2020-03-11. Score: 1
  
  You are misunderstanding something. You are mixing up inner layers with the output layer. But the question was very good.
  
  Fist of all, with the only one layer and one neuron neural networks it does not exist. Only one layer can not bring nonlinearity in the network.
  One neuron network means it's a linear regression or logistic regression if it passes through a sigmoid activation.
  
  You have to look at NN like below.
  The output of a neural networks is the final layer output.
  There are two possible situations in the model.
  
  1. Classification model: For classification people generally use softmax and it has more than one output and the maximum value always below 1(It's a probability distribution).
  2. Regression model: A regression model has a continuous output like the output you mentioned in your problem statement. It has only one output channel. In the regression model, you can make the output as a linear combination of the previous layers (without defining the output range). You can use ReLU if you are sure your prediction is always positive.
  
  
  ---
  
  