Answers.Body: <p>To sort the terms a bit, back propagation (repeatedly applying the chain rule of differentiation) is an implementation of stochastic gradient descent, which is an implementation of optimization. It is not necessarily the square of the error that is reduced by optimization - it is up to us to choose the loss function.</p>
  
  <p>The loss function should ideally mirror the utility function (what you are truly want to optimize), but it needs a useful derivative. For example, the <a href="https://en.wikipedia.org/wiki/Loss_function#0-1_loss_function" rel="nofollow noreferrer">0-1 loss</a> hasn't one.</p>
  
  <p>I think that what is called <em>linear confidence guess</em> in the question is the conditional probability of the event, given the input. Then the <a href="https://gombru.github.io/2018/05/23/cross_entropy_loss/" rel="nofollow noreferrer">binary cross entropy</a> would be a valid choice of a loss function.</p>
  
  <p>From <a href="http://www.deeplearningbook.org/contents/optimization.html" rel="nofollow noreferrer">deeplearningbook.org</a></p>
  
  <blockquote>
    <p>The negative log-likelihood allows the model to estimate the conditional probability of the classes, given the input, and if the model can do that well, then it can pick the classes that yield the least classiﬁcation error in expectation.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T11:37:45.407Z
Answers.Text: This is a good answer. You have helped me understand that I can get back propagation to optimise differently depending on my choice of loss function. Would you be able to tell me which loss function the code I have included in the question is using and how I could change it to make it use binary cross entropy? If that is asking too much or off topic for this site then I think I can eventually work it out with some research based on the links you have provided.
Answers.UserDisplayName: F Chopin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T12:47:23.220Z
Answers.Text: That one doesn't look familiar to me, sorry. May be you can find it among those listed in [this ml-cheatsheet](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#mae-l1)
Answers.UserDisplayName: Xpector
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T10:47:34.443Z
Answers.LastActivityDate: 2019-02-21T10:47:34.443Z
Answers.UserDisplayName: Xpector
Body: <p>I have a dataset with hundreds of thousands of training examples. There are 27 input variables and one output variable which is always a 0 or a 1, based on whether an event happened or not.</p>
  
  <p>My network therefore has 27 inputs and 1 output. I want the network's output to be a confidence guess of how likely the event is to happen, for example if the output is 0.23 then that represents that the network thinks the event has a 23% chance of happening.</p>
  
  <p>I am using back propagation to train the neural network. It does appear to work well and the network outputs a higher number when the event is more likely and a lower number when the event is less likely.</p>
  
  <p>Would it be a valid concern that my training data only has 0 or 1 values as outputs, when this is not truly what I want the network to output?</p>
  
  <p>My concern comes from the fact that back propagation attempts to reduce the square of the error between the network's output, and the value of the output in the training data, which is always a 0 or a 1. Because it is the <strong>square</strong> of the error it is trying to reduce, I'm concerned that it's probability output may not be a linear mapping to the true probability of the event happening based on the 27 inputs it is seeing.</p>
  
  <p>Is this a valid concern? And are there any techniques I can use to get a neural network to output a linear confidence guess between 0 and 1 when my test data only has outputs of 0 or 1?</p>
  
  <p>I am using the sigmoid activation function for all of my neurons, would there be a better choice of activation function for this problem?</p>
  
  <p>Edit: Thanks to Xpector's answer, I now understand that not all back propagation aims to reduce the square of the error, it depends on the loss function used. I am including a part of the back propagation code I have used here which calculates the error:</p>
  
  <pre><code>var neuronOutput = layerOutputs[i];
  var error = (neuronOutput - desiredOutput[i]);
  errors[i] = error * Maths.SigmoidDerivative(neuronOutput);
  </code></pre>
  
  <p>This is from an open source RProp implementation. I am not sure what loss function is being used here.</p>
  
Tags: <neural-networks>
Title: Training a neural network to output the conditional probability of an event when the training data output is only binary
UserDisplayName: F Chopin
ThreadText: # Training a neural network to output the conditional probability of an event when the training data output is only binary
  
  (Asked by: username_0 on 2019-02-20)
  
  I have a dataset with hundreds of thousands of training examples. There are 27 input variables and one output variable which is always a 0 or a 1, based on whether an event happened or not.
  
  My network therefore has 27 inputs and 1 output. I want the network's output to be a confidence guess of how likely the event is to happen, for example if the output is 0.23 then that represents that the network thinks the event has a 23% chance of happening.
  
  I am using back propagation to train the neural network. It does appear to work well and the network outputs a higher number when the event is more likely and a lower number when the event is less likely.
  
  Would it be a valid concern that my training data only has 0 or 1 values as outputs, when this is not truly what I want the network to output?
  
  My concern comes from the fact that back propagation attempts to reduce the square of the error between the network's output, and the value of the output in the training data, which is always a 0 or a 1. Because it is the **square** of the error it is trying to reduce, I'm concerned that it's probability output may not be a linear mapping to the true probability of the event happening based on the 27 inputs it is seeing.
  
  Is this a valid concern? And are there any techniques I can use to get a neural network to output a linear confidence guess between 0 and 1 when my test data only has outputs of 0 or 1?
  
  I am using the sigmoid activation function for all of my neurons, would there be a better choice of activation function for this problem?
  
  Edit: Thanks to username_1's answer, I now understand that not all back propagation aims to reduce the square of the error, it depends on the loss function used. I am including a part of the back propagation code I have used here which calculates the error:
  
  ```
  var neuronOutput = layerOutputs[i];
  var error = (neuronOutput - desiredOutput[i]);
  errors[i] = error * Maths.SigmoidDerivative(neuronOutput);
  
  ```
  
  This is from an open source RProp implementation. I am not sure what loss function is being used here.
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-21. Score: 3
  
  To sort the terms a bit, back propagation (repeatedly applying the chain rule of differentiation) is an implementation of stochastic gradient descent, which is an implementation of optimization. It is not necessarily the square of the error that is reduced by optimization - it is up to us to choose the loss function.
  
  The loss function should ideally mirror the utility function (what you are truly want to optimize), but it needs a useful derivative. For example, the [0-1 loss](https://en.wikipedia.org/wiki/Loss_function#0-1_loss_function) hasn't one.
  
  I think that what is called *linear confidence guess* in the question is the conditional probability of the event, given the input. Then the [binary cross entropy](https://gombru.github.io/2018/05/23/cross_entropy_loss/) would be a valid choice of a loss function.
  
  From [deeplearningbook.org](http://www.deeplearningbook.org/contents/optimization.html)
  
  > The negative log-likelihood allows the model to estimate the conditional probability of the classes, given the input, and if the model can do that well, then it can pick the classes that yield the least classiﬁcation error in expectation.
  
  > **Comments:**
  > **username_0**: This is a good answer. You have helped me understand that I can get back propagation to optimise differently depending on my choice of loss function. Would you be able to tell me which loss function the code I have included in the question is using and how I could change it to make it use binary cross entropy? If that is asking too much or off topic for this site then I think I can eventually work it out with some research based on the links you have provided.  
  > **username_1**: That one doesn't look familiar to me, sorry. May be you can find it among those listed in [this ml-cheatsheet](https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#mae-l1)  
  
  ---
  
  