Answers.Body: <p>There are multiple potential reasons for having stochastic predictions (instead of categorical/binary).</p>
  <p>First, it often simplifies training and improves the training outcome when training a classifier on producing probabilities per class. For example, it allows for the usage of many nice loss functions like the Mean Squared Error (MSE), which is compatible with the famous back-prop algorithm. Moreover, improving classification based on the loss computed from stochastic probabilities per class allows for improving the classifier even further when the counting loss computed on predicted class labels has reached 0 already. So, if you only have binary decisions and your classifier gets all classes correct, training can stop immediately since the loss becomes 0. However, when predicting probabilities per class and iteratively driving the probability of the correct class towards 1, training can continue for much longer, even after all classes have been classified correctly already. So, the classifier trained on making probabilistic predictions can in the end perform much better since training can progress for much longer, allowing for increasingly pronounced discrimination between classes with every update. A nice introduction is given in this Stanford <a href="https://www.youtube.com/watch?v=h7iBpEHGVNc&amp;list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;index=3" rel="nofollow noreferrer">lecture recording</a>.</p>
  <p>Second, it is often nice for a user to know how much certainty versus uncertainty there is in a classification. If one class has 80% probability, this is a pretty clear decision in favor of the 80%-probability-class. However, if you only get categorical class labels returned by your classifier, you cannot possibly know whether the evidence in favor of the winning class was only marginally higher than that of the other class (as measured by the classifier) or whether there was a clear difference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T22:34:52.340Z
Answers.LastActivityDate: 2020-06-24T23:53:11.147Z
Answers.LastEditDate: 2020-06-24T23:53:11.147Z
Answers.UserDisplayName: Daniel B.
Body: <p>What is the advantage of having a stochastic/probabilistic classification procedure?</p>
  <p>The classifiers I have encountered so far are as follows. Suppose we have two outcomes <span class="math-container">$A = \{0,1\}$</span>. Given a feature vector <span class="math-container">$x$</span>, we have calculated a probability for each outcome and return the outcome <span class="math-container">$a \in A$</span> for which the probability is highest.</p>
  <p>Now, I encountered a classification procedure as follows: first, map each <span class="math-container">$x$</span> to a probability distribution on <span class="math-container">$A$</span> by a mapping <span class="math-container">$H$</span>. To classify <span class="math-container">$x$</span>, choose an outcome <span class="math-container">$a$</span> according to the distribution <span class="math-container">$H(x)$</span>.</p>
  <p>Why not use the deterministic classification? Suppose <span class="math-container">$H(x)$</span> is 1 with probability <span class="math-container">$0.75$</span>. Then, the obvious choice for an outcome would be <span class="math-container">$1$</span> and not <span class="math-container">$0$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-24T23:52:32.870Z
Comments.Text: Please, next time, edit your post to clarify it.
Comments.UserDisplayName: nbro
Tags: <machine-learning><classification>
Title: What is the advantage of having a stochastic classification procedure?
UserDisplayName: MMM
ThreadText: # What is the advantage of having a stochastic classification procedure?
  
  (Asked by: username_0 on 2020-06-24)
  
  What is the advantage of having a stochastic/probabilistic classification procedure?
  
  The classifiers I have encountered so far are as follows. Suppose we have two outcomes $A = \{0,1\}$. Given a feature vector $x$, we have calculated a probability for each outcome and return the outcome $a \in A$ for which the probability is highest.
  
  Now, I encountered a classification procedure as follows: first, map each $x$ to a probability distribution on $A$ by a mapping $H$. To classify $x$, choose an outcome $a$ according to the distribution $H(x)$.
  
  Why not use the deterministic classification? Suppose $H(x)$ is 1 with probability $0.75$. Then, the obvious choice for an outcome would be $1$ and not $0$.
  
  **Tags:** machine-learning, classification
  > **Comments:**
  > **username_1**: Please, next time, edit your post to clarify it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-24. Score: 1 | Accepted answer
  
  There are multiple potential reasons for having stochastic predictions (instead of categorical/binary).
  
  First, it often simplifies training and improves the training outcome when training a classifier on producing probabilities per class. For example, it allows for the usage of many nice loss functions like the Mean Squared Error (MSE), which is compatible with the famous back-prop algorithm. Moreover, improving classification based on the loss computed from stochastic probabilities per class allows for improving the classifier even further when the counting loss computed on predicted class labels has reached 0 already. So, if you only have binary decisions and your classifier gets all classes correct, training can stop immediately since the loss becomes 0. However, when predicting probabilities per class and iteratively driving the probability of the correct class towards 1, training can continue for much longer, even after all classes have been classified correctly already. So, the classifier trained on making probabilistic predictions can in the end perform much better since training can progress for much longer, allowing for increasingly pronounced discrimination between classes with every update. A nice introduction is given in this Stanford [lecture recording](https://www.youtube.com/watch?v=h7iBpEHGVNc&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=3).
  
  Second, it is often nice for a user to know how much certainty versus uncertainty there is in a classification. If one class has 80% probability, this is a pretty clear decision in favor of the 80%-probability-class. However, if you only get categorical class labels returned by your classifier, you cannot possibly know whether the evidence in favor of the winning class was only marginally higher than that of the other class (as measured by the classifier) or whether there was a clear difference.
  
  
  ---
  
  