Answers.Body: <p>Consider the case of binary classification, i.e. you want to classify each input <span class="math-container">$x$</span> into one of two classes: <span class="math-container">$y_1$</span> or <span class="math-container">$y_2$</span>. For example, in the context of object classification, <span class="math-container">$y_1$</span> could be &quot;cat&quot; and <span class="math-container">$y_2$</span> could be &quot;dog&quot;, and <span class="math-container">$x$</span> is an image that contains one main object.</p>
  <p>In certain cases, <span class="math-container">$x$</span> cannot be easily classified. For example, in object classification, if <span class="math-container">$x$</span> is a blurred image where there's some uncertainty about the object in the image, what should the output of the neural network be? Should it be <span class="math-container">$y_1$</span>, <span class="math-container">$y_2$</span>, or maybe it should be an <em>uncertainty value</em> (i.e. a probability) that lies between <span class="math-container">$y_1$</span> and <span class="math-container">$y_2$</span>? The last option is probably the most reasonable, but also the most general one (in the sense that it can also be used in the case there's little or no uncertainty about what the object is).</p>
  <p>That's the reason why we can model or formulate this (or other) supervised learning problem(s) as the estimation of a probability value (or probability distribution).</p>
  <p>To be more concrete, you can formulate this binary classification problem as the estimation of the following probability</p>
  <p><span class="math-container">\begin{align}
  P(y_1 \mid x, \theta_i) \in [0, 1] \label{1}\tag{1}
  \end{align}</span></p>
  <p>where <span class="math-container">$y_1$</span> is the first class (or label), <span class="math-container">$(x, y) \in \mathcal{D}$</span> is a labeled training example, where <span class="math-container">$y$</span> is the ground-truth label for the input <span class="math-container">$x$</span>, <span class="math-container">$\theta_i$</span> are the parameters of the neural network at iteration <span class="math-container">$i$</span>, so, intuitively, <span class="math-container">$P(y_1 \mid x, \theta_i) $</span> is a probability that represents how likely the neural network thinks that <span class="math-container">$x$</span> belongs to the class <span class="math-container">$y_1$</span> given the current estimate of the parameters. The probability that <span class="math-container">$x$</span> belongs to the other class is just <span class="math-container">$1 - P(y_1 \mid x, \theta_i) = P(y_2 \mid x, \theta_i)$</span>.  In this specific case, I have added a subscript to <span class="math-container">$\theta$</span> to indicate that this probability depends on the <span class="math-container">$i$</span>th estimate of the parameters of the neural network.</p>
  <p>Once you have <span class="math-container">$P(y_1 \mid x, \theta_i)$</span>, if you want to perform classification, you will actually need to choose a threshold value <span class="math-container">$t$</span>, such that, if <span class="math-container">$P(y_1 \mid x, \theta_i) &gt; t$</span>, then <span class="math-container">$x$</span> is classified as <span class="math-container">$y_1$</span>, else it is classified as <span class="math-container">$y_2$</span>. This threshold value <span class="math-container">$t$</span> can be <span class="math-container">$0.5$</span>, but it can also not be.</p>
  <p>Note that, in the case above, <span class="math-container">$P(y_1 \mid x, \theta_i)$</span> is a number and not a probability distribution. However, in certain cases, you can also formulate your supervised learning problem so that the output is a probability distribution (rather just a probability). There are also other problems where you don't estimate a conditional probability but maybe a joint probability, but the case above is probably the simplest one that should give you the intuition behind the idea of formulating machine learning problems as the estimation of probabilities or probability distributions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-24T15:06:54.853Z
Answers.LastActivityDate: 2020-06-24T17:03:01.380Z
Answers.LastEditDate: 2020-06-24T17:03:01.380Z
Answers.UserDisplayName: nbro
Body: <p>Neural network training problems are oftentimes formulated as probability estimation problems (such as autoregressive models).</p>
  <p>How does one intuitively understand this idea?</p>
  
Tags: <neural-networks><comparison><training><probability-distribution>
Title: Intuitively, why can the training of a neural network be formulated as a probability estimation problem?
UserDisplayName: C Lu
ThreadText: # Intuitively, why can the training of a neural network be formulated as a probability estimation problem?
  
  (Asked by: username_0 on 2020-06-24)
  
  Neural network training problems are oftentimes formulated as probability estimation problems (such as autoregressive models).
  
  How does one intuitively understand this idea?
  
  **Tags:** neural-networks, comparison, training, probability-distribution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-24. Score: 1 | Accepted answer
  
  Consider the case of binary classification, i.e. you want to classify each input $x$ into one of two classes: $y\_1$ or $y\_2$. For example, in the context of object classification, $y\_1$ could be "cat" and $y\_2$ could be "dog", and $x$ is an image that contains one main object.
  
  In certain cases, $x$ cannot be easily classified. For example, in object classification, if $x$ is a blurred image where there's some uncertainty about the object in the image, what should the output of the neural network be? Should it be $y\_1$, $y\_2$, or maybe it should be an *uncertainty value* (i.e. a probability) that lies between $y\_1$ and $y\_2$? The last option is probably the most reasonable, but also the most general one (in the sense that it can also be used in the case there's little or no uncertainty about what the object is).
  
  That's the reason why we can model or formulate this (or other) supervised learning problem(s) as the estimation of a probability value (or probability distribution).
  
  To be more concrete, you can formulate this binary classification problem as the estimation of the following probability
  
  \begin{align}
  P(y\_1 \mid x, \theta\_i) \in [0, 1] \label{1}\tag{1}
  \end{align}
  
  where $y\_1$ is the first class (or label), $(x, y) \in \mathcal{D}$ is a labeled training example, where $y$ is the ground-truth label for the input $x$, $\theta\_i$ are the parameters of the neural network at iteration $i$, so, intuitively, $P(y\_1 \mid x, \theta\_i) $ is a probability that represents how likely the neural network thinks that $x$ belongs to the class $y\_1$ given the current estimate of the parameters. The probability that $x$ belongs to the other class is just $1 - P(y\_1 \mid x, \theta\_i) = P(y\_2 \mid x, \theta\_i)$. In this specific case, I have added a subscript to $\theta$ to indicate that this probability depends on the $i$th estimate of the parameters of the neural network.
  
  Once you have $P(y\_1 \mid x, \theta\_i)$, if you want to perform classification, you will actually need to choose a threshold value $t$, such that, if $P(y\_1 \mid x, \theta\_i) > t$, then $x$ is classified as $y\_1$, else it is classified as $y\_2$. This threshold value $t$ can be $0.5$, but it can also not be.
  
  Note that, in the case above, $P(y\_1 \mid x, \theta\_i)$ is a number and not a probability distribution. However, in certain cases, you can also formulate your supervised learning problem so that the output is a probability distribution (rather just a probability). There are also other problems where you don't estimate a conditional probability but maybe a joint probability, but the case above is probably the simplest one that should give you the intuition behind the idea of formulating machine learning problems as the estimation of probabilities or probability distributions.
  
  
  ---
  
  