Answers.Body: <p>Check out <a href="https://en.m.wikipedia.org/wiki/Dirac_delta_function" rel="nofollow noreferrer">Dirac delta function</a> on Wikipedia. It is just a special case of a probability density distribution for which the variance is zero. On the other hand, in general, all things are probabilistic once the Dirac delta function is included. Hope this helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-01T14:40:53.840Z
Answers.LastActivityDate: 2023-10-01T14:40:53.840Z
Answers.UserDisplayName: Quantum State
Answers.Body: <p>This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input <span class="math-container">$x$</span>, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.</p>
  <p>People often use this notation to indicate that, in the case of classification, there is a categorical distribution over the labels given the inputs, but this can be misleading, as the <a href="https://stats.stackexchange.com/a/437769/82135">softmax (the function often used to model this categorical distribution) only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network</a>, although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a <a href="https://en.wikipedia.org/wiki/Point_estimation" rel="nofollow noreferrer">point estimate</a> for each parameter of the network is learned and no uncertainty is properly modeled.</p>
  <p>Nevertheless, certain supervised learning problems have a formal probabilistic <strong>interpretation</strong>. For example, the minimization of the <em>mean squared error</em> function is equivalent to the maximization of a log probability, assuming your probability distribution is a Gaussian with a mean equal to the output of your model. In this probabilistic interpretation, you typically attempt to learn a probability (e.g. of the labels in the training dataset) and not a probability distribution. Watch <a href="https://www.youtube.com/watch?v=vEPQNwxd1Y4" rel="nofollow noreferrer">Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning)</a> by G. Hinton or read the paper <a href="http://papers.nips.cc/paper/613-bayesian-learning-via-stochastic-dynamics.pdf" rel="nofollow noreferrer">Bayesian Learning via Stochastic Dynamics</a> or <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.55.524&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method</a> by R. Neal for more details.</p>
  <p>Moreover, there are <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">Bayesian neural networks (BNNs)</a>, which actually maintain a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each parameter of the neural network, then you will learn the mean and variance of these Gaussians.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-25T17:47:56.397Z
Answers.Text: Thank you for the insight! Is it even possible to interpret a (non-Bayesian) trained NN as a distribution over inputs/outputs? The only way I can think of to do so is to represent the distribution as a sum over delta functions, e.g., if the trained model should have input/outputs pairs of (x1, y1), (x2, y2), ..., then the trained distribution would be $p(y|x) = y1*\delta_x1 + y2*\delta_x2 + ...$ . If one applies this interpretation, then is there a relationship between the BNN distribution and the non-BNN one, e.g., the non-BNN distribution results from an MLE over the BNN one?
Answers.UserDisplayName: Jammy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-25T18:29:18.553Z
Answers.Text: @Jammy I am not sure what you mean by a sum of delta functions to represent a probability distribution and how exactly does that yield a distribution. I've never heard anything similar, but I am not a statistician.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-25T18:39:13.317Z
Answers.Text: @Jammy You may want to try asking this question on Math SE or Cross Validated SE. I am also curious now to see a formal definition. What would be a distribution from which you always sample the same point?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T23:57:54.997Z
Answers.LastActivityDate: 2020-07-09T19:53:54.427Z
Answers.LastEditDate: 2020-07-09T19:53:54.427Z
Answers.UserDisplayName: nbro
Body: <p>In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,</p>
  <p><span class="math-container">$$\rho(\vec{y}|\vec{x},\vec{\theta})$$</span></p>
  <p>where <span class="math-container">$\vec{\theta}$</span> denotes a learned set of network parameters, <span class="math-container">$\vec{x}$</span> is an arbitrary input, and <span class="math-container">$\vec{y}$</span> is an arbitrary output. If we assume we have already learned <span class="math-container">$\vec{\theta}$</span>, then, in words, <span class="math-container">$\rho(\vec{y}|\vec{x},\vec{\theta})$</span> is the <em>probability</em> that the network will output an arbitrary <span class="math-container">$\vec{y}$</span> given an arbitrary input <span class="math-container">$\vec{x}$</span>.</p>
  <p>I am having a hard time reconciling how, <strong><em>after</em></strong> learning <span class="math-container">$\vec{\theta}$</span>, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input <span class="math-container">$\vec{x}$</span>, a trained network will always produce the same output.</p>
  <p>Any insight would be appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-25T00:29:40.843Z
Comments.Text: I agree, personally.  I tend to describe it as a *transformation* operation from X to Y via Theta.  Once the network is trained, the ultimate transformation is effectively deterministic unless you've created some type of random feedback layer somewhere.  The transformation returns a probability or  likelihood (as you certainly and clearly know)
Comments.UserDisplayName: David Hoelzer
Tags: <machine-learning><probability><supervised-learning><statistical-ai><bayesian-deep-learning>
Title: How can supervised learning be viewed as a conditional probability of the labels given the inputs?
UserDisplayName: Jammy
ThreadText: # How can supervised learning be viewed as a conditional probability of the labels given the inputs?
  
  (Asked by: username_0 on 2020-01-24)
  
  In the literature and textbooks, one often sees supervised learning expressed as a conditional probability, e.g.,
  
  $$\rho(\vec{y}|\vec{x},\vec{\theta})$$
  
  where $\vec{\theta}$ denotes a learned set of network parameters, $\vec{x}$ is an arbitrary input, and $\vec{y}$ is an arbitrary output. If we assume we have already learned $\vec{\theta}$, then, in words, $\rho(\vec{y}|\vec{x},\vec{\theta})$ is the *probability* that the network will output an arbitrary $\vec{y}$ given an arbitrary input $\vec{x}$.
  
  I am having a hard time reconciling how, ***after*** learning $\vec{\theta}$, there is still a probabilistic aspect to it. Post training, a network is, in general, a deterministic function, not a probability. For any specific input $\vec{x}$, a trained network will always produce the same output.
  
  Any insight would be appreciated.
  
  **Tags:** machine-learning, probability, supervised-learning, statistical-ai, bayesian-deep-learning
  > **Comments:**
  > **username_1**: I agree, personally.  I tend to describe it as a *transformation* operation from X to Y via Theta.  Once the network is trained, the ultimate transformation is effectively deterministic unless you've created some type of random feedback layer somewhere.  The transformation returns a probability or  likelihood (as you certainly and clearly know)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-01-24. Score: 5 | Accepted answer
  
  This formulation/interpretation can indeed be confusing (or even misleading), as the output of a neural network is usually deterministic (i.e. given the same input $x$, the output is always the same, so there is no sampling), and there isn't really a probability distribution that models any uncertainty associated with the parameters of the network or the input.
  
  People often use this notation to indicate that, in the case of classification, there is a categorical distribution over the labels given the inputs, but this can be misleading, as the [softmax (the function often used to model this categorical distribution) only squashes its inputs and doesn't really model any uncertainty associated with the input or the parameter of the neural network](https://stats.stackexchange.com/a/437769/82135), although the elements of the resulting vector add up to 1. In other words, in traditional deep learning, only a [point estimate](https://en.wikipedia.org/wiki/Point_estimation) for each parameter of the network is learned and no uncertainty is properly modeled.
  
  Nevertheless, certain supervised learning problems have a formal probabilistic **interpretation**. For example, the minimization of the *mean squared error* function is equivalent to the maximization of a log probability, assuming your probability distribution is a Gaussian with a mean equal to the output of your model. In this probabilistic interpretation, you typically attempt to learn a probability (e.g. of the labels in the training dataset) and not a probability distribution. Watch [Lecture 9.5 — The Bayesian interpretation of weight decay (Neural Networks for Machine Learning)](https://www.youtube.com/watch?v=vEPQNwxd1Y4) by G. Hinton or read the paper [Bayesian Learning via Stochastic Dynamics](http://papers.nips.cc/paper/613-bayesian-learning-via-stochastic-dynamics.pdf) or [Bayesian Training of Backpropagation Networks by the Hybrid Monte Carlo Method](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.55.524&rep=rep1&type=pdf) by R. Neal for more details.
  
  Moreover, there are [Bayesian neural networks (BNNs)](http://proceedings.mlr.press/v37/blundell15.pdf), which actually maintain a probability distribution over each parameter of the neural network that models the uncertainty associated with the value of this parameter. During the forward pass of this BNN, the specific parameters are actually sampled from the corresponding probability distributions. The actual learnable parameters of a BNN are the parameters of these distributions. For example, if you decide to have a Gaussian distribution over each parameter of the neural network, then you will learn the mean and variance of these Gaussians.
  
  > **Comments:**
  > **username_0**: Thank you for the insight! Is it even possible to interpret a (non-Bayesian) trained NN as a distribution over inputs/outputs? The only way I can think of to do so is to represent the distribution as a sum over delta functions, e.g., if the trained model should have input/outputs pairs of (x1, y1), (x2, y2), ..., then the trained distribution would be $p(y|x) = y1*\delta_x1 + y2*\delta_x2 + ...$ . If one applies this interpretation, then is there a relationship between the BNN distribution and the non-BNN one, e.g., the non-BNN distribution results from an MLE over the BNN one?  
  > **username_3**: @Jammy I am not sure what you mean by a sum of delta functions to represent a probability distribution and how exactly does that yield a distribution. I've never heard anything similar, but I am not a statistician.  
  > **username_3**: @Jammy You may want to try asking this question on Math SE or Cross Validated SE. I am also curious now to see a formal definition. What would be a distribution from which you always sample the same point?  
  
  ---
  
  ### Answer by username_2 on 2023-10-01. Score: 0
  
  Check out [Dirac delta function](https://en.m.wikipedia.org/wiki/Dirac_delta_function) on Wikipedia. It is just a special case of a probability density distribution for which the variance is zero. On the other hand, in general, all things are probabilistic once the Dirac delta function is included. Hope this helps.
  
  
  ---
  
  