Answers.Body: <p>A probability distribution in ML is the same as a probability distribution elsewhere.</p>
  
  <p>A probability distribution (or probability function, or probability mass function, or probability density function) is any function that accepts as input elements of some specific set <span class="math-container">$x \in X$</span>, and produces as output, real-valued numbers between 0 and 1 (inclusive), such that <span class="math-container">$\int_{x \in X} p(x) = 1$</span> or for discrete sets, <span class="math-container">$\sum_{x \in X} p(x) = 1$</span>.</p>
  
  <p>These distributions can also be more complex. For example, a conditional probability distribution <span class="math-container">$P(Y|X)$</span> or a joint probability distribution <span class="math-container">$P(X,Y)$</span> accept more than one input, but again, are constrained to producing an output in the range 0 to 1, and to ensuring that the summation of the output over all possible inputs is exactly 1.</p>
  
  <p>When these conditions are met, the functions output can be interpreted as a belief about the percentage of times the input event will occur, out of all events, or as a degree of believe in the input event having occurred vs. other events (i.e. it can be interpreted as a probability).  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-28T15:17:21.000Z
Answers.LastActivityDate: 2019-11-28T15:17:21.000Z
Answers.UserDisplayName: John Doucette
Answers.Body: <h3>Random variables</h3>
  <p>You do not necessarily need to understand the concept of a <a href="https://en.wikipedia.org/wiki/Random_variable" rel="nofollow noreferrer">random variable (r.v.)</a> to understand the concept of a probability distribution, but the concept of a random variable is strictly connected to the concept of a probability distribution (given that each random variable has an associated probability distribution), so, before proceeding, you should get familiar with the concept of an r.v., which is a (measurable) <strong>function</strong> from the sample space (the set of possible outcomes of an experiment) to a measurable space (you can ignore the definition of a measurable space and assume that the codomain of the random variable is a finite set of numbers).</p>
  <h3>Probability measure, cdf, pdf and pmf</h3>
  <p>The expression &quot;probability distribution&quot; can be ambiguous because it can be used to refer to different (even though related) mathematical concepts, such as <a href="https://en.wikipedia.org/wiki/Probability_measure" rel="nofollow noreferrer">probability measure</a>, <a href="https://en.wikipedia.org/wiki/Cumulative_distribution_function" rel="nofollow noreferrer">cumulative distribution function (c.d.f.)</a>, <a href="https://en.wikipedia.org/wiki/Probability_density_function" rel="nofollow noreferrer">probability density function (p.d.f.)</a>, <a href="https://en.wikipedia.org/wiki/Probability_mass_function" rel="nofollow noreferrer">probability mass function (p.m.f.)</a>. If a person uses the expression &quot;probability distribution&quot;, he (or she) intentionally (or not) refers to one or more of these mathematical concepts, depending on the context. However, <strong>a probability distribution is almost always a synonym for <a href="https://en.wikipedia.org/wiki/Probability_measure" rel="nofollow noreferrer">probability measure</a> or <a href="https://en.wikipedia.org/wiki/Probability_density_function" rel="nofollow noreferrer">c.d.f.</a></strong>.</p>
  <p>For example, if I say &quot;Consider the Gaussian probability distribution&quot;, in that case, I could be referring to either the c.d.f. or the p.d.f. (or both) of the <a href="https://en.wikipedia.org/wiki/Normal_distribution" rel="nofollow noreferrer">Gaussian distribution</a>. Why couldn't I be referring to the p.m.f. of the Gaussian distribution? Because the Gaussian distribution is a continuous distribution, so it is a distribution associated with a continuous random variable, that is, a random variable that can take on continuous values (e.g. real numbers), so a Gaussian distribution does not have an associated p.m.f. or, in other words, no p.m.f. is defined for the Gaussian distribution. Why don't I simply say &quot;Consider the p.d.f. of the Gaussian distribution.&quot; or &quot;Consider a Gaussian p.d.f.&quot;? Because it is unnecessarily restrictive, given that, if I say &quot;Consider the Gaussian distribution&quot; I am implicitly also considering a p.d.f. and c.d.f. of the Gaussian distribution.</p>
  <p>Similarly, in the case of a discrete distribution, such as the Bernoulli distribution, only the c.d.f. and p.m.f. are defined, so the Bernoulli distribution does not have an associated p.d.f.</p>
  <p>However, it is important to recall that both continuous and discrete distributions have an associated c.d.f., so the expression &quot;probability distribution&quot; almost always (implicitly) refers to a c.d.f., which is defined based on a probability measure (as stated above).</p>
  <h3>Notation</h3>
  <p>In the same vein, the notation <span class="math-container">$p(x)$</span> can be as ambiguous as the expression &quot;probability distribution&quot;, given that it can refer to different (but again related) concepts. However, <span class="math-container">$p(x)$</span> usually refers to a <a href="https://en.wikipedia.org/wiki/Probability_measure" rel="nofollow noreferrer">probability measure</a> (so it refers to a probability distribution, given that a probability distribution is almost always a synonym for probability measure). In this case, assuming for simplicity that the r.v. is discrete, <strong><span class="math-container">$p(x)$</span> is a shorthand for <span class="math-container">$p(X=x)$</span>, which is also written as <span class="math-container">$\mathbb{P}(X=x)$</span></strong> or <span class="math-container">$\operatorname{Pr}(X=x)$</span>, where <span class="math-container">$X$</span> is a r.v., <span class="math-container">$x$</span> a <a href="https://en.wikipedia.org/wiki/Realization_(probability)" rel="nofollow noreferrer">realization</a> of <span class="math-container">$X$</span> (that is, a value that the r.v. <span class="math-container">$X$</span> can take) and <span class="math-container">$X=x$</span> represents an <a href="https://en.wikipedia.org/wiki/Event_(probability_theory)" rel="nofollow noreferrer">event</a>. Given that an r.v. is a function, the notation <span class="math-container">$X=x$</span> may look a bit weird.</p>
  <p>In the case of a discrete r.v., <span class="math-container">$p(x)$</span> can also refer to a p.m.f. and it can be defined as <span class="math-container">$p_X(x) = \mathbb{P}(X=x)$</span> (I added the subscript <span class="math-container">$X$</span> to <span class="math-container">$p$</span> to emphasize that this is the p.m.f. of the discrete r.v. <span class="math-container">$X$</span>). In the case of a continuous r.v., the p.d.f. is often denoted as <span class="math-container">$f$</span>. In the case of both discrete and continuous r.v.s, the c.d.f is usually denoted with <span class="math-container">$F$</span> and it is defined as <span class="math-container">$F_X(x) = \mathbb{P}(X \leq x)$</span>, where <span class="math-container">$\mathbb{P}$</span> is again a probability measure (or probability distribution). The p.d.f. of a continuous r.v. is then defined as the derivative of <span class="math-container">$F$</span>. At this point, it should be clear why a probability distribution can refer to different but related concepts, but, in any case, it always refers to a <a href="https://en.wikipedia.org/wiki/Probability_measure" rel="nofollow noreferrer">probability measure</a>.</p>
  <h3>Empirical distributions</h3>
  <p>There are also <a href="https://en.wikipedia.org/wiki/Empirical_distribution_function" rel="nofollow noreferrer">empirical distributions</a>, which are distributions of the data that you have collected. For example, if you toss a coin 10 times, you will collect the results (&quot;heads&quot; or &quot;tails&quot;). You can count the number of times the coin landed on heads and tails, then you plot these numbers as a histogram, which essentially represents your empirical distribution, where the adjective &quot;empirical&quot; usually refers to the fact that there is an experiment involved.</p>
  <h3>Multivariate r.v.s and distributions</h3>
  <p>To complicate things even more, there are also <a href="https://en.wikipedia.org/wiki/Multivariate_normal_distribution" rel="nofollow noreferrer">multivariate random variables and probability distributions</a>. However, all the concepts above more or less are also applicable in this case.</p>
  <h3>Parametrized distributions</h3>
  <p>A parametrized probability distribution, often denoted by <span class="math-container">$p_{\theta}$</span>, is
  a <a href="https://en.wikipedia.org/wiki/Parametric_model" rel="nofollow noreferrer">family of probability distributions</a> (defined by the parameters <span class="math-container">$\theta$</span>), rather than a single probability distribution. For example, <span class="math-container">$\mathcal{N}(0, 1)$</span> refers to a <em>single</em> Gaussian distribution with zero mean and unit variance. However, <span class="math-container">$\mathcal{N}(\mu, \sigma)$</span>, where <span class="math-container">$\theta=(\mu,  \sigma)$</span> is a variable, is a family (or collection) of distributions.</p>
  <h3>Conclusion</h3>
  <p>To conclude, it is completely understandable that you are confused, given that the terminology and notation are used inconsistently, and there are several involved concepts, which I have not extensively covered in this answer (for example, I have not mentioned the concept of a probability space). If you get familiar with the concepts of probability measures, random variables, p.m.f., p.d.f., c.d.f., etc., and how they are related, then you will start to get a better feeling of the whole picture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-28T15:50:11.347Z
Answers.LastActivityDate: 2021-01-21T15:34:16.217Z
Answers.LastEditDate: 2021-01-21T15:34:16.217Z
Answers.UserDisplayName: nbro
Body: <p>If we were learning or working in the machine learning field, then we frequently come across the term &quot;probability distribution&quot;. I know what probability, conditional probability, and probability distribution/density in math mean, but what is its meaning in machine learning?</p>
  <p>Take this example where <span class="math-container">$x$</span> is an element of <span class="math-container">$D$</span>, which is a dataset,</p>
  <p><span class="math-container">$$x \in D$$</span></p>
  <p>Let's say our dataset (<span class="math-container">$D$</span>) is MNIST with about 70,000 images, so then <span class="math-container">$x$</span> becomes any image of those 70,000 images.</p>
  <p>In many papers and web articles, these terms are often denoted as <em>probability distributions</em></p>
  <p><span class="math-container">$$p(x)$$</span></p>
  <p>or</p>
  <p><span class="math-container">$$p\left(z \mid x \right)$$</span></p>
  <ul>
  <li>What does <span class="math-container">$p(\cdot)$</span> even mean, and what kind of output does <span class="math-container">$p(\cdot)$</span> give?</li>
  <li>Is the output of <span class="math-container">$p(\cdot)$</span> a scalar, vector, or matrix?</li>
  <li>If the output is vector or matrix, then will the sum of all elements of this vector/matrix always be <span class="math-container">$1$</span>?</li>
  </ul>
  <p>This is my understanding,</p>
  <p><span class="math-container">$p(\cdot)$</span> is a function which maps the real distribution of the whole dataset <span class="math-container">$D$</span>. Then  <span class="math-container">$p(x)$</span> gives a scalar probability value given <span class="math-container">$x$</span>, which is calculated from real distribution <span class="math-container">$p(\cdot)$</span>. Similar to <span class="math-container">$p(H)=0.5$</span> in a coin toss experiment <span class="math-container">$D={\{H,T}\}$</span>.</p>
  <p><span class="math-container">$p\left(z \mid x \right)$</span> is another function that maps the real distribution of the whole dataset to a vector <span class="math-container">$z$</span> given an input <span class="math-container">$x$</span> and the <span class="math-container">$z$</span> vector is a probability distribution that sums to <span class="math-container">$1$</span>.</p>
  <p>Are my assumptions correct?</p>
  <p>An example would be a <a href="https://lilianweng.github.io/lil-log/2018/08/12/from-autoencoder-to-beta-vae.html#vae-variational-autoencoder" rel="nofollow noreferrer">VAE's data generation process</a>, which is represented in this equation</p>
  <p><span class="math-container">$$p_\theta(\mathbf{x}^{(i)}) = \int p_\theta(\mathbf{x}^{(i)}\vert\mathbf{z}) p_\theta(\mathbf{z}) d\mathbf{z}$$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-28T10:03:34.870Z
Comments.Text: *In many papers and web articles we notice these terms as probabiltiy distributions* Please link the paper or web page so that it makes easier to get hold of the question and write an answer.
Comments.UserDisplayName: naive
Tags: <machine-learning><terminology><definitions><probability-distribution><notation>
Title: What is a probability distribution in machine learning?
UserDisplayName: Eka
ThreadText: # What is a probability distribution in machine learning?
  
  (Asked by: username_0 on 2019-11-28)
  
  If we were learning or working in the machine learning field, then we frequently come across the term "probability distribution". I know what probability, conditional probability, and probability distribution/density in math mean, but what is its meaning in machine learning?
  
  Take this example where $x$ is an element of $D$, which is a dataset,
  
  $$x \in D$$
  
  Let's say our dataset ($D$) is MNIST with about 70,000 images, so then $x$ becomes any image of those 70,000 images.
  
  In many papers and web articles, these terms are often denoted as *probability distributions*
  
  $$p(x)$$
  
  or
  
  $$p\left(z \mid x \right)$$
  
  * What does $p(\cdot)$ even mean, and what kind of output does $p(\cdot)$ give?
  * Is the output of $p(\cdot)$ a scalar, vector, or matrix?
  * If the output is vector or matrix, then will the sum of all elements of this vector/matrix always be $1$?
  
  This is my understanding,
  
  $p(\cdot)$ is a function which maps the real distribution of the whole dataset $D$. Then $p(x)$ gives a scalar probability value given $x$, which is calculated from real distribution $p(\cdot)$. Similar to $p(H)=0.5$ in a coin toss experiment $D={\{H,T}\}$.
  
  $p\left(z \mid x \right)$ is another function that maps the real distribution of the whole dataset to a vector $z$ given an input $x$ and the $z$ vector is a probability distribution that sums to $1$.
  
  Are my assumptions correct?
  
  An example would be a [VAE's data generation process](https://lilianweng.github.io/lil-log/2018/08/12/from-autoencoder-to-beta-vae.html#vae-variational-autoencoder), which is represented in this equation
  
  $$p\_\theta(\mathbf{x}^{(i)}) = \int p\_\theta(\mathbf{x}^{(i)}\vert\mathbf{z}) p\_\theta(\mathbf{z}) d\mathbf{z}$$
  
  **Tags:** machine-learning, terminology, definitions, probability-distribution, notation
  > **Comments:**
  > **username_1**: *In many papers and web articles we notice these terms as probabiltiy distributions* Please link the paper or web page so that it makes easier to get hold of the question and write an answer.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-11-28. Score: 3 | Accepted answer
  
  ### Random variables
  
  You do not necessarily need to understand the concept of a [random variable (r.v.)](https://en.wikipedia.org/wiki/Random_variable) to understand the concept of a probability distribution, but the concept of a random variable is strictly connected to the concept of a probability distribution (given that each random variable has an associated probability distribution), so, before proceeding, you should get familiar with the concept of an r.v., which is a (measurable) **function** from the sample space (the set of possible outcomes of an experiment) to a measurable space (you can ignore the definition of a measurable space and assume that the codomain of the random variable is a finite set of numbers).
  
  ### Probability measure, cdf, pdf and pmf
  
  The expression "probability distribution" can be ambiguous because it can be used to refer to different (even though related) mathematical concepts, such as [probability measure](https://en.wikipedia.org/wiki/Probability_measure), [cumulative distribution function (c.d.f.)](https://en.wikipedia.org/wiki/Cumulative_distribution_function), [probability density function (p.d.f.)](https://en.wikipedia.org/wiki/Probability_density_function), [probability mass function (p.m.f.)](https://en.wikipedia.org/wiki/Probability_mass_function). If a person uses the expression "probability distribution", he (or she) intentionally (or not) refers to one or more of these mathematical concepts, depending on the context. However, **a probability distribution is almost always a synonym for [probability measure](https://en.wikipedia.org/wiki/Probability_measure) or [c.d.f.](https://en.wikipedia.org/wiki/Probability_density_function)**.
  
  For example, if I say "Consider the Gaussian probability distribution", in that case, I could be referring to either the c.d.f. or the p.d.f. (or both) of the [Gaussian distribution](https://en.wikipedia.org/wiki/Normal_distribution). Why couldn't I be referring to the p.m.f. of the Gaussian distribution? Because the Gaussian distribution is a continuous distribution, so it is a distribution associated with a continuous random variable, that is, a random variable that can take on continuous values (e.g. real numbers), so a Gaussian distribution does not have an associated p.m.f. or, in other words, no p.m.f. is defined for the Gaussian distribution. Why don't I simply say "Consider the p.d.f. of the Gaussian distribution." or "Consider a Gaussian p.d.f."? Because it is unnecessarily restrictive, given that, if I say "Consider the Gaussian distribution" I am implicitly also considering a p.d.f. and c.d.f. of the Gaussian distribution.
  
  Similarly, in the case of a discrete distribution, such as the Bernoulli distribution, only the c.d.f. and p.m.f. are defined, so the Bernoulli distribution does not have an associated p.d.f.
  
  However, it is important to recall that both continuous and discrete distributions have an associated c.d.f., so the expression "probability distribution" almost always (implicitly) refers to a c.d.f., which is defined based on a probability measure (as stated above).
  
  ### Notation
  
  In the same vein, the notation $p(x)$ can be as ambiguous as the expression "probability distribution", given that it can refer to different (but again related) concepts. However, $p(x)$ usually refers to a [probability measure](https://en.wikipedia.org/wiki/Probability_measure) (so it refers to a probability distribution, given that a probability distribution is almost always a synonym for probability measure). In this case, assuming for simplicity that the r.v. is discrete, **$p(x)$ is a shorthand for $p(X=x)$, which is also written as $\mathbb{P}(X=x)$** or $\operatorname{Pr}(X=x)$, where $X$ is a r.v., $x$ a [realization](https://en.wikipedia.org/wiki/Realization_(probability)) of $X$ (that is, a value that the r.v. $X$ can take) and $X=x$ represents an [event](https://en.wikipedia.org/wiki/Event_(probability_theory)). Given that an r.v. is a function, the notation $X=x$ may look a bit weird.
  
  In the case of a discrete r.v., $p(x)$ can also refer to a p.m.f. and it can be defined as $p\_X(x) = \mathbb{P}(X=x)$ (I added the subscript $X$ to $p$ to emphasize that this is the p.m.f. of the discrete r.v. $X$). In the case of a continuous r.v., the p.d.f. is often denoted as $f$. In the case of both discrete and continuous r.v.s, the c.d.f is usually denoted with $F$ and it is defined as $F\_X(x) = \mathbb{P}(X \leq x)$, where $\mathbb{P}$ is again a probability measure (or probability distribution). The p.d.f. of a continuous r.v. is then defined as the derivative of $F$. At this point, it should be clear why a probability distribution can refer to different but related concepts, but, in any case, it always refers to a [probability measure](https://en.wikipedia.org/wiki/Probability_measure).
  
  ### Empirical distributions
  
  There are also [empirical distributions](https://en.wikipedia.org/wiki/Empirical_distribution_function), which are distributions of the data that you have collected. For example, if you toss a coin 10 times, you will collect the results ("heads" or "tails"). You can count the number of times the coin landed on heads and tails, then you plot these numbers as a histogram, which essentially represents your empirical distribution, where the adjective "empirical" usually refers to the fact that there is an experiment involved.
  
  ### Multivariate r.v.s and distributions
  
  To complicate things even more, there are also [multivariate random variables and probability distributions](https://en.wikipedia.org/wiki/Multivariate_normal_distribution). However, all the concepts above more or less are also applicable in this case.
  
  ### Parametrized distributions
  
  A parametrized probability distribution, often denoted by $p\_{\theta}$, is
  a [family of probability distributions](https://en.wikipedia.org/wiki/Parametric_model) (defined by the parameters $\theta$), rather than a single probability distribution. For example, $\mathcal{N}(0, 1)$ refers to a *single* Gaussian distribution with zero mean and unit variance. However, $\mathcal{N}(\mu, \sigma)$, where $\theta=(\mu, \sigma)$ is a variable, is a family (or collection) of distributions.
  
  ### Conclusion
  
  To conclude, it is completely understandable that you are confused, given that the terminology and notation are used inconsistently, and there are several involved concepts, which I have not extensively covered in this answer (for example, I have not mentioned the concept of a probability space). If you get familiar with the concepts of probability measures, random variables, p.m.f., p.d.f., c.d.f., etc., and how they are related, then you will start to get a better feeling of the whole picture.
  
  
  ---
  
  ### Answer by username_2 on 2019-11-28. Score: 2
  
  A probability distribution in ML is the same as a probability distribution elsewhere.
  
  A probability distribution (or probability function, or probability mass function, or probability density function) is any function that accepts as input elements of some specific set $x \in X$, and produces as output, real-valued numbers between 0 and 1 (inclusive), such that $\int\_{x \in X} p(x) = 1$ or for discrete sets, $\sum\_{x \in X} p(x) = 1$.
  
  These distributions can also be more complex. For example, a conditional probability distribution $P(Y|X)$ or a joint probability distribution $P(X,Y)$ accept more than one input, but again, are constrained to producing an output in the range 0 to 1, and to ensuring that the summation of the output over all possible inputs is exactly 1.
  
  When these conditions are met, the functions output can be interpreted as a belief about the percentage of times the input event will occur, out of all events, or as a degree of believe in the input event having occurred vs. other events (i.e. it can be interpreted as a probability).
  
  
  ---
  
  