Answers.Body: <p>The probability density is used to 'measure how good' the parameters are because it is a natural way of quantifying if these parameters are good for the observed data.</p>
  <p>Also, as the notation often causes some confusion, <span class="math-container">$L(\theta | x)$</span> denotes the probability of <em>all</em> of your observed data, not just one value. Also the &quot;<span class="math-container">$|$</span>&quot; may cause confusion as it looks like we are conditioning on <span class="math-container">$x$</span> but this is not the case - it may be better practice to use <span class="math-container">$L(\theta; x)$</span> which is the notation used when I was learning likelihood. Further, as you have written <span class="math-container">$L(\theta | x) = p_\theta(x)$</span> I would like to clarify that if we are being precise in our definitions then this is only correct if you have one observed data point, assuming that you meant <span class="math-container">$p_\theta(x)$</span> is the density of <span class="math-container">$X$</span>.</p>
  <p>In my example below I use <span class="math-container">$p_\mu(x)$</span> to denote the density of the normal distribution with mean parameter <span class="math-container">$\mu$</span>, but the density of the likelihood is the <em>product</em> of all the densities (because we assumed iid data). It is crucial that you understand that in general the likelihood is the probability of your observed data and not just the density or the product of densities as you may not always have iid and so it will not always boil down to taking the product of some densities.</p>
  <p>The idea of Maximum Likelihood is to maximising the (log-)likelihood for a given set of data. This means that we need to choose a probability distribution that is parameterised by some parameters <span class="math-container">$\boldsymbol{\theta}$</span> and then optimise the parameters such that the likelihood is maximum.</p>
  <p>Assuming we don't remove any constants then this makes intuitive sense as maximising the likelihood would be maximising the probability that the data came from this distribution -- i.e. the data we observed is most likely to have come from the given distribution with the given parameters. This is important as it means we then have the most likely model of our data which allows us to use this distribution to make inference about our data.</p>
  <p>As an example, imagine if I had some iid data <span class="math-container">$x_1, x_2, ..., x_n \sim \mathcal{N}(0, 1)$</span>. Now I could try to fit a <span class="math-container">$\mathcal{N}(\mu, 1)$</span> to this data and optimise for <span class="math-container">$\mu$</span>. If I chose e.g. <span class="math-container">$\mu = -1000000$</span> then the likelihood (again assuming we don't remove any constants) would be <span class="math-container">$\approx 0$</span>. If I chose a value of <span class="math-container">$\mu = 0.1$</span> then the likelihood would be much higher because this parameter is closer to the true parameter value.</p>
  <p>To see why it is higher, recall that the likelihood for iid data is given by
  <span class="math-container">$$\prod_{i=1}^n p_\mu(x_i)$$</span>
  and if we evaluated our likelihood at <span class="math-container">$\mu=-10000000$</span> then you're going to be taking the product of lots of numbers that are <span class="math-container">$\approx 0$</span> - if you think about the bell curve shape of a Normal distribution that is centred at <span class="math-container">$-10000000$</span> with variance 1 then the density at the true <span class="math-container">$x_i$</span> values (recalling they are simulated from a unit Normal) would be approx <span class="math-container">$0$</span> - whereas if we evaluated the <span class="math-container">$x_i$</span> at the density of a Normal distribution centred at <span class="math-container">$0.1$</span> then the density will be non-zero and so your likelihood will have a higher value.</p>
  <p>To summarise, the density value can be used to measure how good parameters are for a set of data as maximising wrt the parameters is analogous to maximising the probability that your data arose from said distribution.</p>
  <p>As an aside, note that the definition of likelihood is the probability of observing your data under some assumed distribution. For discrete random variables this is fine, but for continuous distributions we have to be a more subtle. For any continuous random variable <span class="math-container">$X$</span> we have <span class="math-container">$\mathbb{P}(X=x) = 0$</span>. However, for a very small <span class="math-container">$\delta$</span> we can say that</p>
  <p><span class="math-container">$$\mathbb{P}(x - \frac{\delta}{2} &lt; X \leq x + \frac{\delta}{2}) = \int_{x - \frac{\delta}{2}}^{x + \frac{\delta}{2}} f_X(x) dx \approx \delta p_X(x) \; ;$$</span></p>
  <p>you can think of this approximation by visualising an integral and recalling that an integral represents the area under the curve, and so for small <span class="math-container">$\delta$</span> this integral can be approximated by taking the area of a rectangle which is width <span class="math-container">$\times$</span> height, where the width is <span class="math-container">$\delta$</span> and the height is <span class="math-container">$f_X(x)$</span>. This justifies our use of the density function for continuous random variables. Note that typically in maximum likelihood we omit any multiplicative constants as they don't depend on the parameters which is what happens with the <span class="math-container">$\delta$</span> from this justification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T12:55:07.193Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/118208/discussion-between-david-ireland-and-duttaa).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T11:11:04.687Z
Answers.LastActivityDate: 2021-01-08T12:53:49.683Z
Answers.LastEditDate: 2021-01-08T12:53:49.683Z
Answers.UserDisplayName: David
Body: <p>Consider our parametric model <span class="math-container">$p_\theta$</span> for an underlying probabilistic distribution <span class="math-container">$p_{data}$</span>.</p>
  <p>Now, the likelihood of an observation <span class="math-container">$x$</span> is generally defined as <span class="math-container">$L(\theta|x) = p_{\theta}(x)$</span>.</p>
  <p>The purpose of the likelihood is to quantify how good the parameters are. How can the probability density at a given observation, <span class="math-container">$p_{\theta}(x)$</span>, measure how good the parameters <span class="math-container">$\theta$</span> are?</p>
  <p>Is there any relation between the <em>goodness of parameters</em> and <em>the probability density value of an observation</em>?</p>
  
Tags: <machine-learning><comparison><probability-distribution><maximum-likelihood>
Title: How can a probability density value be used for the likelihood calculation?
UserDisplayName: hanugm
ThreadText: # How can a probability density value be used for the likelihood calculation?
  
  (Asked by: username_0 on 2021-01-08)
  
  Consider our parametric model $p\_\theta$ for an underlying probabilistic distribution $p\_{data}$.
  
  Now, the likelihood of an observation $x$ is generally defined as $L(\theta|x) = p\_{\theta}(x)$.
  
  The purpose of the likelihood is to quantify how good the parameters are. How can the probability density at a given observation, $p\_{\theta}(x)$, measure how good the parameters $\theta$ are?
  
  Is there any relation between the *goodness of parameters* and *the probability density value of an observation*?
  
  **Tags:** machine-learning, comparison, probability-distribution, maximum-likelihood
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-08. Score: 6
  
  The probability density is used to 'measure how good' the parameters are because it is a natural way of quantifying if these parameters are good for the observed data.
  
  Also, as the notation often causes some confusion, $L(\theta | x)$ denotes the probability of *all* of your observed data, not just one value. Also the "$|$" may cause confusion as it looks like we are conditioning on $x$ but this is not the case - it may be better practice to use $L(\theta; x)$ which is the notation used when I was learning likelihood. Further, as you have written $L(\theta | x) = p\_\theta(x)$ I would like to clarify that if we are being precise in our definitions then this is only correct if you have one observed data point, assuming that you meant $p\_\theta(x)$ is the density of $X$.
  
  In my example below I use $p\_\mu(x)$ to denote the density of the normal distribution with mean parameter $\mu$, but the density of the likelihood is the *product* of all the densities (because we assumed iid data). It is crucial that you understand that in general the likelihood is the probability of your observed data and not just the density or the product of densities as you may not always have iid and so it will not always boil down to taking the product of some densities.
  
  The idea of Maximum Likelihood is to maximising the (log-)likelihood for a given set of data. This means that we need to choose a probability distribution that is parameterised by some parameters $\boldsymbol{\theta}$ and then optimise the parameters such that the likelihood is maximum.
  
  Assuming we don't remove any constants then this makes intuitive sense as maximising the likelihood would be maximising the probability that the data came from this distribution -- i.e. the data we observed is most likely to have come from the given distribution with the given parameters. This is important as it means we then have the most likely model of our data which allows us to use this distribution to make inference about our data.
  
  As an example, imagine if I had some iid data $x\_1, x\_2, ..., x\_n \sim \mathcal{N}(0, 1)$. Now I could try to fit a $\mathcal{N}(\mu, 1)$ to this data and optimise for $\mu$. If I chose e.g. $\mu = -1000000$ then the likelihood (again assuming we don't remove any constants) would be $\approx 0$. If I chose a value of $\mu = 0.1$ then the likelihood would be much higher because this parameter is closer to the true parameter value.
  
  To see why it is higher, recall that the likelihood for iid data is given by
  $$\prod\_{i=1}^n p\_\mu(x\_i)$$
  and if we evaluated our likelihood at $\mu=-10000000$ then you're going to be taking the product of lots of numbers that are $\approx 0$ - if you think about the bell curve shape of a Normal distribution that is centred at $-10000000$ with variance 1 then the density at the true $x\_i$ values (recalling they are simulated from a unit Normal) would be approx $0$ - whereas if we evaluated the $x\_i$ at the density of a Normal distribution centred at $0.1$ then the density will be non-zero and so your likelihood will have a higher value.
  
  To summarise, the density value can be used to measure how good parameters are for a set of data as maximising wrt the parameters is analogous to maximising the probability that your data arose from said distribution.
  
  As an aside, note that the definition of likelihood is the probability of observing your data under some assumed distribution. For discrete random variables this is fine, but for continuous distributions we have to be a more subtle. For any continuous random variable $X$ we have $\mathbb{P}(X=x) = 0$. However, for a very small $\delta$ we can say that
  
  $$\mathbb{P}(x - \frac{\delta}{2} < X \leq x + \frac{\delta}{2}) = \int\_{x - \frac{\delta}{2}}^{x + \frac{\delta}{2}} f\_X(x) dx \approx \delta p\_X(x) \; ;$$
  
  you can think of this approximation by visualising an integral and recalling that an integral represents the area under the curve, and so for small $\delta$ this integral can be approximated by taking the area of a rectangle which is width $\times$ height, where the width is $\delta$ and the height is $f\_X(x)$. This justifies our use of the density function for continuous random variables. Note that typically in maximum likelihood we omit any multiplicative constants as they don't depend on the parameters which is what happens with the $\delta$ from this justification.
  
  > **Comments:**
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/118208/discussion-between-username_1-ireland-and-duttaa).  
  
  ---
  
  