<p>Your understanding is correct. The indicator function ensures that only the term corresponding to the true class <span class="math-container">$y_i$</span> contributes, and all other terms become <span class="math-container">$1$</span>, effectively ignoring them. This is actually a useful way to  write out the <em>likelihood function</em> for mixed continuous–discrete distributions with class-conditional <em>densities</em> similar to the <a href="https://en.wikipedia.org/wiki/Categorical_distribution#Formulating_distributions" rel="nofollow noreferrer">categorical variables</a> as mentioned below, so that you can optimize <span class="math-container">$\pi$</span> and the <span class="math-container">$(\mu_c
, Σ_c)$</span> terms <em>separately</em> based on the log-likelihood later on to derive the MLEs for the Gaussians and tied/diagonal covariances.</p>
<blockquote>
<p>Another formulation that appears more complex but facilitates mathematical manipulations is as follows, using the Iverson bracket:
<span class="math-container">$${f(x\mid {\boldsymbol {p}})=\prod _{i=1}^{k}p_{i}^{[x=i]}},$$</span>
where <span class="math-container">${[x=i]}$</span> evaluates to <span class="math-container">$1$</span> if <span class="math-container">${x=i, 0}$</span> otherwise. There are various advantages of this formulation, e.g.:</p>
</blockquote>
<blockquote>
<p>It is easier to write out the likelihood function of a set of independent identically distributed categorical variables.</p>
</blockquote>
<blockquote>
<p>It connects the categorical distribution with the related multinomial distribution.</p>
</blockquote>
<blockquote>
<p>It shows why the Dirichlet distribution is the conjugate prior of the categorical distribution, and allows the posterior distribution of the parameters to be calculated.</p>
</blockquote>

CC BY-SA 4.0
2024-02-29T05:16:51.110Z
2024-02-29T06:14:35.597Z
2024-02-29T06:14:35.597Z
cinch
<p>Im trying to understand how the likelhood function for gaussian discriminant analysis is derived.  I self studying Murphy's Probabilistic Machine learning, and in it, he states the likelihood function as follows:</p>
<p><span class="math-container">$$P(D|\theta) = \prod_{i=1:n} Cat(y^i|\pi) \prod_{c=1:C} N(x^i | \mu_c, \sum_c)^{I(y^i=c)}$$</span></p>
<p>where <span class="math-container">$N(x^i | \mu_c, \sum_c)$</span> is the normally distributed random variable with mean <span class="math-container">$\mu_c$</span> which is the mean vector for each feature whose class label is <span class="math-container">$y^i=c$</span> and <span class="math-container">$\sum_c$</span> is the covariance matrix between the features for the ith example.</p>
<p>Based on my understanding, the way I have been deriving Likelihood is by assuming the following:</p>
<p>We are given the dataset <span class="math-container">$D = {(x^1, y^1),(x^2, y^2), ..., (x^n, y^n)}$</span> drawn from unknown distribution <span class="math-container">$p(X,Y)$</span>.  B/c all pairs are sampled iid, we obtain:</p>
<p><span class="math-container">$$p(D) = p((x^1, y^1),(x^2, y^2), ..., (x^n, y^n)) = \prod_{i=1:n} p(x^i, y^i | \theta)$$</span></p>
<p>Hence...</p>
<p><span class="math-container">$$p(D|\theta) = \prod_{i=1:n} p(x^i | y^i = c, \theta)p(y^i=c|\theta)$$</span></p>
<p>Now, I can see the prior is just a categorical distribution given by <span class="math-container">$Cat(y | \pi)$</span> so I can see how the Cat function is coming in equation for likelihood since <span class="math-container">$y^i$</span> falls in one of C categories.  But what im not able to understand is why is the equation taking product of all clases from <span class="math-container">$c=1:C$</span> for the class conditional density?</p>
<p>Is it assuming that for the ith sample, the label can only take on one of c values, hence the distribution of <span class="math-container">$p(x|y,\theta)$</span> is given by the product of all the disributions raised to the power of indicator?  hence only one of them can be true?</p>
<p>Thanks</p>

<machine-learning><generative-model><maximum-likelihood>
Likelihood function for Gaussian Discriminant Analsis
turtle_in_mind
# Likelihood function for Gaussian Discriminant Analsis

(Asked by: username_0 on 2024-02-29)

Im trying to understand how the likelhood function for gaussian discriminant analysis is derived. I self studying Murphy's Probabilistic Machine learning, and in it, he states the likelihood function as follows:

$$P(D|\theta) = \prod\_{i=1:n} Cat(y^i|\pi) \prod\_{c=1:C} N(x^i | \mu\_c, \sum\_c)^{I(y^i=c)}$$

where $N(x^i | \mu\_c, \sum\_c)$ is the normally distributed random variable with mean $\mu\_c$ which is the mean vector for each feature whose class label is $y^i=c$ and $\sum\_c$ is the covariance matrix between the features for the ith example.

Based on my understanding, the way I have been deriving Likelihood is by assuming the following:

We are given the dataset $D = {(x^1, y^1),(x^2, y^2), ..., (x^n, y^n)}$ drawn from unknown distribution $p(X,Y)$. B/c all pairs are sampled iid, we obtain:

$$p(D) = p((x^1, y^1),(x^2, y^2), ..., (x^n, y^n)) = \prod\_{i=1:n} p(x^i, y^i | \theta)$$

Hence...

$$p(D|\theta) = \prod\_{i=1:n} p(x^i | y^i = c, \theta)p(y^i=c|\theta)$$

Now, I can see the prior is just a categorical distribution given by $Cat(y | \pi)$ so I can see how the Cat function is coming in equation for likelihood since $y^i$ falls in one of C categories. But what im not able to understand is why is the equation taking product of all clases from $c=1:C$ for the class conditional density?

Is it assuming that for the ith sample, the label can only take on one of c values, hence the distribution of $p(x|y,\theta)$ is given by the product of all the disributions raised to the power of indicator? hence only one of them can be true?

Thanks

**Tags:** machine-learning, generative-model, maximum-likelihood


---

## 1 answers

### Answer by username_1 on 2024-02-29. Score: 3 | Accepted answer

Your understanding is correct. The indicator function ensures that only the term corresponding to the true class $y\_i$ contributes, and all other terms become $1$, effectively ignoring them. This is actually a useful way to write out the *likelihood function* for mixed continuous–discrete distributions with class-conditional *densities* similar to the [categorical variables](https://en.wikipedia.org/wiki/Categorical_distribution#Formulating_distributions) as mentioned below, so that you can optimize $\pi$ and the $(\mu\_c
, Σ\_c)$ terms *separately* based on the log-likelihood later on to derive the MLEs for the Gaussians and tied/diagonal covariances.

> Another formulation that appears more complex but facilitates mathematical manipulations is as follows, using the Iverson bracket:
> $${f(x\mid {\boldsymbol {p}})=\prod \_{i=1}^{k}p\_{i}^{[x=i]}},$$
> where ${[x=i]}$ evaluates to $1$ if ${x=i, 0}$ otherwise. There are various advantages of this formulation, e.g.:

> It is easier to write out the likelihood function of a set of independent identically distributed categorical variables.

> It connects the categorical distribution with the related multinomial distribution.

> It shows why the Dirichlet distribution is the conjugate prior of the categorical distribution, and allows the posterior distribution of the parameters to be calculated.


---

