Answers.Body: <p>Your understanding is correct. The indicator function ensures that only the term corresponding to the true class <span class="math-container">$y_i$</span> contributes, and all other terms become <span class="math-container">$1$</span>, effectively ignoring them. This is actually a useful way to  write out the <em>likelihood function</em> for mixed continuous–discrete distributions with class-conditional <em>densities</em> similar to the <a href="https://en.wikipedia.org/wiki/Categorical_distribution#Formulating_distributions" rel="nofollow noreferrer">categorical variables</a> as mentioned below, so that you can optimize <span class="math-container">$\pi$</span> and the <span class="math-container">$(\mu_c
  , Σ_c)$</span> terms <em>separately</em> based on the log-likelihood later on to derive the MLEs for the Gaussians and tied/diagonal covariances.</p>
  <blockquote>
  <p>Another formulation that appears more complex but facilitates mathematical manipulations is as follows, using the Iverson bracket:
  <span class="math-container">$${f(x\mid {\boldsymbol {p}})=\prod _{i=1}^{k}p_{i}^{[x=i]}},$$</span>
  where <span class="math-container">${[x=i]}$</span> evaluates to <span class="math-container">$1$</span> if <span class="math-container">${x=i, 0}$</span> otherwise. There are various advantages of this formulation, e.g.:</p>
  </blockquote>
  <blockquote>
  <p>It is easier to write out the likelihood function of a set of independent identically distributed categorical variables.</p>
  </blockquote>
  <blockquote>
  <p>It connects the categorical distribution with the related multinomial distribution.</p>
  </blockquote>
  <blockquote>
  <p>It shows why the Dirichlet distribution is the conjugate prior of the categorical distribution, and allows the posterior distribution of the parameters to be calculated.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T05:16:51.110Z
Answers.LastActivityDate: 2024-02-29T06:14:35.597Z
Answers.LastEditDate: 2024-02-29T06:14:35.597Z
Answers.UserDisplayName: cinch
Body: <p>Im trying to understand how the likelhood function for gaussian discriminant analysis is derived.  I self studying Murphy's Probabilistic Machine learning, and in it, he states the likelihood function as follows:</p>
  <p><span class="math-container">$$P(D|\theta) = \prod_{i=1:n} Cat(y^i|\pi) \prod_{c=1:C} N(x^i | \mu_c, \sum_c)^{I(y^i=c)}$$</span></p>
  <p>where <span class="math-container">$N(x^i | \mu_c, \sum_c)$</span> is the normally distributed random variable with mean <span class="math-container">$\mu_c$</span> which is the mean vector for each feature whose class label is <span class="math-container">$y^i=c$</span> and <span class="math-container">$\sum_c$</span> is the covariance matrix between the features for the ith example.</p>
  <p>Based on my understanding, the way I have been deriving Likelihood is by assuming the following:</p>
  <p>We are given the dataset <span class="math-container">$D = {(x^1, y^1),(x^2, y^2), ..., (x^n, y^n)}$</span> drawn from unknown distribution <span class="math-container">$p(X,Y)$</span>.  B/c all pairs are sampled iid, we obtain:</p>
  <p><span class="math-container">$$p(D) = p((x^1, y^1),(x^2, y^2), ..., (x^n, y^n)) = \prod_{i=1:n} p(x^i, y^i | \theta)$$</span></p>
  <p>Hence...</p>
  <p><span class="math-container">$$p(D|\theta) = \prod_{i=1:n} p(x^i | y^i = c, \theta)p(y^i=c|\theta)$$</span></p>
  <p>Now, I can see the prior is just a categorical distribution given by <span class="math-container">$Cat(y | \pi)$</span> so I can see how the Cat function is coming in equation for likelihood since <span class="math-container">$y^i$</span> falls in one of C categories.  But what im not able to understand is why is the equation taking product of all clases from <span class="math-container">$c=1:C$</span> for the class conditional density?</p>
  <p>Is it assuming that for the ith sample, the label can only take on one of c values, hence the distribution of <span class="math-container">$p(x|y,\theta)$</span> is given by the product of all the disributions raised to the power of indicator?  hence only one of them can be true?</p>
  <p>Thanks</p>
  
Tags: <machine-learning><generative-model><maximum-likelihood>
Title: Likelihood function for Gaussian Discriminant Analsis
UserDisplayName: turtle_in_mind
ThreadText: # Likelihood function for Gaussian Discriminant Analsis
  
  (Asked by: username_0 on 2024-02-29)
  
  Im trying to understand how the likelhood function for gaussian discriminant analysis is derived. I self studying Murphy's Probabilistic Machine learning, and in it, he states the likelihood function as follows:
  
  $$P(D|\theta) = \prod\_{i=1:n} Cat(y^i|\pi) \prod\_{c=1:C} N(x^i | \mu\_c, \sum\_c)^{I(y^i=c)}$$
  
  where $N(x^i | \mu\_c, \sum\_c)$ is the normally distributed random variable with mean $\mu\_c$ which is the mean vector for each feature whose class label is $y^i=c$ and $\sum\_c$ is the covariance matrix between the features for the ith example.
  
  Based on my understanding, the way I have been deriving Likelihood is by assuming the following:
  
  We are given the dataset $D = {(x^1, y^1),(x^2, y^2), ..., (x^n, y^n)}$ drawn from unknown distribution $p(X,Y)$. B/c all pairs are sampled iid, we obtain:
  
  $$p(D) = p((x^1, y^1),(x^2, y^2), ..., (x^n, y^n)) = \prod\_{i=1:n} p(x^i, y^i | \theta)$$
  
  Hence...
  
  $$p(D|\theta) = \prod\_{i=1:n} p(x^i | y^i = c, \theta)p(y^i=c|\theta)$$
  
  Now, I can see the prior is just a categorical distribution given by $Cat(y | \pi)$ so I can see how the Cat function is coming in equation for likelihood since $y^i$ falls in one of C categories. But what im not able to understand is why is the equation taking product of all clases from $c=1:C$ for the class conditional density?
  
  Is it assuming that for the ith sample, the label can only take on one of c values, hence the distribution of $p(x|y,\theta)$ is given by the product of all the disributions raised to the power of indicator? hence only one of them can be true?
  
  Thanks
  
  **Tags:** machine-learning, generative-model, maximum-likelihood
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-29. Score: 3 | Accepted answer
  
  Your understanding is correct. The indicator function ensures that only the term corresponding to the true class $y\_i$ contributes, and all other terms become $1$, effectively ignoring them. This is actually a useful way to write out the *likelihood function* for mixed continuous–discrete distributions with class-conditional *densities* similar to the [categorical variables](https://en.wikipedia.org/wiki/Categorical_distribution#Formulating_distributions) as mentioned below, so that you can optimize $\pi$ and the $(\mu\_c
  , Σ\_c)$ terms *separately* based on the log-likelihood later on to derive the MLEs for the Gaussians and tied/diagonal covariances.
  
  > Another formulation that appears more complex but facilitates mathematical manipulations is as follows, using the Iverson bracket:
  > $${f(x\mid {\boldsymbol {p}})=\prod \_{i=1}^{k}p\_{i}^{[x=i]}},$$
  > where ${[x=i]}$ evaluates to $1$ if ${x=i, 0}$ otherwise. There are various advantages of this formulation, e.g.:
  
  > It is easier to write out the likelihood function of a set of independent identically distributed categorical variables.
  
  > It connects the categorical distribution with the related multinomial distribution.
  
  > It shows why the Dirichlet distribution is the conjugate prior of the categorical distribution, and allows the posterior distribution of the parameters to be calculated.
  
  
  ---
  
  