Answers.Body: <p>For a mixture of N Gaussians, the mode (aka the point with the highest likelihood) is either in a mode or connected to a mode (referring to the ones of the single Gaussian distributions mode are aggregating for the mixture).</p>
  <p>Given this observation, you just need to initialize N points in the modes of the models, run Gradient Ascent on them, and check at the end which point is the one with the highest likelihood</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-10T22:05:29.290Z
Answers.LastActivityDate: 2024-08-11T10:58:39.570Z
Answers.LastEditDate: 2024-08-11T10:58:39.570Z
Answers.UserDisplayName: Alberto
Answers.Body: <p><a href="https://i.sstatic.net/xVbpLpUi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xVbpLpUi.png" alt="enter image description here" /></a></p>
  <p>Here you can see that even if the <span class="math-container">$\mu$</span> values are very close that the position at which y can be maximized is not one of the available <span class="math-container">$\mu's$</span>.
  You can see clearly for a normal distribution that the position at which the highest probability <span class="math-container">$\mu$</span> is.</p>
  <p><span class="math-container">$\min[(y - \mu)^2] = 0$</span></p>
  <p>Here you can see directly that for <span class="math-container">$\mu = y$</span> the probability can be maximized, if <span class="math-container">$f[x, \phi] = \mu = y$</span></p>
  <p>Unfortunately the same does not apply for the mixture of gaussian model, since there is no close form that describes that parameter.</p>
  <p>Therefore for more complex gaussian mixtures, with more components you may need an iterative approach to find the y that brings the highest probability.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-10T14:12:27.793Z
Answers.LastActivityDate: 2024-08-10T14:12:27.793Z
Answers.UserDisplayName: dracule22
Body: <p>Let's say you have a model, which consists of two Gaussian graphs, which is great for making predictions on multi-modal data. With this you can create the following probability function for each y.</p>
  <p><span class="math-container">$
      Pr(y \mid \lambda, \mu_1, \mu_2, \sigma_1^2, \sigma_2^2) =  \frac{\lambda}{\sqrt{2\pi \sigma_1^2}} \exp \left[ -\frac{(y - \mu_1)^2}{2\sigma_1^2} \right] 
       + \frac{1 - \lambda}{\sqrt{2\pi \sigma_2^2}} \exp \left[ -\frac{(y - \mu_2)^2}{2\sigma_2^2} \right],
  $</span></p>
  <p>In this case you will need a model, which predicts <span class="math-container">$\theta = \{λ, μ^1, σ^2_1, μ^2, σ^2_2\}$</span>.
  This will result in the following loss function, when you use the negative log-likelihood.
  <span class="math-container">$L = -\sum_{i=1}^I\log\left[\frac{sig[f_1[x_i, \phi]]}{\sqrt{2\pi f_3[x_i, \phi]^2}} \exp \left[ -\frac{(y_i - f_2[x_i, \phi])^2}{2f_3[x_i, \phi]^2} \right] 
       + \frac{1 - sig[f_1[x_i, \phi]]}{\sqrt{2\pi f_5[x_i, \phi]^2}} \exp \left[ -\frac{(y_i - f_4[x_i, \phi])^2}{2f_5[x_i, \phi]^2} \right]\right]$</span></p>
  <p>In the book &quot;Understanding Deep Learning&quot; it was mentioned that making inference on such a model is difficult, as there is no closed form for that and you would need an iterative approach. But I'm very confused about that statement, since I thought that the best y value, for a given set of parameters is always the most likely. Therefore I would think that you could simply do the following:</p>
  <p><span class="math-container">$
  \hat{y} =  \underset{\mu}{\text{argmax}}\left[Pr(y \mid \lambda, \mu_1, \mu_2, \sigma_1^2, \sigma_2^2)\right]
  $</span></p>
  <p>Because normally if we would use one normal distribution to approximate the probability for y, we simply can use the predicted <span class="math-container">$\mu$</span> from the <span class="math-container">$f[x, \phi]$</span> function</p>
  <p>In the example above I simply want to figure out which <span class="math-container">$\mu$</span> value leads to the highest probability. My questions is why I can't simply do this.</p>
  <p>Is it because when we have two Gaussian distributions where the <span class="math-container">$\mu$</span> of these is close to each other, that the maximum probability is not given by neither of those.</p>
  
Tags: <prediction><inference><normal-distribution>
Title: How to perform inference pn a mixture of Gaussian models?
UserDisplayName: dracule22
ThreadText: # How to perform inference pn a mixture of Gaussian models?
  
  (Asked by: username_0 on 2024-08-10)
  
  Let's say you have a model, which consists of two Gaussian graphs, which is great for making predictions on multi-modal data. With this you can create the following probability function for each y.
  
  $
  Pr(y \mid \lambda, \mu\_1, \mu\_2, \sigma\_1^2, \sigma\_2^2) = \frac{\lambda}{\sqrt{2\pi \sigma\_1^2}} \exp \left[ -\frac{(y - \mu\_1)^2}{2\sigma\_1^2} \right]
  + \frac{1 - \lambda}{\sqrt{2\pi \sigma\_2^2}} \exp \left[ -\frac{(y - \mu\_2)^2}{2\sigma\_2^2} \right],
  $
  
  In this case you will need a model, which predicts $\theta = \{λ, μ^1, σ^2\_1, μ^2, σ^2\_2\}$.
  This will result in the following loss function, when you use the negative log-likelihood.
  $L = -\sum\_{i=1}^I\log\left[\frac{sig[f\_1[x\_i, \phi]]}{\sqrt{2\pi f\_3[x\_i, \phi]^2}} \exp \left[ -\frac{(y\_i - f\_2[x\_i, \phi])^2}{2f\_3[x\_i, \phi]^2} \right]
  + \frac{1 - sig[f\_1[x\_i, \phi]]}{\sqrt{2\pi f\_5[x\_i, \phi]^2}} \exp \left[ -\frac{(y\_i - f\_4[x\_i, \phi])^2}{2f\_5[x\_i, \phi]^2} \right]\right]$
  
  In the book "Understanding Deep Learning" it was mentioned that making inference on such a model is difficult, as there is no closed form for that and you would need an iterative approach. But I'm very confused about that statement, since I thought that the best y value, for a given set of parameters is always the most likely. Therefore I would think that you could simply do the following:
  
  $
  \hat{y} = \underset{\mu}{\text{argmax}}\left[Pr(y \mid \lambda, \mu\_1, \mu\_2, \sigma\_1^2, \sigma\_2^2)\right]
  $
  
  Because normally if we would use one normal distribution to approximate the probability for y, we simply can use the predicted $\mu$ from the $f[x, \phi]$ function
  
  In the example above I simply want to figure out which $\mu$ value leads to the highest probability. My questions is why I can't simply do this.
  
  Is it because when we have two Gaussian distributions where the $\mu$ of these is close to each other, that the maximum probability is not given by neither of those.
  
  **Tags:** prediction, inference, normal-distribution
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-08-10. Score: 0 | Accepted answer
  
  For a mixture of N Gaussians, the mode (aka the point with the highest likelihood) is either in a mode or connected to a mode (referring to the ones of the single Gaussian distributions mode are aggregating for the mixture).
  
  Given this observation, you just need to initialize N points in the modes of the models, run Gradient Ascent on them, and check at the end which point is the one with the highest likelihood
  
  
  ---
  
  ### Answer by username_0 on 2024-08-10. Score: 1
  
  [![enter image description here](https://i.sstatic.net/xVbpLpUi.png)](https://i.sstatic.net/xVbpLpUi.png)
  
  Here you can see that even if the $\mu$ values are very close that the position at which y can be maximized is not one of the available $\mu's$.
  You can see clearly for a normal distribution that the position at which the highest probability $\mu$ is.
  
  $\min[(y - \mu)^2] = 0$
  
  Here you can see directly that for $\mu = y$ the probability can be maximized, if $f[x, \phi] = \mu = y$
  
  Unfortunately the same does not apply for the mixture of gaussian model, since there is no close form that describes that parameter.
  
  Therefore for more complex gaussian mixtures, with more components you may need an iterative approach to find the y that brings the highest probability.
  
  
  ---
  
  