text: Exercise - Variational Mean Field Approximation for Univariate Gaussian
  
  Introduction
  
  [TODO]
  
  In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.
  
  Requirements
  
  [TODO]
  
  Python Modules
  
  %matplotlib inline
  import numpy as np
  from matplotlib import pyplot as plt
  import scipy.stats
  np.random.seed(40)
  
  Data
  
  Data: $X \sim \mathcal N(\mu, \frac{1}{\tau})$
  
  Probability Density Function: $p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$
  
  with
  -$\mu$: mean -$\sigma^2$: variance -$\tau =\frac{1}{\sigma^2}$ : precision
  
  # generation of observed data
  N = 10
  mu = 10.
  sigma = 2.
  X = np.random.normal(mu, sigma, N)
  x = np.arange(3,18,0.01)
  p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)
  plt.plot(x, p_x, label="true Gaussian")
  plt.plot(X, np.zeros_like(X), "ro", label="Data")
  plt.title("")
  plt.xlabel("x")
  plt.ylabel("p(x)")
  plt.legend();
  
  Exercises
  
  Exercise - Mean Field Approximation of the Posterior
  
  Find the mean field approximation of the posterior:
  
  $p(\mu, \tau \mid X) \approx q(\mu)q(\tau)$.
  
  • The observed data was sampled from a Gaussian distribution:$X \sim \mathcal N(\mu, \frac{1}{\tau})$.
  • Use a constant prior for the mean and the precision$\tau = \frac{1}{\sigma^2}$: $p(\mu, \tau) = const. \quad \text{ for } \tau > 0$ $\theta = (\theta_0, \theta_1) = (\mu, \tau)$
  
  Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use the constant (improper) prior because it's a little bit easier.
  
  Recap: Mean field approximation
  
  Loop until convergence: $\log q({\theta}_k) = \mathbb E_{q_{-k}} \left[ \log{\hat p( {\theta} \mid {\mathcal D} )} \right] + const.$
  
  Hint:
  
  Use the "Sum of difference of the mean": $\sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n(x_i-\bar{x})^2 + n(\bar{x} -\mu)^2$ with $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$
  
  Exercise - Proof
  
  Show that: $q(\mu) = \mathcal N(\bar{X}, \frac{1}{\gamma_1})$ with$\gamma_1 = m \mathbb E_{q_{\tau}}[\tau]$
  
  and
  
  $q(\tau) = \text{Gamma}(\frac{m}{2}+1, \frac{2}{\gamma_2})$
  
  with $\gamma_2= \sum_{i=1}^m \left(X_i^2\right)-m\bar X^2 + m\text{var}_{q_{\mu}}(\mu)$
  
  Gamma distribution
  
  $\text{Gamma}(k, \theta') = p(x) = x^{k-1}\frac{e^{-x/\theta'}}{\theta'^k\Gamma(k)}$ -$k$ is the shape -$\theta'$ the scale, -$\Gamma(.)$ is the Gamma function
  
  Exercise - Implementation of the Mean Field Approximation
  
  Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.
  
  If everything is correct, executing the cells below should plot figures similar to these:
  
  def optim(X=X,  mean_mu=1, sigma_quare_mu=1, loc_tau=1.,  scale_tau=1.):
  raise NotImplementedError()
  mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)
  mean_mu, sigma_quare_mu, loc_tau, scale_tau
  plt.figure(figsize=(12,4))
  
  x = np.arange(5,15,0.01)
  p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))
  ax = plt.subplot(121)
  ax.plot(x, p_mu)
  ax.set_xlabel("$\mu$")
  ax.set_ylabel("q($\mu$)")
  ax.set_title("Mean: q($\\mu$)")
  print("true mu: ", mu)
  
  x = np.arange(0,1,0.01)
  p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)
  ax = plt.subplot(122)
  ax.plot(x, p_tau)
  ax.set_xlabel("$\\tau$")
  ax.set_ylabel("q($\\tau$)")
  ax.set_title("Precision: q($\\tau$)")
  print("true tau: ", 1/sigma**2)
  
  Literature
  
  [TODO]
  
  The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).
  
  Exercise - Variational Mean Field Approximation for Univariate Gaussian
  by Christian Herta
explanation: The provided text extract is a detailed exercise on the Variational Mean Field Approximation for a Univariate Gaussian distribution. It includes an introduction, Python code for generating observed data, and several exercises with specific tasks and hints. The exercises involve finding the mean field approximation of the posterior, proving certain mathematical relationships, and implementing the mean field approximation.
  
  The document contains multiple questions and tasks, but no answers are provided for these questions.
distilabel_metadata.content: Evaluate the user given text extract and determine if it contains a math question and optionally and answer to this question.
  
  Output rules:
  
  1. Briefly describe the document and mention if it contains a question or a question and answer.
  2. Finalize the output with: Label: no-question,question,question-and-answer,multiple-question,multiple-question-and-answer
distilabel_metadata.role: system
distilabel_metadata.content: Exercise - Variational Mean Field Approximation for Univariate Gaussian
  
  Introduction
  
  [TODO]
  
  In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.
  
  Requirements
  
  [TODO]
  
  Python Modules
  
  %matplotlib inline
  import numpy as np
  from matplotlib import pyplot as plt
  import scipy.stats
  np.random.seed(40)
  
  Data
  
  Data: $X \sim \mathcal N(\mu, \frac{1}{\tau})$
  
  Probability Density Function: $p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$
  
  with
  -$\mu$: mean -$\sigma^2$: variance -$\tau =\frac{1}{\sigma^2}$ : precision
  
  # generation of observed data
  N = 10
  mu = 10.
  sigma = 2.
  X = np.random.normal(mu, sigma, N)
  x = np.arange(3,18,0.01)
  p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)
  plt.plot(x, p_x, label="true Gaussian")
  plt.plot(X, np.zeros_like(X), "ro", label="Data")
  plt.title("")
  plt.xlabel("x")
  plt.ylabel("p(x)")
  plt.legend();
  
  Exercises
  
  Exercise - Mean Field Approximation of the Posterior
  
  Find the mean field approximation of the posterior:
  
  $p(\mu, \tau \mid X) \approx q(\mu)q(\tau)$.
  
  • The observed data was sampled from a Gaussian distribution:$X \sim \mathcal N(\mu, \frac{1}{\tau})$.
  • Use a constant prior for the mean and the precision$\tau = \frac{1}{\sigma^2}$: $p(\mu, \tau) = const. \quad \text{ for } \tau > 0$ $\theta = (\theta_0, \theta_1) = (\mu, \tau)$
  
  Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use the constant (improper) prior because it's a little bit easier.
  
  Recap: Mean field approximation
  
  Loop until convergence: $\log q({\theta}_k) = \mathbb E_{q_{-k}} \left[ \log{\hat p( {\theta} \mid {\mathcal D} )} \right] + const.$
  
  Hint:
  
  Use the "Sum of difference of the mean": $\sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n(x_i-\bar{x})^2 + n(\bar{x} -\mu)^2$ with $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$
  
  Exercise - Proof
  
  Show that: $q(\mu) = \mathcal N(\bar{X}, \frac{1}{\gamma_1})$ with$\gamma_1 = m \mathbb E_{q_{\tau}}[\tau]$
  
  and
  
  $q(\tau) = \text{Gamma}(\frac{m}{2}+1, \frac{2}{\gamma_2})$
  
  with $\gamma_2= \sum_{i=1}^m \left(X_i^2\right)-m\bar X^2 + m\text{var}_{q_{\mu}}(\mu)$
  
  Gamma distribution
  
  $\text{Gamma}(k, \theta') = p(x) = x^{k-1}\frac{e^{-x/\theta'}}{\theta'^k\Gamma(k)}$ -$k$ is the shape -$\theta'$ the scale, -$\Gamma(.)$ is the Gamma function
  
  Exercise - Implementation of the Mean Field Approximation
  
  Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.
  
  If everything is correct, executing the cells below should plot figures similar to these:
  
  def optim(X=X,  mean_mu=1, sigma_quare_mu=1, loc_tau=1.,  scale_tau=1.):
  raise NotImplementedError()
  mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)
  mean_mu, sigma_quare_mu, loc_tau, scale_tau
  plt.figure(figsize=(12,4))
  
  x = np.arange(5,15,0.01)
  p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))
  ax = plt.subplot(121)
  ax.plot(x, p_mu)
  ax.set_xlabel("$\mu$")
  ax.set_ylabel("q($\mu$)")
  ax.set_title("Mean: q($\\mu$)")
  print("true mu: ", mu)
  
  x = np.arange(0,1,0.01)
  p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)
  ax = plt.subplot(122)
  ax.plot(x, p_tau)
  ax.set_xlabel("$\\tau$")
  ax.set_ylabel("q($\\tau$)")
  ax.set_title("Precision: q($\\tau$)")
  print("true tau: ", 1/sigma**2)
  
  Literature
  
  [TODO]
  
  The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).
  
  Exercise - Variational Mean Field Approximation for Univariate Gaussian
  by Christian Herta
distilabel_metadata.role: user
distilabel_metadata.raw_output_q_a_detector_0: The provided text extract is a detailed exercise on the Variational Mean Field Approximation for a Univariate Gaussian distribution. It includes an introduction, Python code for generating observed data, and several exercises with specific tasks and hints. The exercises involve finding the mean field approximation of the posterior, proving certain mathematical relationships, and implementing the mean field approximation.
  
  The document contains multiple questions and tasks, but no answers are provided for these questions.
  
  Label: multiple-question