Answers.Body: <p>I call this difference <strong>prescribed</strong> vs <strong>implicit</strong> models.</p>
  <p>To my knowledge, a <strong>parametric</strong> model has a fixed,finite number of parameters with respect to the sample size (e.g a linear model), while <strong>nonparametric</strong> model have an increasing number of parameters with respect to data. See this <a href="https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html" rel="nofollow noreferrer">blog post</a> and this <a href="https://stats.stackexchange.com/questions/268638/what-exactly-is-the-difference-between-a-parametric-and-non-parametric-model">answer</a> for more precise info. Since most models use a Neural Network to generate an output, I prefer the terms prescribed vs implicit.</p>
  <p>Viewing this way,</p>
  <ul>
  <li><p><strong>Implicit</strong> models directly generate the output by passing a noise vector through a deterministic function (Generally a NN). They do not have a global likelihood, and are less limited in the architecture.
  A great example of an implicit model, as you stated, are GANs.</p>
  </li>
  <li><p><strong>Prescribed</strong> models provides an explicit parametric specification of the output distribution <span class="math-container">$p(x)$</span>. They can estimate the likelihood of data, but are more constrained in the architecture.</p>
  <ul>
  <li>An Autoregressive model fits a Bernoulli distribution , learning a neural network to model <span class="math-container">$p(x_i|x_{&lt;i})$</span></li>
  <li>Normalizing flows learn a change of variable to reshape a gaussian source noise. They must learn an invertible function, so the architecture is heavily constrained.</li>
  <li>VAEs assume a Gaussian prior <span class="math-container">$p(z) \sim \mathcal{N}(0,\mathbb{I})$</span> that is  decoded by a NN which models <span class="math-container">$p(x|z)$</span>. They have less limits in the architecture, but they lose the closed form likelihood, and they are trained by maximizing an ELBO. Since they still have (even if approximated) likelihood, they are prescribed models.</li>
  </ul>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T11:46:53.893Z
Answers.LastActivityDate: 2023-01-13T11:46:53.893Z
Answers.UserDisplayName: Ciodar
Body: <p>I have a little perplexity trying to distinguish <em>parametric</em> vs <em>non-parametric</em> generative model.</p>
  <p>In my understanding, a <strong>parametric</strong> generative model would try to learn the probability density function by estimating the parameters of an underlying distribution we are assuming. So just doing for example,</p>
  <p><span class="math-container">$$\theta^* = arg\max_\theta  \,\prod_{i=1}^N p_\theta(\textbf{x}_i)$$</span></p>
  <p>I realize that in practice, we need to figure out what is the basic distribution that we are going to modify by adjusting the parameters <span class="math-container">$\theta$</span>. So in the case of <em>VAEs</em> we use <em>latent variables</em> assumption to make training feasible, we jointly train <span class="math-container">$q_\phi(\textbf{z}|\textbf{x})$</span> and <span class="math-container">$p_\theta(\textbf{x}|\textbf{z})$</span> prametrizing both distributions with neural networks (i.e. encoder &amp; decoder). In such case, we end up with the situation that all our distributions are <strong>Gaussians</strong> (assuming that prior and conditional are gaussians). So, having said that, can we conclude that VAEs are <strong>parametric</strong>? Also, what could be an example of <strong>non-parametric</strong> generative model?</p>
  <p>I would say that, for example, <em>GANs</em> maybe an example of non-parametric model, as we start with a latent normal distribution but then applying a stack of non-linear transformations ending up with something potentially very complicated.</p>
  
Tags: <generative-adversarial-networks><generative-model>
Title: Parametric vs Non-parametric generative models
UserDisplayName: James Arten
ThreadText: # Parametric vs Non-parametric generative models
  
  (Asked by: username_0 on 2022-07-12)
  
  I have a little perplexity trying to distinguish *parametric* vs *non-parametric* generative model.
  
  In my understanding, a **parametric** generative model would try to learn the probability density function by estimating the parameters of an underlying distribution we are assuming. So just doing for example,
  
  $$\theta^\* = arg\max\_\theta \,\prod\_{i=1}^N p\_\theta(\textbf{x}\_i)$$
  
  I realize that in practice, we need to figure out what is the basic distribution that we are going to modify by adjusting the parameters $\theta$. So in the case of *VAEs* we use *latent variables* assumption to make training feasible, we jointly train $q\_\phi(\textbf{z}|\textbf{x})$ and $p\_\theta(\textbf{x}|\textbf{z})$ prametrizing both distributions with neural networks (i.e. encoder & decoder). In such case, we end up with the situation that all our distributions are **Gaussians** (assuming that prior and conditional are gaussians). So, having said that, can we conclude that VAEs are **parametric**? Also, what could be an example of **non-parametric** generative model?
  
  I would say that, for example, *GANs* maybe an example of non-parametric model, as we start with a latent normal distribution but then applying a stack of non-linear transformations ending up with something potentially very complicated.
  
  **Tags:** generative-adversarial-networks, generative-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-13. Score: 0
  
  I call this difference **prescribed** vs **implicit** models.
  
  To my knowledge, a **parametric** model has a fixed,finite number of parameters with respect to the sample size (e.g a linear model), while **nonparametric** model have an increasing number of parameters with respect to data. See this [blog post](https://sebastianraschka.com/faq/docs/parametric_vs_nonparametric.html) and this [answer](https://stats.stackexchange.com/questions/268638/what-exactly-is-the-difference-between-a-parametric-and-non-parametric-model) for more precise info. Since most models use a Neural Network to generate an output, I prefer the terms prescribed vs implicit.
  
  Viewing this way,
  
  * **Implicit** models directly generate the output by passing a noise vector through a deterministic function (Generally a NN). They do not have a global likelihood, and are less limited in the architecture.
    A great example of an implicit model, as you stated, are GANs.
  * **Prescribed** models provides an explicit parametric specification of the output distribution $p(x)$. They can estimate the likelihood of data, but are more constrained in the architecture.
  
    + An Autoregressive model fits a Bernoulli distribution , learning a neural network to model $p(x\_i|x\_{
    + Normalizing flows learn a change of variable to reshape a gaussian source noise. They must learn an invertible function, so the architecture is heavily constrained.
    + VAEs assume a Gaussian prior $p(z) \sim \mathcal{N}(0,\mathbb{I})$ that is decoded by a NN which models $p(x|z)$. They have less limits in the architecture, but they lose the closed form likelihood, and they are trained by maximizing an ELBO. Since they still have (even if approximated) likelihood, they are prescribed models.
  
  
  ---
  
  