Answers.Body: <h3>What are generative (and discriminative) models?</h3>
  <p>If the model learns a distribution of the form <span class="math-container">$p(x)$</span> or <span class="math-container">$p(x, y)$</span>, where <span class="math-container">$x$</span> are the inputs and <span class="math-container">$y$</span> the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).</p>
  <p>Bishop also defines generative models in this way (<a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf#page=63" rel="nofollow noreferrer">p. 43</a>)</p>
  <blockquote>
  <p>Approaches that explicitly or implicitly model the distribution of
  inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space</p>
  </blockquote>
  <p>If it learns a distribution of the form <span class="math-container">$p(y \mid x)$</span>, then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses <em>implicitly</em> or <em>explicitly</em>).</p>
  <p>Bishop also defines discriminative models in this way (<a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf#page=63" rel="nofollow noreferrer">p. 43</a>)</p>
  <blockquote>
  <p>Approaches that model the posterior probabilities directly
  are called discriminative models</p>
  </blockquote>
  <p>The <a href="https://en.wikipedia.org/wiki/Generative_model" rel="nofollow noreferrer">related Wikipedia article</a> claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.</p>
  <h3>GPTs are autoregressive</h3>
  <p>As far as I know, GPTs are <a href="https://deepgenerativemodels.github.io/notes/autoregressive/" rel="nofollow noreferrer">autoregressive models</a>. <a href="https://ml.berkeley.edu/blog/posts/AR_intro/" rel="nofollow noreferrer">Here</a> is another potentially useful post that explains what autoregressive models are.</p>
  <p>My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the <a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf" rel="nofollow noreferrer">GPT-2 paper</a> seem to start the descriptions from the assumption that you can factorize some joint distribution like <span class="math-container">$p(x)$</span> into conditional distributions.</p>
  <p>ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are <a href="https://openai.com/blog/chatgpt/" rel="nofollow noreferrer">several steps involved</a> to create this model, so it may not be super clear how to categorise this model.</p>
  <p>Moreover, the authors of the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">transformer</a>, which GPT models are based on, claim that the transformer is an autoregressive model.</p>
  <h3>Conclusion</h3>
  <p>It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.</p>
  <p>I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form <span class="math-container">$p(x, y)$</span>, but they model some distribution of the form <span class="math-container">$p(x)$</span>, from which you can sample, otherwise, how could you even sample data (words)?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T21:50:41.270Z
Answers.Text: from the wikipedia article you linked to, a generative model must model $p(x,y)$ so why a model learns only $p(x)$ is also generative?
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T21:53:07.877Z
Answers.Text: You can define $x = z, y$, for example. Or $x = (x_1, x_2, x_3, ..., x_n)$, or $x = x_1$. In other words, you can have a distribution over many variables. In my answer, I actually say $p(x)$ or $p(x, y)$ in the first sentence
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-02T22:14:37.353Z
Answers.Text: I feel that is still a bit rhetorical... the way I understand it is, a generative model must be able to model $p(x,y) = p(y|x)p(x)$. So here $p(x)$ is not a joint but indeed a prior over $x$. In the case of VAE, it is clear to me how it models it (like you said in this case $x$ is $z$), but it is not clear to me how do GPT-series models model this distribution.
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T11:37:34.100Z
Answers.Text: @Sam Let's say you have $p(x, y) = p(y \mid x) p(x)$. Now, let $x = z, w$, so $p(x, y) = p(y \mid x) p(x) = p(y \mid z, w) p(z, w)$. Now, $p(z, w)$ is the prior, but it's a joint? Yes. You can do a similar thing for the evidence (denominator). Now, let $x = x_1$ and $y = x_2$, then $p(x, y) = p(x_1, x_2) = p(x_1 \mid x_2) p(x_2)$. Now, in the case of models like GPTs, you probably can model them as models that produce $x_{t+1}$ given previous tokens. So, in a way, they might model kind of a joint distribution over the current token and the previous ones.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T11:37:35.917Z
Answers.Text: To be honest, I don't know how they were originally formulated (I didn't have the chance to read the papers yet), but what I am trying to say is that $p(x)$ is not necessarily a distribution over just one variable. You can do some kind of algebra with these probabilities and changes of variables, etc.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-04T15:04:05.447Z
Answers.Text: they formulated it as $p(x) = \prod_{i=1}^{n} p(s_n|s_1, \ldots, s_{n-1})$. So I guess I can agree on if it indeed models $p(s_1)$ (the $p(x)$ in your example I guess) then it is indeed a generative model. But it is hard for me to see how it does that..
Answers.UserDisplayName: Sam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-06T20:48:02.293Z
Answers.Text: @nbro "Approaches that explicitly or **implicitly** model". What is an implicit model of a distribution $p(x)$? A model that $q$ simply generates data looking similar to that of $p(x)$ but can't be written as function $q_{\theta}(x)$?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T13:38:58.943Z
Answers.LastActivityDate: 2023-02-03T13:44:06.157Z
Answers.LastEditDate: 2023-02-03T13:44:06.157Z
Answers.UserDisplayName: nbro
Answers.Body: <p>They both refer to the same type of models. However, the second definition is a more 'intuitive' explanation of what generative AI <em>does</em>, while the first is a definition that refers more to what a generative model <em>is</em>.</p>
  <p>To generate new data similar to some training data (definition 2), a model needs to learn the training data distributions (definition 1). Only if the model has learned that distribution it can use that distribution to sample (generate) new data from that distribution.</p>
  <p>During training, ChatGPT also learned the distribution of the training data that OpenAI provided the model with. After training, the model simply takes in the input and uses the input to sample from the learned distribution to generate an output. So ChatGPT also follows both of your definitions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T09:16:59.060Z
Answers.Text: is it the distribution over, "what word is most probable when seeing this part of the sentence"? Or how would you discribe over what the distribution is learned?
Answers.UserDisplayName: Ai4l2s
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T10:00:30.393Z
Answers.Text: Its exactly that for large language models (LLMs)
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T11:26:00.907Z
Answers.Text: is a Vanilla RNN that predicts time series data then also considered as a generative model?
Answers.UserDisplayName: Ai4l2s
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T12:24:33.177Z
Answers.Text: Prediction models can be seen as a broader category of models of which generative models could be a 'subgroup'. Prediction models can also 'predict the class' of some data instance. I guess ChatGPT can be seen as a prediction model as well to some extend.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T12:25:03.730Z
Answers.Text: An RNN that predicts a time series can be seen as 'generating the future of the time-series'. Definitions of model-groups are never clear cut.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T09:11:59.360Z
Answers.LastActivityDate: 2023-02-02T09:11:59.360Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>Well, I also stumbled upon this question. Models like GPTs blurs the lines between Generative and Discriminative models.
  According to standard definition of Generative models,
  <a href="https://i.sstatic.net/AJ0wwkJ8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AJ0wwkJ8.png" alt="Screenshot from Wikipedia" /></a></p>
  <p>So, a generative model must map joint probability distribution P(X,Y) or probability distribution of data itself P(X). In case of GPT, it outputs conditional distribution for next word, given all the previous outputs and input,</p>
  <blockquote>
  <p><strong>P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1)</strong> (at a time)</p>
  </blockquote>
  <p>which obviously looks like a discriminative model. But GPTs predicts next word in Autoregressive manner. i.e.</p>
  <blockquote>
  <p><strong>P(Y1,Y2,....,Yn|Xm,Xm-1,....X1)</strong> = P(Y1|Xm, Xm-1,....X1) * P(Y2|Y1,Xm, Xm-1,....X1) * ... * P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1) (collectively)</p>
  </blockquote>
  <p>So, collectively, it acts like a generative model.
  So, it is safe to say that GPTs and other similar autoregressive models are Generative models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-20T14:32:08.303Z
Answers.Text: Please, edit your answer to format the symbols with mathjax. Also, don't provide screenshots, but copy and paste the text and cite it using `>`
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-08T12:04:27.863Z
Answers.LastActivityDate: 2024-10-08T12:04:27.863Z
Answers.UserDisplayName: Parag Londhe
Body: <p>I'm working my way through how ChatGPT works. So I read that ChatGPT is a generative model. When searching for generative models, I found two defintions:</p>
  <ul>
  <li><a href="https://developers.google.com/machine-learning/gan/generative" rel="nofollow noreferrer">A <strong>generative</strong> model includes the distribution of the data itself, and tells you how likely a given example is</a> by Google</li>
  <li><a href="https://www.mdpi.com/2227-7102/14/2/172" rel="nofollow noreferrer"><strong>Generative</strong> artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content</a></li>
  </ul>
  <p>Do they both mean the same? That is, for generating new content, a model must learn the distribution of data itself? Or do we call ChatGPT generative because it just generates new text?</p>
  <p>I see that ChatGPT is something other than a discriminative model that learns a boundary to split data, however, I can not bring ChatGPT in line with a more traditional generative model like naive Bayes, where class distributions are inferred.</p>
  
Tags: <machine-learning><definitions><generative-model><chatgpt>
Title: What makes ChatGPT a generative model?
UserDisplayName: Ai4l2s
ThreadText: # What makes ChatGPT a generative model?
  
  (Asked by: username_0 on 2023-02-02)
  
  I'm working my way through how ChatGPT works. So I read that ChatGPT is a generative model. When searching for generative models, I found two defintions:
  
  * [A **generative** model includes the distribution of the data itself, and tells you how likely a given example is](https://developers.google.com/machine-learning/gan/generative) by Google
  * [**Generative** artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content](https://www.mdpi.com/2227-7102/14/2/172)
  
  Do they both mean the same? That is, for generating new content, a model must learn the distribution of data itself? Or do we call ChatGPT generative because it just generates new text?
  
  I see that ChatGPT is something other than a discriminative model that learns a boundary to split data, however, I can not bring ChatGPT in line with a more traditional generative model like naive Bayes, where class distributions are inferred.
  
  **Tags:** machine-learning, definitions, generative-model, chatgpt
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-02-03. Score: 4
  
  ### What are generative (and discriminative) models?
  
  If the model learns a distribution of the form $p(x)$ or $p(x, y)$, where $x$ are the inputs and $y$ the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).
  
  Bishop also defines generative models in this way ([p. 43](https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf#page=63))
  
  > Approaches that explicitly or implicitly model the distribution of
  > inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space
  
  If it learns a distribution of the form $p(y \mid x)$, then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses *implicitly* or *explicitly*).
  
  Bishop also defines discriminative models in this way ([p. 43](https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf#page=63))
  
  > Approaches that model the posterior probabilities directly
  > are called discriminative models
  
  The [related Wikipedia article](https://en.wikipedia.org/wiki/Generative_model) claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.
  
  ### GPTs are autoregressive
  
  As far as I know, GPTs are [autoregressive models](https://deepgenerativemodels.github.io/notes/autoregressive/). [Here](https://ml.berkeley.edu/blog/posts/AR_intro/) is another potentially useful post that explains what autoregressive models are.
  
  My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) seem to start the descriptions from the assumption that you can factorize some joint distribution like $p(x)$ into conditional distributions.
  
  ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are [several steps involved](https://openai.com/blog/chatgpt/) to create this model, so it may not be super clear how to categorise this model.
  
  Moreover, the authors of the [transformer](https://arxiv.org/pdf/1706.03762.pdf), which GPT models are based on, claim that the transformer is an autoregressive model.
  
  ### Conclusion
  
  It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.
  
  I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form $p(x, y)$, but they model some distribution of the form $p(x)$, from which you can sample, otherwise, how could you even sample data (words)?
  
  > **Comments:**
  > **username_2**: from the wikipedia article you linked to, a generative model must model $p(x,y)$ so why a model learns only $p(x)$ is also generative?  
  > **username_1**: You can define $x = z, y$, for example. Or $x = (x_1, x_2, x_3, ..., x_n)$, or $x = x_1$. In other words, you can have a distribution over many variables. In my answer, I actually say $p(x)$ or $p(x, y)$ in the first sentence  
  > **username_2**: I feel that is still a bit rhetorical... the way I understand it is, a generative model must be able to model $p(x,y) = p(y|x)p(x)$. So here $p(x)$ is not a joint but indeed a prior over $x$. In the case of VAE, it is clear to me how it models it (like you said in this case $x$ is $z$), but it is not clear to me how do GPT-series models model this distribution.  
  > **username_1**: @username_2 Let's say you have $p(x, y) = p(y \mid x) p(x)$. Now, let $x = z, w$, so $p(x, y) = p(y \mid x) p(x) = p(y \mid z, w) p(z, w)$. Now, $p(z, w)$ is the prior, but it's a joint? Yes. You can do a similar thing for the evidence (denominator). Now, let $x = x_1$ and $y = x_2$, then $p(x, y) = p(x_1, x_2) = p(x_1 \mid x_2) p(x_2)$. Now, in the case of models like GPTs, you probably can model them as models that produce $x_{t+1}$ given previous tokens. So, in a way, they might model kind of a joint distribution over the current token and the previous ones.  
  > **username_1**: To be honest, I don't know how they were originally formulated (I didn't have the chance to read the papers yet), but what I am trying to say is that $p(x)$ is not necessarily a distribution over just one variable. You can do some kind of algebra with these probabilities and changes of variables, etc.  
  > **username_2**: they formulated it as $p(x) = \prod_{i=1}^{n} p(s_n|s_1, \ldots, s_{n-1})$. So I guess I can agree on if it indeed models $p(s_1)$ (the $p(x)$ in your example I guess) then it is indeed a generative model. But it is hard for me to see how it does that..  
  > **username_3**: @username_1 "Approaches that explicitly or **implicitly** model". What is an implicit model of a distribution $p(x)$? A model that $q$ simply generates data looking similar to that of $p(x)$ but can't be written as function $q_{\theta}(x)$?  
  
  ---
  
  ### Answer by username_4 on 2023-02-02. Score: 2
  
  They both refer to the same type of models. However, the second definition is a more 'intuitive' explanation of what generative AI *does*, while the first is a definition that refers more to what a generative model *is*.
  
  To generate new data similar to some training data (definition 2), a model needs to learn the training data distributions (definition 1). Only if the model has learned that distribution it can use that distribution to sample (generate) new data from that distribution.
  
  During training, ChatGPT also learned the distribution of the training data that OpenAI provided the model with. After training, the model simply takes in the input and uses the input to sample from the learned distribution to generate an output. So ChatGPT also follows both of your definitions.
  
  > **Comments:**
  > **username_0**: is it the distribution over, "what word is most probable when seeing this part of the sentence"? Or how would you discribe over what the distribution is learned?  
  > **username_4**: Its exactly that for large language models (LLMs)  
  > **username_0**: is a Vanilla RNN that predicts time series data then also considered as a generative model?  
  > **username_4**: Prediction models can be seen as a broader category of models of which generative models could be a 'subgroup'. Prediction models can also 'predict the class' of some data instance. I guess ChatGPT can be seen as a prediction model as well to some extend.  
  > **username_4**: An RNN that predicts a time series can be seen as 'generating the future of the time-series'. Definitions of model-groups are never clear cut.  
  
  ---
  
  ### Answer by username_5 on 2024-10-08. Score: -1
  
  Well, I also stumbled upon this question. Models like GPTs blurs the lines between Generative and Discriminative models.
  According to standard definition of Generative models,
  [![Screenshot from Wikipedia](https://i.sstatic.net/AJ0wwkJ8.png)](https://i.sstatic.net/AJ0wwkJ8.png)
  
  So, a generative model must map joint probability distribution P(X,Y) or probability distribution of data itself P(X). In case of GPT, it outputs conditional distribution for next word, given all the previous outputs and input,
  
  > **P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1)** (at a time)
  
  which obviously looks like a discriminative model. But GPTs predicts next word in Autoregressive manner. i.e.
  
  > **P(Y1,Y2,....,Yn|Xm,Xm-1,....X1)** = P(Y1|Xm, Xm-1,....X1) \* P(Y2|Y1,Xm, Xm-1,....X1) \* ... \* P(Yn|Yn-1,Yn-2....Y1,Xm,Xm-1,....X1) (collectively)
  
  So, collectively, it acts like a generative model.
  So, it is safe to say that GPTs and other similar autoregressive models are Generative models.
  
  > **Comments:**
  > **username_1**: Please, edit your answer to format the symbols with mathjax. Also, don't provide screenshots, but copy and paste the text and cite it using `>`  
  
  ---
  
  