Answers.Body: <p><a href="https://openai.com/blog/chatgpt/" rel="noreferrer">ChatGPT</a> is in the <a href="https://huggingface.co/blog/large-language-models" rel="noreferrer">Large Language Models (LLM)</a> category. The most (in)famous GPT model is probably <a href="https://en.wikipedia.org/wiki/GPT-3" rel="noreferrer">GPT-3</a>, because since then, researchers realized that LLMs mostly follow a <a href="https://arxiv.org/abs/2001.08361v1" rel="noreferrer">predictable scaling law</a>, thus the more data and the bigger model, the better. It is accurate to say that ChatGPT was <strong>trained</strong> with Stack Overflow data, but it should be <em>all</em> Stack Overflow instead of <em>just most upvoted answers/comments</em>.</p>
  <p>The <a href="https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilities" rel="noreferrer">Wikipedia page of GPT-3</a> and <a href="https://arxiv.org/pdf/2005.14165.pdf" rel="noreferrer">their paper</a> mentions that GPT-3 was trained on multiple datasets, and one of which is the <strong>Common Crawl</strong>, which basically <a href="https://en.wikipedia.org/wiki/Common_Crawl" rel="noreferrer">crawls everything on the Internet</a>. Some data pre-processing was done before training, but the authors did not mention removing the comments, so we can say that it is all Stack Overflow data.</p>
  <p>If we <a href="https://commoncrawl.org/2022/09/host-and-domain-level-web-graphs-may-jun-aug-2022/" rel="noreferrer">look at the Common Crawl data in Sep 2022</a>, there is indeed the domain <code>com.stackoverflow</code> in their list. Thus, <strong>while ChatGPT was trained on Stack Overflow data, it is trained on all Stack Overflow data instead of just most upvoted answers</strong>.</p>
  <p>However, if you think ChatGPT's code output is of high quality, think again, because <a href="https://meta.stackoverflow.com/questions/421831/temporary-policy-chatgpt-is-banned">Stack Overflow temporarily bans ChatGPT</a> <code>because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers</code> (quoted from the link).</p>
  <p>Here is the justification of them:</p>
  <blockquote>
  <p>The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting. Because such answers are so easy to produce, a large number of people are posting a lot of answers. The volume of these answers (thousands) and the fact that the answers often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer is actually bad has effectively swamped our volunteer-based quality curation infrastructure.</p>
  </blockquote>
  <p>EDIT: a comment below also confirmed that GPT-2, the predecessor of GPT-3 and ChatGPT, was trained with Stack Overflow data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T14:32:06.893Z
Answers.Text: How do you know it was trained on all data? The crawler might have accessed all data. But how do we know this data hasn't been preprocessed? E.g. remove comments etc.
Answers.UserDisplayName: Thomas Weller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T17:02:46.110Z
Answers.Text: @ThomasWeller See Appendix A of the paper. I don't think they removed comments.
Answers.UserDisplayName: wizzwizz4
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T00:39:20.713Z
Answers.Text: Note that *because the average rate of getting correct answers from ChatGPT is too low* is not sourced in the link you've provided.
Answers.UserDisplayName: JonathanReez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T09:55:44.647Z
Answers.Text: I would argue that (at least in the general population), ChatGPT is now the most famous GPT model, by quite a large margin.
Answers.UserDisplayName: Arthur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T11:21:48.833Z
Answers.Text: @Arthur yes, I would agree so, but I believe it is because ChatGPT is now publicly available, while GPT-3 was not (and is still not), making it extremely unpopular compared to ChatGPT. [If only](https://blog.eleuther.ai/why-release-a-large-language-model/) it was publicly released like [GPT-NeoX](https://github.com/EleutherAI/gpt-neox), it could have the [fame of NeoX](https://blog.eleuther.ai/year-one/#gpt-neo-and-gpt-j) and dominated the world's press
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T01:39:32.270Z
Answers.Text: It is also known that GPT2 was trained on [Stack Exchange](https://github.com/openai/gpt-2/blob/master/domains.txt#L106), and [Stack Overflow](https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/domains.txt#L131).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T13:33:38.113Z
Answers.LastActivityDate: 2023-01-15T15:15:45.367Z
Answers.LastEditDate: 2023-01-15T15:15:45.367Z
Answers.UserDisplayName: Minh-Long Luu
Answers.Body: <p>Technically speaking, we don't know whether ChatGPT used Stack Overflow data for the training.
  That is because ChatGPT is a proprietary model, and OpenAI hasn't published training details to the public.</p>
  <p>We know that, from the GPT-3 paper [1], the GPT-3 training dataset includes Common Crawl. However, there is no indication of the relation between GPT-3.5 series (including ChatGPT) and GPT-3 models according to <a href="https://platform.openai.com/docs/model-index-for-researchers" rel="nofollow noreferrer">OpenAI's account</a>.</p>
  <p>See also this question <a href="https://ai.stackexchange.com/questions/39023/are-gpt-3-5-series-models-based-on-gpt-3">Are GPT-3.5 series models based on GPT-3?</a> for the discussion on GPT-3 vs. GPT-3.5.
  I add: the token set is different between <code>davinci</code> (GPT-3) and <code>text-davinci-003</code> (GPT-3.5) (<a href="https://github.com/openai/tiktoken/blob/e1c661edf3604706bb2db59cfc7bf92f73c09761/tiktoken/model.py#L13" rel="nofollow noreferrer">source</a>) thus the later cannot be a simple fine-tuning of the former.</p>
  <p>I <em>think</em>, though, it is natural to infer OpenAI uses StackOverflow data because it is generally considered high quality [2] and easily obtainable. Also, if OpenAI ever uses web crawling data, it will see many StackOverflow and its clone sites.</p>
  <ul>
  <li>[1]: T Brown et al. &quot;<a href="https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf" rel="nofollow noreferrer">Language Models are Few-Shot Learners</a>.&quot; <em>NeurIPS</em> (2020).</li>
  <li>[2]: Guilherme Penedo et al. &quot;<a href="https://arxiv.org/abs/2306.01116" rel="nofollow noreferrer">The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only</a>.&quot; arXiv preprint (2023).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-10T04:47:25.673Z
Answers.LastActivityDate: 2023-07-10T04:47:25.673Z
Answers.UserDisplayName: pcpthm
Answers.Body: <p>it was trained using supervised from open ai team for initializing the conversational model and for more customize style, it can highly possible using chosen stackoverflow sample dataset since the answer of the question from the website was moderated. ChatGPT was take the advantage of GPT-3 model that has been trained from various corpus of data and can perform few-shot learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-12T15:32:10.763Z
Answers.Text: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-10T03:58:34.240Z
Answers.LastActivityDate: 2023-04-10T03:58:34.240Z
Answers.UserDisplayName: gustin hrat 
Body: <p>Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data?</p>
  <p>For me it makes complete sense to take answers that have upwards of 100 upvotes and include them in your training data, but people around me seem to think this hypothesis doesn't make sense. Is there a way to confirm this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-13T10:47:22.490Z
Comments.Text: If the responses are often condescending or unhelpful then 'yes'
Comments.UserDisplayName: adolf garlic 
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-10T16:30:27.537Z
Comments.Text: It's worth noting that even with the option you suggest, the answers produced by ChatGPT wouldn't be more correct. Maybe they would be slightly better written, but this is not the issue.
Comments.UserDisplayName: Erwan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-11T09:56:59.970Z
Comments.Text: Think about the attrition this inbreeding will cause to the quality of SO answers.. People ask ChatGPT for solutions which come in part from SO, then they post the answers to SO, ChatGPT learns from these answers but doesn't get them perfect, so the next time someone asks ChatGPT for a similar answer, the result is a little more incorrect, and is then again posted to SO..
Comments.UserDisplayName: tplive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-11T18:28:02.403Z
Comments.Text: @tplive. That's exactly the reason SO has a feedback system. In my experience, the answers supposedly knowledgeable people post are often no walk in the park.
Comments.UserDisplayName: Mad Physicist
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-24T09:40:44.457Z
Comments.Text: It certainly didn't learn from stackexchange to apologize for its mistakes.
Comments.UserDisplayName: user11235
Tags: <chat-bots><chatgpt><language-model>
Title: Was ChatGPT trained on Stack Overflow data?
UserDisplayName: Nicolas Zein
ThreadText: # Was ChatGPT trained on Stack Overflow data?
  
  (Asked by: username_0 on 2023-01-08)
  
  Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data?
  
  For me it makes complete sense to take answers that have upwards of 100 upvotes and include them in your training data, but people around me seem to think this hypothesis doesn't make sense. Is there a way to confirm this?
  
  **Tags:** chat-bots, chatgpt, language-model
  > **Comments:**
  > **username_2**: It's worth noting that even with the option you suggest, the answers produced by ChatGPT wouldn't be more correct. Maybe they would be slightly better written, but this is not the issue.  
  > **username_3**: Think about the attrition this inbreeding will cause to the quality of SO answers.. People ask ChatGPT for solutions which come in part from SO, then they post the answers to SO, ChatGPT learns from these answers but doesn't get them perfect, so the next time someone asks ChatGPT for a similar answer, the result is a little more incorrect, and is then again posted to SO..  
  > **username_4**: @tplive. That's exactly the reason SO has a feedback system. In my experience, the answers supposedly knowledgeable people post are often no walk in the park.  
  > **username_5**: It certainly didn't learn from stackexchange to apologize for its mistakes.  
  > **username_1**: If the responses are often condescending or unhelpful then 'yes'  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_6 on 2023-01-08. Score: 59
  
  [ChatGPT](https://openai.com/blog/chatgpt/) is in the [Large Language Models (LLM)](https://huggingface.co/blog/large-language-models) category. The most (in)famous GPT model is probably [GPT-3](https://en.wikipedia.org/wiki/GPT-3), because since then, researchers realized that LLMs mostly follow a [predictable scaling law](https://arxiv.org/abs/2001.08361v1), thus the more data and the bigger model, the better. It is accurate to say that ChatGPT was **trained** with Stack Overflow data, but it should be *all* Stack Overflow instead of *just most upvoted answers/comments*.
  
  The [Wikipedia page of GPT-3](https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilities) and [their paper](https://arxiv.org/pdf/2005.14165.pdf) mentions that GPT-3 was trained on multiple datasets, and one of which is the **Common Crawl**, which basically [crawls everything on the Internet](https://en.wikipedia.org/wiki/Common_Crawl). Some data pre-processing was done before training, but the authors did not mention removing the comments, so we can say that it is all Stack Overflow data.
  
  If we [look at the Common Crawl data in Sep 2022](https://commoncrawl.org/2022/09/host-and-domain-level-web-graphs-may-jun-aug-2022/), there is indeed the domain `com.stackoverflow` in their list. Thus, **while ChatGPT was trained on Stack Overflow data, it is trained on all Stack Overflow data instead of just most upvoted answers**.
  
  However, if you think ChatGPT's code output is of high quality, think again, because [Stack Overflow temporarily bans ChatGPT](https://meta.stackoverflow.com/questions/421831/temporary-policy-chatgpt-is-banned) `because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers` (quoted from the link).
  
  Here is the justification of them:
  
  > The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting. Because such answers are so easy to produce, a large number of people are posting a lot of answers. The volume of these answers (thousands) and the fact that the answers often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer is actually bad has effectively swamped our volunteer-based quality curation infrastructure.
  
  EDIT: a comment below also confirmed that GPT-2, the predecessor of GPT-3 and ChatGPT, was trained with Stack Overflow data.
  
  > **Comments:**
  > **username_7**: How do you know it was trained on all data? The crawler might have accessed all data. But how do we know this data hasn't been preprocessed? E.g. remove comments etc.  
  > **username_8**: @ThomasWeller See Appendix A of the paper. I don't think they removed comments.  
  > **username_9**: Note that *because the average rate of getting correct answers from ChatGPT is too low* is not sourced in the link you've provided.  
  > **username_10**: I would argue that (at least in the general population), ChatGPT is now the most famous GPT model, by quite a large margin.  
  > **username_6**: @Arthur yes, I would agree so, but I believe it is because ChatGPT is now publicly available, while GPT-3 was not (and is still not), making it extremely unpopular compared to ChatGPT. [If only](https://blog.eleuther.ai/why-release-a-large-language-model/) it was publicly released like [GPT-NeoX](https://github.com/EleutherAI/gpt-neox), it could have the [fame of NeoX](https://blog.eleuther.ai/year-one/#gpt-neo-and-gpt-j) and dominated the world's press  
  > **username_11**: It is also known that GPT2 was trained on [Stack Exchange](https://github.com/openai/gpt-2/blob/master/domains.txt#L106), and [Stack Overflow](https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/domains.txt#L131).  
  
  ---
  
  ### Answer by username_12 on 2023-07-10. Score: 2
  
  Technically speaking, we don't know whether ChatGPT used Stack Overflow data for the training.
  That is because ChatGPT is a proprietary model, and OpenAI hasn't published training details to the public.
  
  We know that, from the GPT-3 paper [1], the GPT-3 training dataset includes Common Crawl. However, there is no indication of the relation between GPT-3.5 series (including ChatGPT) and GPT-3 models according to [OpenAI's account](https://platform.openai.com/docs/model-index-for-researchers).
  
  See also this question [Are GPT-3.5 series models based on GPT-3?](https://ai.stackexchange.com/questions/39023/are-gpt-3-5-series-models-based-on-gpt-3) for the discussion on GPT-3 vs. GPT-3.5.
  I add: the token set is different between `davinci` (GPT-3) and `text-davinci-003` (GPT-3.5) ([source](https://github.com/openai/tiktoken/blob/e1c661edf3604706bb2db59cfc7bf92f73c09761/tiktoken/model.py#L13)) thus the later cannot be a simple fine-tuning of the former.
  
  I *think*, though, it is natural to infer OpenAI uses StackOverflow data because it is generally considered high quality [2] and easily obtainable. Also, if OpenAI ever uses web crawling data, it will see many StackOverflow and its clone sites.
  
  * [1]: T Brown et al. "[Language Models are Few-Shot Learners](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)." *NeurIPS* (2020).
  * [2]: Guilherme Penedo et al. "[The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://arxiv.org/abs/2306.01116)." arXiv preprint (2023).
  
  
  ---
  
  ### Answer by username_13 on 2023-04-10. Score: 0
  
  it was trained using supervised from open ai team for initializing the conversational model and for more customize style, it can highly possible using chosen stackoverflow sample dataset since the answer of the question from the website was moderated. ChatGPT was take the advantage of GPT-3 model that has been trained from various corpus of data and can perform few-shot learning.
  
  > **Comments:**
  > **username_14**: As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  