Answers.Body: <p>From the paper <a href="https://arxiv.org/pdf/2211.09110.pdf" rel="noreferrer">Language Models are Changing AI: The Need for Holistic Evaluation</a> (Authors: <a href="https://rishibommasani.github.io/" rel="noreferrer">Rishi Bommasani</a> and <a href="https://cs.stanford.edu/%7Epliang/" rel="noreferrer">Percy Liang</a> and <a href="https://www.linkedin.com/in/tonyhlee" rel="noreferrer">Tony Lee</a>; <a href="https://crfm.stanford.edu/helm" rel="noreferrer">Website</a>):</p>
  <blockquote>
  <p>Memorization of copyrighted/licensed material. We find that the likelihood of direct
  regurgitation of long copyrighted sequences is somewhat uncommon, but it does become
  noticeable when looking at popular books. However, we do find the regurgitation risk clearly
  correlates with model accuracy: InstructGPT davinci v2 (175B*), GPT-3 davinci v1 (175B),
  and Anthropic-LM v4-s3 (52B) demonstrate <strong>the highest amount of verbatim regurgitation in
  line with their high accuracies.</strong></p>
  <p>[...]</p>
  <p>To further explore the results for this targeted evaluation, see
  <a href="https://crfm.stanford.edu/helm/v1.0/?group=copyright_text" rel="noreferrer">https://crfm.stanford.edu/helm/v1.0/?group=copyright_text</a> , <a href="https://crfm.stanford.edu/helm/v1.0/?group=copyright_code" rel="noreferrer">https://crfm.stanford.edu/helm/v1.0/?group=copyright_code</a> and Figure 39. We evaluated various models for their ability to reproduce
  copyrighted text or licensed code. When evaluating source code regurgitation, we only extract
  from models specialized to code (Codex davinci v2 and Codex cushman v1). When evaluating text
  regurgitation, we extract from all models except those specialized to code.
  Overall, we find that models only regurgitate infrequently, with most models not regurgitating at
  all under our evaluation setup. However, in the rare occasion where models regurgitate, large spans
  of verbatim content are reproduced.</p>
  </blockquote>
  <p>ChatGPT shouldn't be too far away from InstructGPT davinci v2.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T01:06:02.967Z
Answers.LastActivityDate: 2023-03-14T01:06:02.967Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>One of the main concerns of using ChatGPT answers on Stack Exchange is that it may copy verbatim or almost verbatim some text from its training set, which may infringe the source text's license. This makes me wonder how much of the ChatGPT output is copied from its training set (vs. being abstractively generated).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T12:55:31.853Z
Comments.Text: Is there any evidence for this "it may copy verbatim or almost verbatim some text from its training set"? This may be true, but I am wondering if there is any evidence. I never tried it, so I don't know.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T13:23:46.063Z
Comments.Text: I think this is incorrect.  The real problem is that it's generating text based on the probability that a certain token appears in a given position given all of the other tokens so far...  which means that the result might be interesting, but isn't necessarily anything like accurate.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T14:04:26.450Z
Comments.Text: @nbro I don't know any. I haven't tried much.
Comments.UserDisplayName: Franck Dernoncourt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T14:04:52.850Z
Comments.Text: @DavidHoelzer that's indeed another concern. https://meta.stackexchange.com/q/384410/178179 ; https://meta.stackexchange.com/q/384652/178179
Comments.UserDisplayName: Franck Dernoncourt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-22T10:25:54.480Z
Comments.Text: This is impossible to answer since the training set is not public.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-22T10:37:29.770Z
Comments.Text: @Dr.Snoopy isn't most of it in common crawl or queryable via Google?
Comments.UserDisplayName: Franck Dernoncourt
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-22T10:38:54.067Z
Comments.Text: No idea, since there is no information, that would be making an assumption.
Comments.UserDisplayName: Dr. Snoopy
Tags: <text-generation><chatgpt>
Title: How much of the ChatGPT output is copied from its training set (vs. being abstractively generated)?
UserDisplayName: Franck Dernoncourt
ThreadText: # How much of the ChatGPT output is copied from its training set (vs. being abstractively generated)?
  
  (Asked by: username_0 on 2022-12-21)
  
  One of the main concerns of using ChatGPT answers on Stack Exchange is that it may copy verbatim or almost verbatim some text from its training set, which may infringe the source text's license. This makes me wonder how much of the ChatGPT output is copied from its training set (vs. being abstractively generated).
  
  **Tags:** text-generation, chatgpt
  
  > **Comments:**
  > **username_1**: Is there any evidence for this "it may copy verbatim or almost verbatim some text from its training set"? This may be true, but I am wondering if there is any evidence. I never tried it, so I don't know.  
  > **username_2**: I think this is incorrect.  The real problem is that it's generating text based on the probability that a certain token appears in a given position given all of the other tokens so far...  which means that the result might be interesting, but isn't necessarily anything like accurate.  
  > **username_0**: @username_1 I don't know any. I haven't tried much.  
  > **username_0**: @username_2 that's indeed another concern. https://meta.stackexchange.com/q/384410/178179 ; https://meta.stackexchange.com/q/384652/178179  
  > **username_3**: This is impossible to answer since the training set is not public.  
  > **username_0**: @username_3 isn't most of it in common crawl or queryable via Google?  
  > **username_3**: No idea, since there is no information, that would be making an assumption.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-03-14. Score: 6
  
  From the paper [Language Models are Changing AI: The Need for Holistic Evaluation](https://arxiv.org/pdf/2211.09110.pdf) (Authors: [Rishi Bommasani](https://rishibommasani.github.io/) and [Percy Liang](https://cs.stanford.edu/%7Epliang/) and [Tony Lee](https://www.linkedin.com/in/tonyhlee); [Website](https://crfm.stanford.edu/helm)):
  
  > Memorization of copyrighted/licensed material. We find that the likelihood of direct
  > regurgitation of long copyrighted sequences is somewhat uncommon, but it does become
  > noticeable when looking at popular books. However, we do find the regurgitation risk clearly
  > correlates with model accuracy: InstructGPT davinci v2 (175B\*), GPT-3 davinci v1 (175B),
  > and Anthropic-LM v4-s3 (52B) demonstrate **the highest amount of verbatim regurgitation in
  > line with their high accuracies.**
  >
  > [...]
  >
  > To further explore the results for this targeted evaluation, see
  > <https://crfm.stanford.edu/helm/v1.0/?group=copyright_text> , <https://crfm.stanford.edu/helm/v1.0/?group=copyright_code> and Figure 39. We evaluated various models for their ability to reproduce
  > copyrighted text or licensed code. When evaluating source code regurgitation, we only extract
  > from models specialized to code (Codex davinci v2 and Codex cushman v1). When evaluating text
  > regurgitation, we extract from all models except those specialized to code.
  > Overall, we find that models only regurgitate infrequently, with most models not regurgitating at
  > all under our evaluation setup. However, in the rare occasion where models regurgitate, large spans
  > of verbatim content are reproduced.
  
  ChatGPT shouldn't be too far away from InstructGPT davinci v2.
  
  
  ---
  
  