Answers.Body: <p>Google &quot;call me ishmael. some years ago—never mind how long precisely—having&quot; and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, hence your observation. Note that some LLM training datasets do try to deduplicate data, e.g. <a href="https://huggingface.co/datasets/uonlp/CulturaX" rel="noreferrer">https://huggingface.co/datasets/uonlp/CulturaX</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-04T20:04:08.603Z
Answers.LastActivityDate: 2024-01-04T20:04:08.603Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the &quot;decompression&quot; of parts to be recognizable and reasonably faithful. I don't think there's a really good understanding of which parts are recoverable to this extent. Naively one might think it's only very common material that appears repeatedly in the corpus, but they've also been found to reproduce exact or near-exact text for single-instance personal information. I'm not familiar with it but I suspect this is an exciting research area.</p>
  <p>&quot;The probable outcome given some prompt&quot; is one way of looking at what is computationally being <em>done with the model</em>, but it's important to realize that this assumes a probability model, which was derived from the training corpus. It might help as an exercise to imagine doing the model with a naive Markov chain instead, where it's mentally tractable to see how this kind of text can be encoded reproducibly in the probability model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-05T23:23:38.193Z
Answers.Text: If the description of GPT-4 in the New York Times lawsuit is correct, it's not even always *lossy* compression. GPT-4 apparently has enough parameters to losslessly encode a complete copy of its training data at a 10:1 compression ratio, while still having several hundred billion parameters left over.
Answers.UserDisplayName: Mark
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-06T08:20:32.483Z
Answers.Text: Those remaining parameters could allow for alternate paths in the stored text chain to provide variety and context matching. I got an amazing response once: "translation provided by [real company name here]". It looked like a direct leak of training data. It came as a response of a failed input, some breath in speech input to ChatGPT that was transcribed as empty text.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-06T10:09:48.377Z
Answers.Text: Years ago, a technique for text to speech was "unit selection". A raw speech database was labeled and optimum path chosen according to phonetic transcription and prosody. A similar approach could be thought of for text generation.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-06T19:09:49.097Z
Answers.Text: @JaumeOliverLafont Note: such responses are quite common from non-Chat GPTs, since they are only playing a game of "predict the next word". I don't know - and perhaps nobody outside of OpenAI knows - exactly what special sauce they wrapped around a predict-the-next-word game to turn it into a good question and answer machine, but it's not crazy that sometimes you see some of its inner nature peek through.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-06T19:48:07.107Z
Answers.Text: @user253751: My naive understanding is that they do some transform to the question/prompt text to make it into a prior context for the "predict the next word game".
Answers.UserDisplayName: R.. GitHub STOP HELPING ICE
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T15:27:09.023Z
Answers.Text: ("ICE" is ***heavily*** overloaded, but it probably refers to [U.S. Immigration and Customs Enforcement](https://en.wikipedia.org/wiki/U.S._Immigration_and_Customs_Enforcement).)
Answers.UserDisplayName: Peter Mortensen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-07T23:07:47.933Z
Answers.Text: I'm not sure I agree with the dismissal of the probabilistic interpretation here in favor of the compression interpretation. Probability in this case is a way of _encoding information_. If certain sequences of inputs result in very high probabilities of specific words, that's because knowing that sequence tells you a lot about the next word to come. I would argue that probability and information are in fact the fundamental operating principles that allow these models to learn anything at all.
Answers.UserDisplayName: shadowtalker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-05T13:36:18.430Z
Answers.LastActivityDate: 2024-01-05T13:36:18.430Z
Answers.UserDisplayName: R.. GitHub STOP HELPING ICE
Answers.Body: <p>When we create and train neural networks, the goal is to get them to model a <em>general representation</em> of the input data, so that they will produce the desired output for inputs that are <em>generally similar</em> to inputs that were in the training data.</p>
  <p>But if your network has too many parameters, it doesn't need to generalize during the training stage. It might instead use those parameters to instead store representations of the input data and generate the desired output <em>precisely</em> - basically storing input/output pairs for everything input/output pair that's present in the training data. That won't necessarily happen, but it can happen. If you give it enough parameters, and train it long enough, it probably will happen.</p>
  <p>And then you feed it something that wasn't in the training set, and it produces garbage because it didn't generalize at all, it just encoded the inputs and outputs that it was trained with.</p>
  <p>This is one of the reasons that it's a good idea to train with one set of input/output pairs, and test with another set. If the network is generalizing, it will do a pretty good job with the test set even though those input/output pairs aren't part of the training set. But if the network is memorizing, it will do a pretty terrible job with the test set, because those input/output pairs were not memorized.</p>
  <p>We generally* want the network to have the smallest number of parameters that will give acceptably good results. If you constrain the network to have a small number of parameters, it <em>can't</em> just encode the inputs and outputs verbatim - to perform well on the test set, it pretty much needs to encode a generalization of the training data, rather than the specifics of the training data.</p>
  <p>In the context of the &quot;predict the next word&quot; approach used with LLMs, memorizing input/output pairs basically means that when prompted with a fragment of training data, it will regurgitate the remainder of that training data.</p>
  <p>So, if an LLM reproduces lots of training data verbatim, that's probably a pretty good sign that the model has far too many parameters. A model with fewer parameters can probably perform just as well (probably better) by generalizing more and memorizing less.</p>
  <p>* All generalizations are false, but some generalizations are useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-09T14:59:36.637Z
Answers.Text: This response isn't inherently bad, but it is misleading if you don't already know what you're looking at. Suggest rewriting it. Eliminate irrelevant language about irritating practitioners, reword so that it's clear that having too parameters doesn't definitely lead to overfitting (also as I mentioned in my update ChatGPT can't reproduce the entire book so suggest clarifying your point here), smaller networks do generally avoid overfitting but this is all highly dependent on use case, avoid saying generally and just give the specific scenarios where you think that's true
Answers.UserDisplayName: Grant Curell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T20:31:22.227Z
Answers.Text: I moved the disclaimer into a footnote and clarified that memorization is not inevitable.
  
  The question "Why are LLMs able to reproduce bodies of known text exactly?" is asking a general question about LLMs, so I don't think it would be useful to remove the generalizations.
Answers.UserDisplayName: NSFW
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-09T06:43:03.257Z
Answers.LastActivityDate: 2024-01-10T20:29:43.573Z
Answers.LastEditDate: 2024-01-10T20:29:43.573Z
Answers.UserDisplayName: NSFW
Body: <p>Mathematically, I wouldn't expect LLMs to be able to reproduce source texts exactly unless the source text was the probable outcome given some prompt. However, I have now tested HuggingFaceH4/zephyr-7b-beta, TheBloke/Llama-2-7B-Chat-GGUF, and ChatGPT and they are all able to accurately reproduce Moby Dick.</p>
  <h3>HuggingFaceH4/zephyr-7b-beta</h3>
  <p>Prompt:</p>
  <pre><code>Print the first page of Moby Dick.
  </code></pre>
  <p>Response:</p>
  <pre><code>I'm unable to print documents directly, but I can provide you with the text of the first page of moby dick by herman melville:
  
  call me ishmael. some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation.  
  ...SNIP...
  </code></pre>
  <h3>TheBloke/Llama-2-7B-Chat-GGUF</h3>
  <pre><code>print the first page of moby dick
  </code></pre>
  <pre><code>Certainly! Here is the first page of Moby Dick by Herman Melville:
  Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin-makers' shops, and 
  ...SNIP...
  </code></pre>
  <p>I would have expected them to generate a probabilistic answer but this appears to be exact. Is that simply because it was the likely output? Is there some alternate facility that is producing these results? I used <a href="https://github.com/h2oai/h2ogpt" rel="noreferrer">h2ogpt</a> for testing.</p>
  <h2>Update</h2>
  <p>It seems like it's probably just that it's mathematically likely because if you ask it other question it does just generate nonsense. Ex: What are the chapters in Moby Dick? I haven't read Moby Dick in English, but I don't recall a chapter 90, &quot;The Whale's Eye Cups&quot;</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-07T18:07:11.240Z
Comments.Text: How do you know it's reproducing and not just regurgitating? Bard would say "I'm also able to access and process information from the real world through google search and **keep my response consistent with search results**." - all it's doing is googling it. Get it to reproduce something that's *not* part of Project Gutenberg *for free* and then you might have something.
Comments.UserDisplayName: Mazura
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-08T14:33:55.610Z
Comments.Text: @Mazura That's not how h2ogpt works. h2ogpt is a 100% on prem model. Being on-prem and not talking to the Internet is one of the core purposes of the project.
Comments.UserDisplayName: Grant Curell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-12T19:42:21.760Z
Comments.Text: So sort of like Watson with the entirety of the internet downloaded; that doesn't really count.  It was a part of the "pre-generated databases" you fed it? https://github.com/h2oai/h2ogpt/blob/main/docs/README_LangChain.md - "A large Wikipedia database is also available." (that is like a 1.5 megs DL; I don't get it. That should be over 20GB worth of files...). Kinda like asking how it knows everything on wiki; *because it has wiki downloaded*? But if it's like a 2MB exe program with no databases thrown at it and it can do that w/o the internet....
Comments.UserDisplayName: Mazura
Tags: <generative-model><large-language-models><text-generation>
Title: Why are LLMs able to reproduce bodies of known text exactly?
UserDisplayName: Grant Curell
ThreadText: # Why are LLMs able to reproduce bodies of known text exactly?
  
  (Asked by: username_0 on 2024-01-04)
  
  Mathematically, I wouldn't expect LLMs to be able to reproduce source texts exactly unless the source text was the probable outcome given some prompt. However, I have now tested HuggingFaceH4/zephyr-7b-beta, TheBloke/Llama-2-7B-Chat-GGUF, and ChatGPT and they are all able to accurately reproduce Moby Dick.
  
  ### HuggingFaceH4/zephyr-7b-beta
  
  Prompt:
  
  ```
  Print the first page of Moby Dick.
  
  ```
  
  Response:
  
  ```
  I'm unable to print documents directly, but I can provide you with the text of the first page of moby dick by herman melville:
  
  call me ishmael. some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation.  
  ...SNIP...
  
  ```
  
  ### TheBloke/Llama-2-7B-Chat-GGUF
  
  ```
  print the first page of moby dick
  
  ```
  
  ```
  Certainly! Here is the first page of Moby Dick by Herman Melville:
  Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin-makers' shops, and 
  ...SNIP...
  
  ```
  
  I would have expected them to generate a probabilistic answer but this appears to be exact. Is that simply because it was the likely output? Is there some alternate facility that is producing these results? I used [h2ogpt](https://github.com/h2oai/h2ogpt) for testing.
  
  ## Update
  
  It seems like it's probably just that it's mathematically likely because if you ask it other question it does just generate nonsense. Ex: What are the chapters in Moby Dick? I haven't read Moby Dick in English, but I don't recall a chapter 90, "The Whale's Eye Cups"
  
  **Tags:** generative-model, large-language-models, text-generation
  
  > **Comments:**
  > **username_1**: How do you know it's reproducing and not just regurgitating? Bard would say "I'm also able to access and process information from the real world through google search and **keep my response consistent with search results**." - all it's doing is googling it. Get it to reproduce something that's *not* part of Project Gutenberg *for free* and then you might have something.  
  > **username_0**: @username_1 That's not how h2ogpt works. h2ogpt is a 100% on prem model. Being on-prem and not talking to the Internet is one of the core purposes of the project.  
  > **username_1**: So sort of like Watson with the entirety of the internet downloaded; that doesn't really count.  It was a part of the "pre-generated databases" you fed it? https://github.com/h2oai/h2ogpt/blob/main/docs/README_LangChain.md - "A large Wikipedia database is also available." (that is like a 1.5 megs DL; I don't get it. That should be over 20GB worth of files...). Kinda like asking how it knows everything on wiki; *because it has wiki downloaded*? But if it's like a 2MB exe program with no databases thrown at it and it can do that w/o the internet....  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2024-01-04. Score: 15
  
  Google "call me ishmael. some years ago—never mind how long precisely—having" and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, hence your observation. Note that some LLM training datasets do try to deduplicate data, e.g. <https://huggingface.co/datasets/uonlp/CulturaX>
  
  
  ---
  
  ### Answer by username_3 on 2024-01-05. Score: 14
  
  LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the "decompression" of parts to be recognizable and reasonably faithful. I don't think there's a really good understanding of which parts are recoverable to this extent. Naively one might think it's only very common material that appears repeatedly in the corpus, but they've also been found to reproduce exact or near-exact text for single-instance personal information. I'm not familiar with it but I suspect this is an exciting research area.
  
  "The probable outcome given some prompt" is one way of looking at what is computationally being *done with the model*, but it's important to realize that this assumes a probability model, which was derived from the training corpus. It might help as an exercise to imagine doing the model with a naive Markov chain instead, where it's mentally tractable to see how this kind of text can be encoded reproducibly in the probability model.
  
  > **Comments:**
  > **username_4**: If the description of GPT-4 in the New York Times lawsuit is correct, it's not even always *lossy* compression. GPT-4 apparently has enough parameters to losslessly encode a complete copy of its training data at a 10:1 compression ratio, while still having several hundred billion parameters left over.  
  > **username_5**: Those remaining parameters could allow for alternate paths in the stored text chain to provide variety and context matching. I got an amazing response once: "translation provided by [real company name here]". It looked like a direct leak of training data. It came as a response of a failed input, some breath in speech input to ChatGPT that was transcribed as empty text.  
  > **username_5**: Years ago, a technique for text to speech was "unit selection". A raw speech database was labeled and optimum path chosen according to phonetic transcription and prosody. A similar approach could be thought of for text generation.  
  > **username_6**: @username_5 Note: such responses are quite common from non-Chat GPTs, since they are only playing a game of "predict the next word". I don't know - and perhaps nobody outside of OpenAI knows - exactly what special sauce they wrapped around a predict-the-next-word game to turn it into a good question and answer machine, but it's not crazy that sometimes you see some of its inner nature peek through.  
  > **username_3**: @user253751: My naive understanding is that they do some transform to the question/prompt text to make it into a prior context for the "predict the next word game".  
  > **username_7**: ("ICE" is ***heavily*** overloaded, but it probably refers to [U.S. Immigration and Customs Enforcement](https://en.wikipedia.org/wiki/U.S._Immigration_and_Customs_Enforcement).)  
  > **username_8**: I'm not sure I agree with the dismissal of the probabilistic interpretation here in favor of the compression interpretation. Probability in this case is a way of _encoding information_. If certain sequences of inputs result in very high probabilities of specific words, that's because knowing that sequence tells you a lot about the next word to come. I would argue that probability and information are in fact the fundamental operating principles that allow these models to learn anything at all.  
  
  ---
  
  ### Answer by username_9 on 2024-01-09. Score: -1
  
  When we create and train neural networks, the goal is to get them to model a *general representation* of the input data, so that they will produce the desired output for inputs that are *generally similar* to inputs that were in the training data.
  
  But if your network has too many parameters, it doesn't need to generalize during the training stage. It might instead use those parameters to instead store representations of the input data and generate the desired output *precisely* - basically storing input/output pairs for everything input/output pair that's present in the training data. That won't necessarily happen, but it can happen. If you give it enough parameters, and train it long enough, it probably will happen.
  
  And then you feed it something that wasn't in the training set, and it produces garbage because it didn't generalize at all, it just encoded the inputs and outputs that it was trained with.
  
  This is one of the reasons that it's a good idea to train with one set of input/output pairs, and test with another set. If the network is generalizing, it will do a pretty good job with the test set even though those input/output pairs aren't part of the training set. But if the network is memorizing, it will do a pretty terrible job with the test set, because those input/output pairs were not memorized.
  
  We generally\* want the network to have the smallest number of parameters that will give acceptably good results. If you constrain the network to have a small number of parameters, it *can't* just encode the inputs and outputs verbatim - to perform well on the test set, it pretty much needs to encode a generalization of the training data, rather than the specifics of the training data.
  
  In the context of the "predict the next word" approach used with LLMs, memorizing input/output pairs basically means that when prompted with a fragment of training data, it will regurgitate the remainder of that training data.
  
  So, if an LLM reproduces lots of training data verbatim, that's probably a pretty good sign that the model has far too many parameters. A model with fewer parameters can probably perform just as well (probably better) by generalizing more and memorizing less.
  
  \* All generalizations are false, but some generalizations are useful.
  
  > **Comments:**
  > **username_0**: This response isn't inherently bad, but it is misleading if you don't already know what you're looking at. Suggest rewriting it. Eliminate irrelevant language about irritating practitioners, reword so that it's clear that having too parameters doesn't definitely lead to overfitting (also as I mentioned in my update ChatGPT can't reproduce the entire book so suggest clarifying your point here), smaller networks do generally avoid overfitting but this is all highly dependent on use case, avoid saying generally and just give the specific scenarios where you think that's true  
  > **username_9**: I moved the disclaimer into a footnote and clarified that memorization is not inevitable.  The question "Why are LLMs able to reproduce bodies of known text exactly?" is asking a general question about LLMs, so I don't think it would be useful to remove the generalizations.  
  
  ---
  
  