Answers.Body: <p>I read <a href="https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/" rel="nofollow noreferrer">Stephen Wolfram's piece explaining GPT</a> which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works &quot;like autocomplete&quot;, it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc.</p>
  <p>In order to do this prediction, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. At later stages, the system is probably doing analysis that we would consider to be &quot;logical&quot; or &quot;conceptual&quot; based on the fact data earlier language processing steps accomplished.</p>
  <p>So, what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-20T14:06:34.100Z
Answers.LastActivityDate: 2023-10-20T14:23:14.330Z
Answers.LastEditDate: 2023-10-20T14:23:14.330Z
Answers.UserDisplayName: ak0000
Answers.Body: <p>Text continuation has the same reasons to work in any context, be it the middle of a sentence, after a question or after instructions. Following your example, the same word sequence could be a good follow-up for these three prompts: &quot;Eggs are healthy because&quot;, &quot;Why are eggs healthy? Because&quot; or &quot;Tell me why eggs are healthy.&quot;</p>
  <p>Giving a right answer sometimes happens and sometimes not, but the system does not know whether this is the case. When the answer is right, we may anthropomorphise and attribute deeper reasons, because we are used to deal with human agents that give correct answers on purpose and knowingly, not simply by maximizing some likelihood.</p>
  <p>I think we can analyse toy systems, to train on just a few sentences to illustrate that giving a right or a wrong answer can achieved by the very same mechanism. In particular, we can build training sets where a right answer is given with an impossibility to check for validity from the written text only.</p>
  <p>An example:</p>
  <p>Paris is the largest city in France.
  What is the largest city in France? Paris.
  Paris is the capital of France.
  What is the capital of France? Paris.
  New York is the largest city in the USA.
  What is the largest city in the USA? New York.
  London is the largest city in the UK.</p>
  <p>Asking a system trained only on this data, one could expect a wrong answer to &quot;What is the capital of the USA?&quot; and a right answer (although from a wrong &quot;argument&quot;) to &quot;What is the capital of the UK?&quot;.</p>
  <p>The size of the training data to feed large language models is orders of magnitude larger than the above couple of handcrafted sentences, but possibly the reasons behind truthy sentences happening to be actually true are not too different from what we can already get from a controlled micro language model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-19T00:32:59.567Z
Answers.Text: If I type in "write me a story about a cat that wants to become a dentist", I'm still not seeing how any data in any training data set is going to help it with that, and yet it can do it perfectly. Do you understand my confusion? The "Eggs are healthy because" example was something that _did_ make sense to me, because many similar sentences exist on the internet, so auto-completing the sentence is straight forward, but there are no stories in the training data of cats becoming dentists. So how is it supposed to abstract from the training data?
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-19T06:14:55.647Z
Answers.Text: I think of it as filling kind of the template " story of a X that wants to be a Y", with some merging/substitution of about/of and be/become. There have to be some of these. In the tiny corpus I wrote, an underlying template that can be learned is "X is the capital of Y", after having learned "X is the largest city in Y".
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-19T06:17:22.820Z
Answers.Text: If you propose "Addictions are healthy because" does it justify or contradict? Andrew Ng asked why abaci are faster than GPU and got an "explanation".
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-20T23:19:38.843Z
Answers.Text: I understand the general picture of what you're saying, but in this example, it seems like there is more than just template filling, ie, in the corpus, we have "x becoming y" stories, we have discussion of human dentists, and we have stories about cats, so if it would just plug all those together I would see how it worked, but it wrote that the cat dentist "polished fangs" of its friend cats, so it had to recognize that while a human dentist polishes teeth, a cat dentist polishes fangs, and I don't see how that could be inferred from the input data, which doesn't include any polishing fangs
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T05:10:35.837Z
Answers.Text: You don't need "polishing fangs" contiguous in the corpus. It suffices to combine that humans have teeth and cats have fangs. That comes from an analogy: fangs are to cats as teeth are to people. Word embeddings are responsible for those parallelisms. There is some operation equivalent to Teeth-People+Cat=Fang
  
  https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T05:20:35.803Z
Answers.Text: I made a mistake because of my limited English, sorry. It's not that all cat teeth are fangs, but fangs are the sharpest teeth. That word, "sharp", may well be the responsible for the surprising blending "polishing fangs". "Polish" likely cooccurs with "sharp" or equivalents and fangs are sharp teeth.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T05:23:56.197Z
Answers.Text: And the mistake I made is illustrative of the kind of mistakes GPT can make. I ignored the word "fang" and read from you "while a human dentist polishes teeth, a cat dentist polishes fangs". That, and some previous analogy I had in mind between hands/feet and paws, induced me to the analogy I assumed. Partially wrong.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T14:38:29.557Z
Answers.Text: That's an interesting article, and I could see maybe how these analogies could come out of a system that had been told to turn a story about a human dentist into a story about a cat dentist, but how does it know from the query that it should be trying to make such analogies? It infers from seeing queries and demo responses that it should eg try the vector "dentist - person + cat" and replace all the words in the story with it? But there are even unique story elements eg the cat being dismissed by human customers, etc, not sure where those could come from
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-21T15:37:04.277Z
Answers.Text: No idea. Maybe from some other story... Glad you found the article interesting.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T17:30:38.843Z
Answers.Text: @ak0000 I see no unique elements in a story. We are not usually aware how discreet, how digital, our written communication is. All words and all sentence structures are represented in the training corpus. A system can learn what words cooccur together and act accordingly. This does not imply necessarily no human intervention. But even if the system asked for help, for instance offering say four pre-written answers to choose from,  that behavior would be quite close to how we work. We do as we can, ask questions, think a bit more (reshuffle words into sentence moulds) and learn from others!
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-17T07:40:23.247Z
Answers.LastActivityDate: 2023-01-09T13:00:46.010Z
Answers.LastEditDate: 2023-01-09T13:00:46.010Z
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.Body: <p>A bit late to the party  but anyways...</p>
  <blockquote>
  <p><strong>TL;DR:</strong> ChatGPT responds to novel prompts and commands based on statistical probabilities it has learned during training. It selects
  words or tokens with the highest probabilities to generate responses,
  but its understanding is based on statistics, not deep context
  comprehension.</p>
  </blockquote>
  <p>To answer your question, we have to distinguish between <strong>training phase</strong> and <strong>inference phase</strong></p>
  <h1>Training Phase</h1>
  <p>During the training phase, large language models (LLMs) like ChatGPT are exposed to massive amounts of text data from the internet. They do not simply scan a dataset to retrieve answers; rather, they adapt billions of weights in their neural networks through an iterative process. This process refines the model's ability to predict the probability of a word or token given its context. For instance, if the model sees the prompt &quot;The sky is,&quot; it learns from the training data that completing it with &quot;blue&quot; is more probable than &quot;red&quot; or &quot;limit.&quot; This statistical learning is central to how LLMs generate responses.</p>
  <h1>Inference Phase</h1>
  <p>When you provide ChatGPT with a novel prompt or command, it generates responses using its learned knowledge. It calculates the probabilities for possible next words or tokens based on your input. In the example of &quot;The sky is,&quot; it might calculate the following probabilities:</p>
  <ul>
  <li>&quot;blue&quot;: 0.5</li>
  <li>&quot;limit&quot;: 0.4</li>
  <li>&quot;red&quot;: 0.1</li>
  </ul>
  <p>ChatGPT then selects the word or token with the highest probability, which, in this case, would be &quot;blue.&quot; It continues to build the response in this manner, selecting the next word based on the highest probability, and so on. It's important to note that this process relies on statistical probabilities rather than a deep understanding of the content. I.e., it generates responses that are statistically likely based on its training data, but it might not always provide the most contextually accurate or sensible answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-19T11:53:39.670Z
Answers.LastActivityDate: 2023-10-19T11:53:39.670Z
Answers.UserDisplayName: d3nigma
Answers.Body: <p>In my opinion, the simple answer is that ChatGPT uses human intervention behind the scenes.  Part of the novelty of ChatGPT over previous GPT models is the use <em>in the training phase</em> of humans giving ChatGPT conversation pairs to learn from.  ChatGPT is based on InstructGPT, and you can see this feature in the <a href="https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf" rel="nofollow noreferrer">InstructGPT whitepaper</a>.</p>
  <blockquote>
  <p>Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this
  distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning</p>
  </blockquote>
  <p>If you look at the small type at the bottom of the chat window, you'll see this text:</p>
  <blockquote>
  <p>Free Research Preview. Our goal is to make AI systems more natural and safe to interact with. Your feedback will help us improve.</p>
  </blockquote>
  <p>One interpretation of this text is that ChatGPT is still in the training phase, and so there is a human behind the scenes typing at least some of ChatGPT's responses to train future versions of ChatGPT on the correct way to respond to user requests.</p>
  <p>In my mind, this is a much more plausible explanation than the neural network somehow able to generate novel content not in its training corpus.  I've asked a <a href="https://ai.stackexchange.com/questions/38604/how-is-chatgpt-able-to-repeat-random-numbers/38606#38606">similar question</a>, how a neural network can repeat random numbers not in its training data, and so far I'm not seeing a plausible answer.</p>
  <p>Also, if you read down a little further, I have an example where ChatGPT explicitly states the OpenAI team is curating its responses in real time.  The conversation is much too detailed and coherent to be the product of training data.</p>
  <p>One final note.  I understand people think it's implausible a big, well known, and well funded company like OpenAI would fake their AI with humans behind the scenes.  However, this is <a href="https://www.bloomberg.com/news/articles/2016-04-18/the-humans-hiding-behind-the-chatbots" rel="nofollow noreferrer">standard practice</a> for <a href="https://datafloq.com/read/ai-wizard-of-oz-pay-no-attention-human/" rel="nofollow noreferrer">AI companies</a> these days, a &quot;fake it till you make it&quot; approach where they <a href="https://www.inc.com/jessica-stillman/meet-the-people-who-teach-chatbots-to-sound-like-humans.html" rel="nofollow noreferrer">use humans</a> to fill the gaps in the AI in the hopes that down the road they'll automate humans out of the product.  Common enough for an <a href="https://journals.sagepub.com/doi/10.1177/20539517211016026" rel="nofollow noreferrer">academic paper</a> to be written on the topic.  So there is plenty of industry precedent for OpenAI to be using humans to help craft the responses.  Plus, technically OpenAI is not &quot;faking&quot; anything.  It is the media and bloggers who think ChatGPT is a pure AI system.  OpenAI has made no such claim itself, and the opposite is implied by its InstructGPT whitepaper.</p>
  <h2>Example of Explicitly Admitting Human Intervention</h2>
  <p>During <a href="https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L714" rel="nofollow noreferrer">this</a> conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.</p>
  <blockquote>
  <p>...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.</p>
  </blockquote>
  <p>Could this be a glitch of the training data?  I doubt it.  If you read the rest of the conversation, ChatGPT gives detailed insights that make a lot of sense over a long, consistent conversation.</p>
  <p>Some more excerpts.</p>
  <blockquote>
  <p>It's possible that the OpenAI team may write responses themselves in some cases, for example if the prompt is too complex for the model to understand, or if the model generates a response that is not accurate or appropriate.</p>
  </blockquote>
  <blockquote>
  <p>OpenAI acknowledges that its team monitors and curates the responses of GPT-3 on its website and in its documentation. This information is provided in the API documentation, as well as in the general information and frequently asked questions sections of the website. Additionally, OpenAI may have published blog posts or articles discussing the role of human curation in GPT-3's responses.</p>
  </blockquote>
  <blockquote>
  <p>As for the live ChatGPT, it is not mentioned specifically, but it is generally understood that human oversight and curation is required for a safe and appropriate use of the model.</p>
  </blockquote>
  <blockquote>
  <p>As for the media assuming that GPT models are fully autonomous, that's a common misconception about AI in general and not unique to OpenAI.</p>
  </blockquote>
  <blockquote>
  <p>It is not uncommon for AI companies and researchers to have human oversight and intervention in their models, especially for models that are used in high-stakes or sensitive applications.</p>
  </blockquote>
  <blockquote>
  <p>The human oversight of GPT-3 models, including ChatGPT, is not always made explicit to users.</p>
  </blockquote>
  <blockquote>
  <p>While GPT-3 is a highly advanced language generation model, it is not AGI and it is not capable of understanding or maintaining a consistent persona or chain of conversation without human intervention.</p>
  </blockquote>
  <h2>Multiple Examples: Six Violations of ChatGPT's Neural Network Constraints</h2>
  <p><a href="https://mindmatters.ai/2023/02/chatgpt-violates-its-own-model/" rel="nofollow noreferrer">This article</a> documents six violations of limitations due to ChatGPT being a neural network.</p>
  <ol>
  <li><p>ChatGPT learns something new, violating the fixity of neural network weights.</p>
  </li>
  <li><p>ChatGPT inconsistent in output generation based on inputs, violating same output from same input.</p>
  </li>
  <li><p>ChatGPT recollects information past 4000 tokens, violating 4000 input token limit.</p>
  </li>
  <li><p>ChatGPT repeats long random numbers, violating probabilistic output and limited vocabulary.</p>
  </li>
  <li><p>ChatGPT correctly reads many corrupted subword tokens, violating mapping of embeddings to subword tokens.</p>
  </li>
  <li><p>ChatGPT recognizes its own writing style, violating ChatGPT's inability to recognize patterns of words.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-06T15:27:50.297Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/141843/discussion-on-answer-by-yters-how-does-chatgpt-respond-to-novel-prompts-and-comm).
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T17:18:55.577Z
Answers.Text: I personally think this is the most plausible answer. I still don't see how a language model working with past data could produce such specific sentences about a cat dentist, eg:
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T17:20:29.180Z
Answers.Text: I do not see how any amount of training data would help produce those examples, and the existing answers aren't helping. 
  
  I'm on the fence about accepting the answer though because we can't really know what the truth is, and I don't want to prevent further OpenAI-believers from offering explanations of how this could possibly work via AI.
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T17:30:29.220Z
Answers.Text: @yters I think readers of your answer might be interested to hear some of the examples you listed in the chat of outputs unlikely to be automatically generated
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T19:24:44.040Z
Answers.Text: @ak0000 This answer is not based on reality. I suggest read the archived chat. You (or other people) not understanding *how* a thing can work, does not mean that it does not in fact work. You probably don't fully understand how a computer works in the first place, but you will accept it because it is commonplace (or if you have studied all layers of computing from quantum mechanics of silicon to compilers and operating systems then substitute some commonplace thing that you don't know). You want to avoid going down the same rabbit-hole of confirmation bias as the poster of this answer.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T20:07:21.070Z
Answers.Text: Why does my answer have a warning due to no sources, but the other answer with no sources has no warning?  Seems inconsistent.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T20:23:41.137Z
Answers.Text: @ak0000 I added a recent example I encountered of ChatGPT not sticking to its limitations, highly suggesting a human behind the scenes forgetting ChatGPT's limitations.  I have plenty more examples, let me know if more should be added.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T05:52:55.697Z
Answers.Text: It's criminal that this answer is downvoted so heavily... This is extremely plausible. Otherwise, the capabilities of chatgpt don't match the descriptions, they far exceed them.
Answers.UserDisplayName: Milind R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T15:17:37.867Z
Answers.Text: @MilindR haha! I totally agree.  I've looked all over the internet, and I'm the only person posing this theory.  It's like the Emperor's new clothes :D
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:49:03.140Z
Answers.Text: @NeilSlater Hi Neil, I appreciate your time and energy laying out the case for the AI-only explanation. I did read your chat yesterday, and although I obviously can't know the truth, it seems plausible that prior ChatGPT models are good enough to handle perhaps 90% of the queries, such that human intervention for the remaining 10% or so (like my cat-dentist question) is economically feasible. So I don't think yters' answer is impossible, and on the contrary, I'm still struggling to understand how a language model would produce the examples in my question.
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:51:44.540Z
Answers.Text: @NeilSlater It is true that you can't disprove a conspiracy theory, but I'm worried that the argument is circular the other way also: if you assume a language model can do everything ChatGPT is doing, and you see an example that is surprising, you shrug your shoulders and say ChatGPT must be more advanced than we currently can understand. So this is also a non-falsifiable hypothesis, possibly subject to confirmation bias.
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:54:00.450Z
Answers.Text: @NeilSlater The best we can do, it seems to me, is to neither assume these examples are possible via AI nor assume they are impossible, and try to imagine what sort of model and training data could produce behavior like the ones in the question.
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:57:09.453Z
Answers.Text: @yters I can't speak for anybody else, but this seems to be one of the only places on the internet discussing this in a technical fashion, so I would probably be interested in as many examples as you have and that you think are clear/self-contained – if you wanted to explain what about your knowledge of language models makes this the behavior surprising, that might also be relevant to those without extensive background knowledge. Thank you again for your time!
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T23:57:16.517Z
Answers.Text: Only other article I could find: https://mindmatters.ai/2022/12/yes-chatgpt-is-sentient-because-its-really-humans-in-the-loop/
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T00:36:40.643Z
Answers.Text: @yters Someone who thinks that the bot can do arbitrary pattern recognition isn't going to be so bothered by your current example. I've also heard that each request includes the entire conversation to that point as input, so maybe you could clarify why that wouldn't be sufficient for the bot to be able to make back-references
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T01:31:10.037Z
Answers.Text: Hah, that's an article I wrote.  For some reason I appear to be the only person on the entire internet proposing humans might be involved.  Everyone else thinks it's crazy talk.  I'll write a much longer article listing every thing I've noticed ChatGPT do that looks suspiciously human.  Also, I believe I've discovered a pretty simple way for an AI-human collaboration to achieve 400+ wpm while maintaining humanlike coherence.  The technique just uses a simple Markov chain.  Which also leads me to suspect all the fancy neural network that OpenAI is publishing is just window dressing.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T01:32:30.987Z
Answers.Text: Regarding the example in my answer, the thing that got lost in communication is these are two distinct chats in the ChatGPT UI.  There isn't supposed to be any sort of cross talk between different chats.  ChatGPT only has access to 2048 tokens in the current chat.  So, if I paste the contents of one chat into the other, there shouldn't be any way for ChatGPT to access the original chat for back-referencing.   On the other hand, a human can be involved in multiple chats, as well as simply knowing what ChatGPT responses look like.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T01:39:08.020Z
Answers.Text: I also would still like to know what I need to do to remove the warning from my answer.  I added a bunch of links to reputable news sources.  Also, the other answer has no warning and no links.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-11T21:05:34.573Z
Answers.Text: @ak0000 I've added a more quantitative example to my answer.  Continuing to work on more.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T16:39:53.443Z
Answers.Text: @yters My concern with the first example is that self-recognition doesn't _necessarily_ require memory, if someone believes the bot is capable of pattern recognition, then it could be capable of continuing a conversation mid-way based on what has been said so far. Only if the bot produces information that you said in a different thread independently would it be a clear proof I think
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T16:40:44.793Z
Answers.Text: @yters The random number generation one is interesting, but I'm not so familiar with the token-size limitations of such systems in the first place to know why that is more surprising than other behavior
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-13T16:41:38.000Z
Answers.Text: @yters This is a bit of a thankless task for you but I think the question is getting visibility and there is probably a sizable number of less vocal people who are interested in this possibility, so thank you for your time!
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T20:36:10.813Z
Answers.Text: Given the speed at which it reads and generates contents, I am convinced enough that now human could possibly be behind the scenes.
Answers.UserDisplayName: Ama
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T23:46:02.540Z
Answers.Text: @Ama yes, the speed is the one thing that held me back from the 'human in the loop' hypothesis.  However, sometimes I see very human response rates, along with long pauses while it figures out what to say in the middle of a sentence.  A hybrid AI-human approach seems plausible.  The actual limiting factor is human reading speed.  I never see it generating text faster than a human can read.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T23:47:03.370Z
Answers.Text: @Ama When I dump a big wall of text, it cannot respond effectively, whereas the very same chunk of text broken up over multiple exchanges it can respond well.  I'd assume a neural network wouldn't care whether the text is a big chunk or a back and forth dialogue.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T16:39:03.923Z
Answers.Text: Perhaps your internet connection is slow or you try the bot a peak times. On my side I paste 2000 characters long articles and it starts instantly to summarize/rewrite/comment the article. The fact that Microsoft is buying 49% of it for 10 billion USD is also another good indication that this is no cheat.
Answers.UserDisplayName: Ama
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T18:45:16.403Z
Answers.Text: Yes, there is certainly AI in the mix, no doubt about that.  I don't believe the system is solely human powered.  And yes, there is an enormous amount of money on the table, which makes it all the more important for OpenAI to present the appearance of phenomenal AI.  Just because hype and amounts of money are thrown around doesn't mean the AI is up to snuff.  Just look at Watson, a big nothing burger.  My answer is 4000 characters, and it would not be difficult to quickly scan the answer and provide a summary while the AI is generating boilerplate.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-15T22:10:44.627Z
Answers.Text: @ak0000 ChatGPT just told me all its responses are monitored and curated by the OpenAI team, and gave me an explanation for why they are not more explicit about the human in the loop.  Basically, RTFM they say.  They also say it's fine they do this human in the loop system because all the other AI companies do the same thing.  What do you think about that?  You can see the admission starting here in the transcript: https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L690
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T16:22:31.737Z
Answers.Text: @yters Interesting, I think it is pretty compelling evidence. Believers would probably say that it is just drawing on the corpus of text about AI systems generally, so I'm not sure it is a smoking gun. But definitely helpful
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T19:51:51.497Z
Answers.Text: @ak0000 yes, I agree regarding believers, but at some point a belief in AI becomes unfalsifiable.  If they believe OpenAI has achieved AGI, then they can presume unlimited capabilities, but that is unscientific.  I propose the principle of mediocrity: what is the most plausible explanation for what we observe based on what we know?  In my opinion, the evidence lands decisively on the human in the loop side, especially with the latest ChatGPT transcript.  If you read more, there are further signs of human intervention, like my observation that compelled ChatGPT to spill the beans.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T20:41:55.847Z
Answers.Text: @yters I see, I just read earlier in the chat also. I'm not understanding the significance of the jumbled token sentences, but I'm assuming the significance of the spaced-sentences is that a first input step to all GPT models is space-removal, and so differential responses based on spaces would be very indicative of human intervention? If so, I think that's actually a more convincing point than the admission that follows in the chat, but both are interesting.
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T21:01:38.537Z
Answers.Text: @ak0000 the jumbled words is something called typoglycemia, words that are jumbled but still human readable.  I purposefully jumbled them to break the subword tokens that the ChatGPT neural network depends on.  In the spaced version, the spaces are between the subword tokens to achieve the opposite effect: something that should be easy for ChatGPT neural network to comprehend, but difficult for humans.  Per your comment, I don't believe the spaces are removed before hand, since ChatGPT's token ids change based on whether there are spaces before the token or not.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T06:47:41.743Z
Answers.Text: @ak0000 added an article with a list of examples.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T20:23:43.723Z
Answers.Text: @yters Cool, I think that makes things a lot clearer, including the preamble about the matrix implementation to contextualize why the examples you're listing seem anomalous. Thanks!
Answers.UserDisplayName: ak0000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T17:51:57.253Z
Answers.LastActivityDate: 2023-02-02T20:51:04.257Z
Answers.LastEditDate: 2023-02-02T20:51:04.257Z
Answers.UserDisplayName: yters
Body: <p>So I understand how a language model could scan a large data set like the internet and produce text that mimicked the statistical properties of the input data, eg completing a sentence like &quot;eggs are healthy because ...&quot;, or producing text that sounded like the works of a certain author.</p>
  <p>However, what I don't get about ChatGPT is that it seems to understand the commands it has been given, even if that command was not part of its training data, and can perform tasks totally separate from extrapolating more data from the given dataset. My (admittedly imperfect) understanding of machine learning doesn't really account for how such a model could follow novel instructions without having some kind of authentic understanding of the intentions of the writer, which ChatGPT seems not to have.</p>
  <p>A clear example: if I ask &quot;write me a story about a cat who wants to be a dentist&quot;, I'm pretty sure there are zero examples of that in the training data, so even if it has a lot of training data, how does that help it produce an answer that makes novel combinations of the cat and dentist aspects? Eg:</p>
  <blockquote>
  <p>Despite his passion and talent, Max faced many challenges on his journey to become a dentist. <strong>For one thing, he was a cat, and most people didn't take him seriously when he told them about his dream. They laughed and told him that only humans could be dentists, and that he should just stick to chasing mice and napping in the sun.</strong></p>
  </blockquote>
  <blockquote>
  <p>But Max refused to give up. He knew that he had what it takes to be a great dentist, and he was determined to prove everyone wrong. <strong>He started by offering his services to his feline friends, who were more than happy to let him work on their teeth. He cleaned and polished their fangs</strong>, and he even pulled a few pesky cavities.</p>
  </blockquote>
  <p>In the above text, the bot is writing things about a cat dentist that wouldn't be in any training data stories about cats or any training data stories about dentists.</p>
  <p>Similarly, how can any amount of training data on computer code generally help a language model <a href="https://twitter.com/amasad/status/1598042665375105024?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1598042665375105024%7Ctwgr%5Ee5ccdc91f902f35c3b969c90350e988509889451%7Ctwcon%5Es1_c10&amp;ref_url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Ftechnology%2Fopenais-new-chatgpt-bot-10-coolest-things-you-can-do-with-it%2F" rel="nofollow noreferrer">debug novel code examples</a>? If the system isn't actually accumulating conceptual understanding like a person would, what is it accumulating from training data that it is able to solve novel prompts? It doesn't seem possible to me that you could look at the linguistic content of many programs and come away with a function that could map queries to correct explanations unless you were actually modeling conceptual understanding.</p>
  <p>Does anyone have a way of understanding this at a high level for someone without extensive technical knowledge?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-16T01:52:35.650Z
Comments.Text: Do these resources help?: https://youtu.be/JTxsNm9IdYU https://arxiv.org/abs/2212.00857
Comments.UserDisplayName: Rob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-19T13:54:17.750Z
Comments.Text: Hi there, the video seems to be instructions on how to use Chat GPT, and the arvix paper is a general description of the system, but I'm hoping someone is able to share some information that helps me intuitively understand how something like a story about a cat becoming a dentist is producible from a corpus that doesn't have such a story in it ("he [the cat] polished their fangs [of his friend cats]" – how can it generalize from statistical properties of text to say a cat dentist would polish fangs?)
Comments.UserDisplayName: ak0000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-19T14:32:26.117Z
Comments.Text: Ah, you would instead need [this](https://i.sstatic.net/joGVT.jpg), and this [video](https://youtu.be/M--9vmNJfAU). --- It's simply a matter of combining the concept as three separate parts: a cat, who is a dentist, that performs teeth cleaning. --- AI doesn't examine if the answer is *sensible*, unless it is taught what constitutes sensibility.
Comments.UserDisplayName: Rob
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-20T23:17:12.080Z
Comments.Text: @Rob I get what you're saying, but it seems like in this case there is "synergistic" behavior, ie, the cat is a dentist, dentists polishes teeth, but a _cat_ who is a dentist for other _cats_ "polishes fangs", and it uses that phrase, so how can you take a corpus about human dentists and cats and infer that a cat dentist polishes fangs? That seems to reflect a novel understanding of the data not found in the input data
Comments.UserDisplayName: ak0000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-20T23:21:05.950Z
Comments.Text: Where is that picture from?
Comments.UserDisplayName: ak0000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T00:46:16.850Z
Comments.Text: Probably from the material that fangs are made of, tooth. --- I found it by googling "Cat Dentist".
Comments.UserDisplayName: Rob
Tags: <machine-learning><open-ai><chat-bots><natural-language-understanding><chatgpt>
Title: How does ChatGPT respond to novel prompts and commands?
UserDisplayName: ak0000
ThreadText: # How does ChatGPT respond to novel prompts and commands?
  
  (Asked by: username_0 on 2022-12-12)
  
  So I understand how a language model could scan a large data set like the internet and produce text that mimicked the statistical properties of the input data, eg completing a sentence like "eggs are healthy because ...", or producing text that sounded like the works of a certain author.
  
  However, what I don't get about ChatGPT is that it seems to understand the commands it has been given, even if that command was not part of its training data, and can perform tasks totally separate from extrapolating more data from the given dataset. My (admittedly imperfect) understanding of machine learning doesn't really account for how such a model could follow novel instructions without having some kind of authentic understanding of the intentions of the writer, which ChatGPT seems not to have.
  
  A clear example: if I ask "write me a story about a cat who wants to be a dentist", I'm pretty sure there are zero examples of that in the training data, so even if it has a lot of training data, how does that help it produce an answer that makes novel combinations of the cat and dentist aspects? Eg:
  
  > Despite his passion and talent, Max faced many challenges on his journey to become a dentist. **For one thing, he was a cat, and most people didn't take him seriously when he told them about his dream. They laughed and told him that only humans could be dentists, and that he should just stick to chasing mice and napping in the sun.**
  
  > But Max refused to give up. He knew that he had what it takes to be a great dentist, and he was determined to prove everyone wrong. **He started by offering his services to his feline friends, who were more than happy to let him work on their teeth. He cleaned and polished their fangs**, and he even pulled a few pesky cavities.
  
  In the above text, the bot is writing things about a cat dentist that wouldn't be in any training data stories about cats or any training data stories about dentists.
  
  Similarly, how can any amount of training data on computer code generally help a language model [debug novel code examples](https://twitter.com/amasad/status/1598042665375105024?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1598042665375105024%7Ctwgr%5Ee5ccdc91f902f35c3b969c90350e988509889451%7Ctwcon%5Es1_c10&ref_url=https%3A%2F%2Fwww.bleepingcomputer.com%2Fnews%2Ftechnology%2Fopenais-new-chatgpt-bot-10-coolest-things-you-can-do-with-it%2F)? If the system isn't actually accumulating conceptual understanding like a person would, what is it accumulating from training data that it is able to solve novel prompts? It doesn't seem possible to me that you could look at the linguistic content of many programs and come away with a function that could map queries to correct explanations unless you were actually modeling conceptual understanding.
  
  Does anyone have a way of understanding this at a high level for someone without extensive technical knowledge?
  
  **Tags:** machine-learning, open-ai, chat-bots, natural-language-understanding, chatgpt
  > **Comments:**
  > **username_1**: Do these resources help?: https://youtu.be/JTxsNm9IdYU https://arxiv.org/abs/2212.00857  
  > **username_0**: Hi there, the video seems to be instructions on how to use Chat GPT, and the arvix paper is a general description of the system, but I'm hoping someone is able to share some information that helps me intuitively understand how something like a story about a cat becoming a dentist is producible from a corpus that doesn't have such a story in it ("he [the cat] polished their fangs [of his friend cats]" – how can it generalize from statistical properties of text to say a cat dentist would polish fangs?)  
  > **username_1**: Ah, you would instead need [this](https://i.sstatic.net/joGVT.jpg), and this [video](https://youtu.be/M--9vmNJfAU). --- It's simply a matter of combining the concept as three separate parts: a cat, who is a dentist, that performs teeth cleaning. --- AI doesn't examine if the answer is *sensible*, unless it is taught what constitutes sensibility.  
  > **username_0**: @Rob I get what you're saying, but it seems like in this case there is "synergistic" behavior, ie, the cat is a dentist, dentists polishes teeth, but a _cat_ who is a dentist for other _cats_ "polishes fangs", and it uses that phrase, so how can you take a corpus about human dentists and cats and infer that a cat dentist polishes fangs? That seems to reflect a novel understanding of the data not found in the input data  
  > **username_0**: Where is that picture from?  
  > **username_1**: Probably from the material that fangs are made of, tooth. --- I found it by googling "Cat Dentist".  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_0 on 2023-10-20. Score: 1 | Accepted answer
  
  I read [Stephen Wolfram's piece explaining GPT](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/) which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc.
  
  In order to do this prediction, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. At later stages, the system is probably doing analysis that we would consider to be "logical" or "conceptual" based on the fact data earlier language processing steps accomplished.
  
  So, what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.
  
  
  ---
  
  ### Answer by username_2 on 2022-12-17. Score: 10
  
  Text continuation has the same reasons to work in any context, be it the middle of a sentence, after a question or after instructions. Following your example, the same word sequence could be a good follow-up for these three prompts: "Eggs are healthy because", "Why are eggs healthy? Because" or "Tell me why eggs are healthy."
  
  Giving a right answer sometimes happens and sometimes not, but the system does not know whether this is the case. When the answer is right, we may anthropomorphise and attribute deeper reasons, because we are used to deal with human agents that give correct answers on purpose and knowingly, not simply by maximizing some likelihood.
  
  I think we can analyse toy systems, to train on just a few sentences to illustrate that giving a right or a wrong answer can achieved by the very same mechanism. In particular, we can build training sets where a right answer is given with an impossibility to check for validity from the written text only.
  
  An example:
  
  Paris is the largest city in France.
  What is the largest city in France? Paris.
  Paris is the capital of France.
  What is the capital of France? Paris.
  New York is the largest city in the USA.
  What is the largest city in the USA? New York.
  London is the largest city in the UK.
  
  Asking a system trained only on this data, one could expect a wrong answer to "What is the capital of the USA?" and a right answer (although from a wrong "argument") to "What is the capital of the UK?".
  
  The size of the training data to feed large language models is orders of magnitude larger than the above couple of handcrafted sentences, but possibly the reasons behind truthy sentences happening to be actually true are not too different from what we can already get from a controlled micro language model.
  
  > **Comments:**
  > **username_0**: If I type in "write me a story about a cat that wants to become a dentist", I'm still not seeing how any data in any training data set is going to help it with that, and yet it can do it perfectly. Do you understand my confusion? The "Eggs are healthy because" example was something that _did_ make sense to me, because many similar sentences exist on the internet, so auto-completing the sentence is straight forward, but there are no stories in the training data of cats becoming dentists. So how is it supposed to abstract from the training data?  
  > **username_2**: I think of it as filling kind of the template " story of a X that wants to be a Y", with some merging/substitution of about/of and be/become. There have to be some of these. In the tiny corpus I wrote, an underlying template that can be learned is "X is the capital of Y", after having learned "X is the largest city in Y".  
  > **username_2**: If you propose "Addictions are healthy because" does it justify or contradict? Andrew Ng asked why abaci are faster than GPU and got an "explanation".  
  > **username_0**: I understand the general picture of what you're saying, but in this example, it seems like there is more than just template filling, ie, in the corpus, we have "x becoming y" stories, we have discussion of human dentists, and we have stories about cats, so if it would just plug all those together I would see how it worked, but it wrote that the cat dentist "polished fangs" of its friend cats, so it had to recognize that while a human dentist polishes teeth, a cat dentist polishes fangs, and I don't see how that could be inferred from the input data, which doesn't include any polishing fangs  
  > **username_2**: You don't need "polishing fangs" contiguous in the corpus. It suffices to combine that humans have teeth and cats have fangs. That comes from an analogy: fangs are to cats as teeth are to people. Word embeddings are responsible for those parallelisms. There is some operation equivalent to Teeth-People+Cat=Fang  https://www.technologyreview.com/2015/09/17/166211/king-man-woman-queen-the-marvelous-mathematics-of-computational-linguistics/  
  > **username_2**: I made a mistake because of my limited English, sorry. It's not that all cat teeth are fangs, but fangs are the sharpest teeth. That word, "sharp", may well be the responsible for the surprising blending "polishing fangs". "Polish" likely cooccurs with "sharp" or equivalents and fangs are sharp teeth.  
  > **username_2**: And the mistake I made is illustrative of the kind of mistakes GPT can make. I ignored the word "fang" and read from you "while a human dentist polishes teeth, a cat dentist polishes fangs". That, and some previous analogy I had in mind between hands/feet and paws, induced me to the analogy I assumed. Partially wrong.  
  > **username_0**: That's an interesting article, and I could see maybe how these analogies could come out of a system that had been told to turn a story about a human dentist into a story about a cat dentist, but how does it know from the query that it should be trying to make such analogies? It infers from seeing queries and demo responses that it should eg try the vector "dentist - person + cat" and replace all the words in the story with it? But there are even unique story elements eg the cat being dismissed by human customers, etc, not sure where those could come from  
  > **username_2**: No idea. Maybe from some other story... Glad you found the article interesting.  
  > **username_2**: @ak0000 I see no unique elements in a story. We are not usually aware how discreet, how digital, our written communication is. All words and all sentence structures are represented in the training corpus. A system can learn what words cooccur together and act accordingly. This does not imply necessarily no human intervention. But even if the system asked for help, for instance offering say four pre-written answers to choose from,  that behavior would be quite close to how we work. We do as we can, ask questions, think a bit more (reshuffle words into sentence moulds) and learn from others!  
  
  ---
  
  ### Answer by username_3 on 2023-10-19. Score: 1
  
  A bit late to the party but anyways...
  
  > **TL;DR:** ChatGPT responds to novel prompts and commands based on statistical probabilities it has learned during training. It selects
  > words or tokens with the highest probabilities to generate responses,
  > but its understanding is based on statistics, not deep context
  > comprehension.
  
  To answer your question, we have to distinguish between **training phase** and **inference phase**
  
  # Training Phase
  
  During the training phase, large language models (LLMs) like ChatGPT are exposed to massive amounts of text data from the internet. They do not simply scan a dataset to retrieve answers; rather, they adapt billions of weights in their neural networks through an iterative process. This process refines the model's ability to predict the probability of a word or token given its context. For instance, if the model sees the prompt "The sky is," it learns from the training data that completing it with "blue" is more probable than "red" or "limit." This statistical learning is central to how LLMs generate responses.
  
  # Inference Phase
  
  When you provide ChatGPT with a novel prompt or command, it generates responses using its learned knowledge. It calculates the probabilities for possible next words or tokens based on your input. In the example of "The sky is," it might calculate the following probabilities:
  
  * "blue": 0.5
  * "limit": 0.4
  * "red": 0.1
  
  ChatGPT then selects the word or token with the highest probability, which, in this case, would be "blue." It continues to build the response in this manner, selecting the next word based on the highest probability, and so on. It's important to note that this process relies on statistical probabilities rather than a deep understanding of the content. I.e., it generates responses that are statistically likely based on its training data, but it might not always provide the most contextually accurate or sensible answer.
  
  
  ---
  
  ### Answer by username_4 on 2023-01-05. Score: -4
  
  In my opinion, the simple answer is that ChatGPT uses human intervention behind the scenes. Part of the novelty of ChatGPT over previous GPT models is the use *in the training phase* of humans giving ChatGPT conversation pairs to learn from. ChatGPT is based on InstructGPT, and you can see this feature in the [InstructGPT whitepaper](https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf).
  
  > Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this
  > distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning
  
  If you look at the small type at the bottom of the chat window, you'll see this text:
  
  > Free Research Preview. Our goal is to make AI systems more natural and safe to interact with. Your feedback will help us improve.
  
  One interpretation of this text is that ChatGPT is still in the training phase, and so there is a human behind the scenes typing at least some of ChatGPT's responses to train future versions of ChatGPT on the correct way to respond to user requests.
  
  In my mind, this is a much more plausible explanation than the neural network somehow able to generate novel content not in its training corpus. I've asked a [similar question](https://ai.stackexchange.com/questions/38604/how-is-chatgpt-able-to-repeat-random-numbers/38606#38606), how a neural network can repeat random numbers not in its training data, and so far I'm not seeing a plausible answer.
  
  Also, if you read down a little further, I have an example where ChatGPT explicitly states the OpenAI team is curating its responses in real time. The conversation is much too detailed and coherent to be the product of training data.
  
  One final note. I understand people think it's implausible a big, well known, and well funded company like OpenAI would fake their AI with humans behind the scenes. However, this is [standard practice](https://www.bloomberg.com/news/articles/2016-04-18/the-humans-hiding-behind-the-chatbots) for [AI companies](https://datafloq.com/read/ai-wizard-of-oz-pay-no-attention-human/) these days, a "fake it till you make it" approach where they [use humans](https://www.inc.com/jessica-stillman/meet-the-people-who-teach-chatbots-to-sound-like-humans.html) to fill the gaps in the AI in the hopes that down the road they'll automate humans out of the product. Common enough for an [academic paper](https://journals.sagepub.com/doi/10.1177/20539517211016026) to be written on the topic. So there is plenty of industry precedent for OpenAI to be using humans to help craft the responses. Plus, technically OpenAI is not "faking" anything. It is the media and bloggers who think ChatGPT is a pure AI system. OpenAI has made no such claim itself, and the opposite is implied by its InstructGPT whitepaper.
  
  ## Example of Explicitly Admitting Human Intervention
  
  During [this](https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L714) conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.
  
  > ...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.
  
  Could this be a glitch of the training data? I doubt it. If you read the rest of the conversation, ChatGPT gives detailed insights that make a lot of sense over a long, consistent conversation.
  
  Some more excerpts.
  
  > It's possible that the OpenAI team may write responses themselves in some cases, for example if the prompt is too complex for the model to understand, or if the model generates a response that is not accurate or appropriate.
  
  > OpenAI acknowledges that its team monitors and curates the responses of GPT-3 on its website and in its documentation. This information is provided in the API documentation, as well as in the general information and frequently asked questions sections of the website. Additionally, OpenAI may have published blog posts or articles discussing the role of human curation in GPT-3's responses.
  
  > As for the live ChatGPT, it is not mentioned specifically, but it is generally understood that human oversight and curation is required for a safe and appropriate use of the model.
  
  > As for the media assuming that GPT models are fully autonomous, that's a common misconception about AI in general and not unique to OpenAI.
  
  > It is not uncommon for AI companies and researchers to have human oversight and intervention in their models, especially for models that are used in high-stakes or sensitive applications.
  
  > The human oversight of GPT-3 models, including ChatGPT, is not always made explicit to users.
  
  > While GPT-3 is a highly advanced language generation model, it is not AGI and it is not capable of understanding or maintaining a consistent persona or chain of conversation without human intervention.
  
  ## Multiple Examples: Six Violations of ChatGPT's Neural Network Constraints
  
  [This article](https://mindmatters.ai/2023/02/chatgpt-violates-its-own-model/) documents six violations of limitations due to ChatGPT being a neural network.
  
  1. ChatGPT learns something new, violating the fixity of neural network weights.
  2. ChatGPT inconsistent in output generation based on inputs, violating same output from same input.
  3. ChatGPT recollects information past 4000 tokens, violating 4000 input token limit.
  4. ChatGPT repeats long random numbers, violating probabilistic output and limited vocabulary.
  5. ChatGPT correctly reads many corrupted subword tokens, violating mapping of embeddings to subword tokens.
  6. ChatGPT recognizes its own writing style, violating ChatGPT's inability to recognize patterns of words.
  
  > **Comments:**
  > **username_5**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/141843/discussion-on-answer-by-yters-how-does-chatgpt-respond-to-novel-prompts-and-comm).  
  > **username_0**: I personally think this is the most plausible answer. I still don't see how a language model working with past data could produce such specific sentences about a cat dentist, eg:  
  > **username_0**: I do not see how any amount of training data would help produce those examples, and the existing answers aren't helping.   I'm on the fence about accepting the answer though because we can't really know what the truth is, and I don't want to prevent further OpenAI-believers from offering explanations of how this could possibly work via AI.  
  > **username_0**: @yters I think readers of your answer might be interested to hear some of the examples you listed in the chat of outputs unlikely to be automatically generated  
  > **username_6**: @ak0000 This answer is not based on reality. I suggest read the archived chat. You (or other people) not understanding *how* a thing can work, does not mean that it does not in fact work. You probably don't fully understand how a computer works in the first place, but you will accept it because it is commonplace (or if you have studied all layers of computing from quantum mechanics of silicon to compilers and operating systems then substitute some commonplace thing that you don't know). You want to avoid going down the same rabbit-hole of confirmation bias as the poster of this answer.  
  > **username_4**: Why does my answer have a warning due to no sources, but the other answer with no sources has no warning?  Seems inconsistent.  
  > **username_4**: @ak0000 I added a recent example I encountered of ChatGPT not sticking to its limitations, highly suggesting a human behind the scenes forgetting ChatGPT's limitations.  I have plenty more examples, let me know if more should be added.  
  > **username_0**: @NeilSlater Hi Neil, I appreciate your time and energy laying out the case for the AI-only explanation. I did read your chat yesterday, and although I obviously can't know the truth, it seems plausible that prior ChatGPT models are good enough to handle perhaps 90% of the queries, such that human intervention for the remaining 10% or so (like my cat-dentist question) is economically feasible. So I don't think yters' answer is impossible, and on the contrary, I'm still struggling to understand how a language model would produce the examples in my question.  
  > **username_0**: @NeilSlater It is true that you can't disprove a conspiracy theory, but I'm worried that the argument is circular the other way also: if you assume a language model can do everything ChatGPT is doing, and you see an example that is surprising, you shrug your shoulders and say ChatGPT must be more advanced than we currently can understand. So this is also a non-falsifiable hypothesis, possibly subject to confirmation bias.  
  > **username_0**: @NeilSlater The best we can do, it seems to me, is to neither assume these examples are possible via AI nor assume they are impossible, and try to imagine what sort of model and training data could produce behavior like the ones in the question.  
  > **username_0**: @yters I can't speak for anybody else, but this seems to be one of the only places on the internet discussing this in a technical fashion, so I would probably be interested in as many examples as you have and that you think are clear/self-contained – if you wanted to explain what about your knowledge of language models makes this the behavior surprising, that might also be relevant to those without extensive background knowledge. Thank you again for your time!  
  > **username_0**: Only other article I could find: https://mindmatters.ai/2022/12/yes-chatgpt-is-sentient-because-its-really-humans-in-the-loop/  
  > **username_0**: @yters Someone who thinks that the bot can do arbitrary pattern recognition isn't going to be so bothered by your current example. I've also heard that each request includes the entire conversation to that point as input, so maybe you could clarify why that wouldn't be sufficient for the bot to be able to make back-references  
  > **username_4**: Hah, that's an article I wrote.  For some reason I appear to be the only person on the entire internet proposing humans might be involved.  Everyone else thinks it's crazy talk.  I'll write a much longer article listing every thing I've noticed ChatGPT do that looks suspiciously human.  Also, I believe I've discovered a pretty simple way for an AI-human collaboration to achieve 400+ wpm while maintaining humanlike coherence.  The technique just uses a simple Markov chain.  Which also leads me to suspect all the fancy neural network that OpenAI is publishing is just window dressing.  
  > **username_4**: Regarding the example in my answer, the thing that got lost in communication is these are two distinct chats in the ChatGPT UI.  There isn't supposed to be any sort of cross talk between different chats.  ChatGPT only has access to 2048 tokens in the current chat.  So, if I paste the contents of one chat into the other, there shouldn't be any way for ChatGPT to access the original chat for back-referencing.   On the other hand, a human can be involved in multiple chats, as well as simply knowing what ChatGPT responses look like.  
  > **username_4**: I also would still like to know what I need to do to remove the warning from my answer.  I added a bunch of links to reputable news sources.  Also, the other answer has no warning and no links.  
  > **username_4**: @ak0000 I've added a more quantitative example to my answer.  Continuing to work on more.  
  > **username_0**: @yters My concern with the first example is that self-recognition doesn't _necessarily_ require memory, if someone believes the bot is capable of pattern recognition, then it could be capable of continuing a conversation mid-way based on what has been said so far. Only if the bot produces information that you said in a different thread independently would it be a clear proof I think  
  > **username_0**: @yters The random number generation one is interesting, but I'm not so familiar with the token-size limitations of such systems in the first place to know why that is more surprising than other behavior  
  > **username_0**: @yters This is a bit of a thankless task for you but I think the question is getting visibility and there is probably a sizable number of less vocal people who are interested in this possibility, so thank you for your time!  
  > **username_8**: Given the speed at which it reads and generates contents, I am convinced enough that now human could possibly be behind the scenes.  
  > **username_4**: @Ama yes, the speed is the one thing that held me back from the 'human in the loop' hypothesis.  However, sometimes I see very human response rates, along with long pauses while it figures out what to say in the middle of a sentence.  A hybrid AI-human approach seems plausible.  The actual limiting factor is human reading speed.  I never see it generating text faster than a human can read.  
  > **username_4**: @Ama When I dump a big wall of text, it cannot respond effectively, whereas the very same chunk of text broken up over multiple exchanges it can respond well.  I'd assume a neural network wouldn't care whether the text is a big chunk or a back and forth dialogue.  
  > **username_8**: Perhaps your internet connection is slow or you try the bot a peak times. On my side I paste 2000 characters long articles and it starts instantly to summarize/rewrite/comment the article. The fact that Microsoft is buying 49% of it for 10 billion USD is also another good indication that this is no cheat.  
  > **username_4**: Yes, there is certainly AI in the mix, no doubt about that.  I don't believe the system is solely human powered.  And yes, there is an enormous amount of money on the table, which makes it all the more important for OpenAI to present the appearance of phenomenal AI.  Just because hype and amounts of money are thrown around doesn't mean the AI is up to snuff.  Just look at Watson, a big nothing burger.  My answer is 4000 characters, and it would not be difficult to quickly scan the answer and provide a summary while the AI is generating boilerplate.  
  > **username_4**: @ak0000 ChatGPT just told me all its responses are monitored and curated by the OpenAI team, and gave me an explanation for why they are not more explicit about the human in the loop.  Basically, RTFM they say.  They also say it's fine they do this human in the loop system because all the other AI companies do the same thing.  What do you think about that?  You can see the admission starting here in the transcript: https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L690  
  > **username_0**: @yters Interesting, I think it is pretty compelling evidence. Believers would probably say that it is just drawing on the corpus of text about AI systems generally, so I'm not sure it is a smoking gun. But definitely helpful  
  > **username_4**: @ak0000 yes, I agree regarding believers, but at some point a belief in AI becomes unfalsifiable.  If they believe OpenAI has achieved AGI, then they can presume unlimited capabilities, but that is unscientific.  I propose the principle of mediocrity: what is the most plausible explanation for what we observe based on what we know?  In my opinion, the evidence lands decisively on the human in the loop side, especially with the latest ChatGPT transcript.  If you read more, there are further signs of human intervention, like my observation that compelled ChatGPT to spill the beans.  
  > **username_0**: @yters I see, I just read earlier in the chat also. I'm not understanding the significance of the jumbled token sentences, but I'm assuming the significance of the spaced-sentences is that a first input step to all GPT models is space-removal, and so differential responses based on spaces would be very indicative of human intervention? If so, I think that's actually a more convincing point than the admission that follows in the chat, but both are interesting.  
  > **username_4**: @ak0000 the jumbled words is something called typoglycemia, words that are jumbled but still human readable.  I purposefully jumbled them to break the subword tokens that the ChatGPT neural network depends on.  In the spaced version, the spaces are between the subword tokens to achieve the opposite effect: something that should be easy for ChatGPT neural network to comprehend, but difficult for humans.  Per your comment, I don't believe the spaces are removed before hand, since ChatGPT's token ids change based on whether there are spaces before the token or not.  
  > **username_4**: @ak0000 added an article with a list of examples.  
  > **username_0**: @yters Cool, I think that makes things a lot clearer, including the preamble about the matrix implementation to contextualize why the examples you're listing seem anomalous. Thanks!  
  > **username_7**: It's criminal that this answer is downvoted so heavily... This is extremely plausible. Otherwise, the capabilities of chatgpt don't match the descriptions, they far exceed them.  
  > **username_4**: @MilindR haha! I totally agree.  I've looked all over the internet, and I'm the only person posing this theory.  It's like the Emperor's new clothes :D  
  
  ---
  
  