Answers.Body: <p>Turns out ChatGPT is indeed human curated, by open admission.</p>
  <p>During <a href="https://github.com/yters/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L714" rel="nofollow noreferrer">this</a> conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.</p>
  <blockquote>
  <p>...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.</p>
  </blockquote>
  <p>Apparently, the fact that OpenAI actively curates ChatGPT's responses is indirectly implied in the documentation <a href="https://beta.openai.com/docs/guides/safety-best-practices/human-in-the-loop-hitl" rel="nofollow noreferrer">here</a>.</p>
  <blockquote>
  <p>Human in the loop (HITL):
  Wherever possible, we recommend having a
  human review outputs before they are used in practice. This is
  especially critical in high-stakes domains, and for code generation.
  Humans should be aware of the limitations of the system, and have
  access to any information needed to verify the outputs (for example,
  if the application summarizes notes, a human should have easy access
  to the original notes to refer back).</p>
  </blockquote>
  <p>So, that explains that :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:32:52.470Z
Answers.Text: @yters **You** are the one begging the question since **you** are the one assuming the chatbot is human since **you** are using the chatbots response which **you** assume to be human to **you** use to prove that it is human.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:36:13.007Z
Answers.Text: @yters I would also say you are treating neuroscience as a magical black box capable of anything. I would encourage you to research how neurons work, and then ask yourself what is the likelihood the biological associations in a neuron network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team? As compared to the theory put forth that your brain doesn't do well over a long conversation, and needs alien intervention to stay on course?
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:40:31.143Z
Answers.Text: Additionally, why does ChatGPT fail many human tasks as "Please type this sentence with the letters reversed?" You base your argument on circular logic and indirect implications, ignoring facts such as ChatGPT's failure to do tasks easily accomplishable for a human being and facts such as the amount of workload ChatGPT received not manageable by the OpenAI team. The team must have been curating most of the chats for repeated tests to work.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:43:55.653Z
Answers.Text: We have debunked your transcript, and your documentation does not provide any evidence or "indirect implication" for your argument. Please consider doing some research, such as how neural networks work and the size of the neural network in use.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:47:41.387Z
Answers.Text: You have biased ChatGPT in your messages, and ChatGPT is known for making errors. The GPT generated response could easily have been an error. Additionally, you have p-hacked (cherrypicked) your argument. Multiple times in the transcript, and I haven't even read all of them, ChatGPT says that: **GPT-3 does not rely on humans to understand the sentences**, **It's important to note that the text generated by GPT-3 is not created by a human but by the model**, etc.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:49:55.417Z
Answers.Text: Meanwhile, you have confused ChatGPT and biased it by repeatedly stating that **So if the extra spaces confuse a human, but not a neural network, what does that imply when ChatGPT gets confused by extra spaces?**, **Does GPT-3 rely on humans to understand the sentences?**, **So there are two possibilities I see: the ChatGPT has very inconsistent responses to sentences with extra spaces, or humans are getting confused, which seems most reasonable?**, **what I mean is there are humans reading these requests and getting confused while writing responses for ChatGPT**
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:51:56.477Z
Answers.Text: **So how do the humans write responses so quickly for ChatGPT?**, **But you just said that humans are writing responses for GPT-3 and get confused by the text with extra spaces.**, etc.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-26T08:56:15.063Z
Answers.Text: Additionally, it is your responsibility to maintain and cite readable sources, instead of commanding people to read your entire 100KB transcript which is 1000 lines long. I hope you can do some research, be logical, and be realistic, instead of writing non-sensical arguments trying to spread your logically unsound belief without or with very little useable sources.
Answers.UserDisplayName: Sny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T10:36:18.030Z
Answers.Text: The HITL section does _not_ imply that this is what they do with ChatGPT. Please, stop spreading your conspiracy theories without _solid_ evidence.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T10:47:03.943Z
Answers.Text: @DavidIreland did you read the transcript I linked in this answer?  It's pretty clear.  I just added a small excerpt to give the gist.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T11:18:19.667Z
Answers.Text: ChatGPT has no awareness or anything like that of concepts such as "true" and "false". It does not truly understand your questions and there are no guarantees that it provides truthful answers. It just produces text that is statistically likely, based on its training data. So, if you ask it whether a human wrote its output, and it responds saying that this indeed happened... this gives us **absolutely no** new information. It may or may not be true. But this cannot be a basis of your answer.
  
  Your second link is a set of Safety Best Practices, it is not a statement of what they actually did.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T16:49:39.330Z
Answers.Text: @DennisSoemers did you read the transcript?  It makes a lot of sense, and ChatGPT volunteers a fair amount of unsolicited information.  Also, look at what precedes the confession.  The chatbot confessed to being human because I identified a clear pattern that indicated human involvement, and I called them on it.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T18:35:29.387Z
Answers.Text: @yters As I explained in the previous comment, it "confessing" anything is meaningless. It does not know what is true and what is not true, and has no self-awareness. You can probably get it to confess anything you like, **especially** with leading prompts where you're basically first telling it that its output is generated by a human. I've seen lots of examples where humans tell the bot that its output is incorrect (even if it actually was correct), and it will consistently apologise and agree.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T19:41:17.063Z
Answers.Text: @DennisSoemers you are begging the question assuming the chatbot is just AI.  The scientific approach is to weigh each hypothesis on its own merits based on the evidence, instead of a priori biasing towards one hypothesis.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T19:47:24.107Z
Answers.Text: @DennisSoemers  I would also say you are treating the neural network as a magical black box capable of anything.  I would encourage you to research how neural networks work, and then ask yourself what is the likelihood the statistical associations in a neural network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team?  As compared to the theory put forth in our conversation that ChatGPT's neural network doesn't do well over a long conversation, and needs human intervention to stay on course?
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-16T13:07:07.313Z
Answers.LastActivityDate: 2023-01-16T13:12:37.450Z
Answers.LastEditDate: 2023-01-16T13:12:37.450Z
Answers.UserDisplayName: yters
Answers.Body: <p>As well as decent modelling of purposes or strings of digits, ChatGPT can identify when a completely novel string might be a product name, a fictional person or place etc.</p>
  <p>The language model in ChatGPT is partially character based. There are around 100,000 tokens in latest GPT - the encoding is called <code>c100k_base</code>. Most of them are common English words (e.g. &quot;the&quot;), but it also supports individual characters including individual digits, and processes sequences of characters and effectively models groups of them at a time.</p>
  <blockquote>
  <p>From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.</p>
  </blockquote>
  <p>That's a reasonable analogy. ChatGPT is a bit like a 2048-gram prediction engine for the next token, shift the sequence along one and repeat. No different to toy fantasy name generators when viewed from 10,000 feet up.</p>
  <blockquote>
  <p>As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.</p>
  </blockquote>
  <p>A couple of misunderstandings here. First, the random number will not become a single token, but will be one token per digit, or pair of digits or triple digits, depensing on sequence - <a href="https://platform.openai.com/tokenizer" rel="nofollow noreferrer">you can give this a try to help visualise it</a>, in the encoding that ChatGPT uses. Of course each of those tokens will have been seen before, millions of times in the training data.</p>
  <p>Second, sequences do not need to be seen in the training data in order for ChatGPT to work with them. In fact, with an input sequence length of 2048, pretty much <em>all</em> inputs to ChatGPT in inference mode are unique never-seen-before sequences. Regardless if some of the tokens represent a long random number, the chances of any 2048 long sequence of letters and short words being unique when generated are very high.</p>
  <p>This is where the neural network model differs from a true 2048-gram. It has generalised from the training data well enough that it actually <em>can</em> predict meaningful and useful values for probability of next token, even though in all likelihood it has never before seen the exact same sequence. In this regard it is an approximation of a &quot;perfect&quot; 2048-gram prediction engine that somehow been trained on infinite human writings.</p>
  <p>A lot of language modelling is about correctly processing the <em>context</em> of a subsequence, so recognising a number sequence as being a grammatical &quot;unit&quot; that can be reused as-is is not a surprising feature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T17:40:34.963Z
Answers.Text: Can you expand on how a neural network is able to recognize a grammatical unit and then also repeat it?  My understanding of a generative neural network is it doesn't repeat anything in the input.  It is a big network of statistical associations, and is actually identical to a Markov chain, and outputs the terms statistically associated with the input.  So, in order for it to output the input, there has to be a specific statistical entry that associates input "repeat number X" with output "X".  That's only possible if such entry is in its training, and that cannot happen with random numbers.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T22:53:41.313Z
Answers.Text: @yters The tokens in the "Markov chain" here are down to character level, and one big difference between a Markov chain trained on word frequency and a neural network is the the neural network will generalise and be able to assign frequencies to sequences that have never been observed before, due to analogous sequences seen in training data. This is what allows a large language model like ChatGPT to manipulate never-before-seen sequences, where a frequency-based Markov chain cannot, (an actual Markov chain model would also be seriously constrained by sequence length)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T23:21:55.240Z
Answers.Text: Neural networks only generalize in a very shallow sense, like with convolutional layers, they essentially create a statistical model based on smaller features in the image, instead of the image as a whole.  In this case, it'd be finding features based on letter and word combinations.  I don't see how this can generalize to the level where we have an imperative grammar and the neural network is able to understand the command 'repeat this' in many different representations.  Do you have details on how this command generalization occurs in neural networks?
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T23:32:31.290Z
Answers.Text: Also, neural networks are a Markov process.  The definition of a Markov process is where the future is independent of the past given the present.  The GPT models have a limited horizon of 2048 tokens, and have no memory.  This fits the definition of a Markov process.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T23:45:49.557Z
Answers.Text: @yters: Production of next token is a Markov process yes. Perhaps I was confused assuming you were talking about an ngrams process because you made the (incorrect) claim that any repeated/understood sequence somehow had to be in the training data. The ngram is an interesting analogy. The neural networks are approximating (and generalising to) an imaginary 2048-gram with perfect knowledge of all possible human discourse.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T23:47:13.737Z
Answers.Text: Unlike the 2048-gram, the neural network really can predict from and work with sequences that it has not seen before by analogy to similar sequences that it *has* seen in the training data. This makes it an approximation of an imaginary perfect 2048-gram that somehow been trained on all valid text output ever. That includes 2048-grams that include the input random number entered by a human and corresponding copy (or manipulated copy) of it output by the chat. The difference between a LLM and one you or I can train is that it will actually have some sensible probabilities for that
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-06T01:06:18.223Z
Answers.Text: Not that the exact phrase had to be in the markov chain, but there is an equivalent high probability transition that would go from 'repeat X' to generating 'X', where X is a long random number.  This seems highly implausible for a neural network.  I can see a rule based ELIZA type system implementing such a thing, but it doesn't make sense OpenAI would add such a niche rule to their system.  Also, the ability of ChatGPT to 'repeat X' is very flexible representation wise.  I can write the request in many different ways, and break X up, and ChatGPT can still repeat X with complete accuracy.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-06T01:08:11.827Z
Answers.Text: On the other hand, ChatGPT has trouble exactly where I'd expect a human to have trouble, when I asked it to 'reverse X'.  It does so very inaccurately, just like a human would if they tried reversing a long random number by hand very quickly.
Answers.UserDisplayName: yters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-06T20:29:07.350Z
Answers.Text: What do you mean by "There are around 2000 tokens in GPT"? Is this really true - hard to imagine.
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-06T20:53:41.063Z
Answers.Text: @Hans-PeterStricker: Yes it is true. It is a hybrid system for efficiency.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-06T21:16:47.663Z
Answers.Text: Thanks. Where can I find an explicit list of these tokens? Do you have a reference?
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-06T21:27:08.947Z
Answers.Text: Are we talking about the same tokens OpenAI's tokenizer counts? https://platform.openai.com/tokenizer
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T09:07:18.157Z
Answers.Text: @Hans-PeterStricker Yes. At least for GPT-3 and ChatGPT based on GPT-3 - https://blog.quickchat.ai/post/tokens-entropy-question/  Thanks for the link BTW, added it to the answer.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T09:08:22.700Z
Answers.Text: @Hans-PeterStricker It looks like my quoted numbers are for an earlier version of GPT-3, and that the later versions have a vocab of around 50,000.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T09:18:37.140Z
Answers.Text: @Hans-PeterStricker Took a while to find the raw list for GPT-4 (also same for GPT 3.5 used in ChatGPT I think), but here it is: https://github.com/kaisugi/gpt4_vocab_list/blob/main/cl100k_base_vocab_list.txt - this one has 100k tokens, and my original assertion of 2000 tokens looks very old, but I did get that from a published source (it just must of been a about a different model)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-07T09:35:08.950Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/145840/discussion-between-hans-peter-stricker-and-neil-slater).
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-05T11:06:15.017Z
Answers.LastActivityDate: 2023-05-07T09:20:49.453Z
Answers.LastEditDate: 2023-05-07T09:20:49.453Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Existing answer is great about model generalization, but I would like to add about an important <a href="https://en.wikipedia.org/wiki/Inductive_bias" rel="nofollow noreferrer">inductive bias</a> of the <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="nofollow noreferrer">Transformer</a> model architecture used for ChatGPT.</p>
  <p>In the Transformer model architecture, there is a mechanism called <em>attention</em>. An attention block in a model can access all context (input and previously-generated output) and reminder what is needed based on the model's choice.
  A partial analogy of the attention mechanism is how humans' eye move to some previous text while writing.</p>
  <p>Due the attention mechanism, a Transformer model is very good at repeating the input as is to the output. This is different from previous architectures such as <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" rel="nofollow noreferrer">RNN</a>, where the model has to convert the input to their internal representation.</p>
  <p>That being said, the provided example is something more than a mere repeat, and it is very difficult to say &quot;how&quot; a large language model performs in general, especially as ChatGPT model is not public.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T11:43:50.863Z
Answers.LastActivityDate: 2023-03-13T11:43:50.863Z
Answers.UserDisplayName: pcpthm
Body: <p>From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.  As such, it should only be able to generate tokens that are in its training dataset.  One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.</p>
  <p>However, <a href="https://mindmatters.ai/2023/01/is-chatgpt-solely-a-neural-network-i-tested-that/" rel="nofollow noreferrer">ChatGPT very easily, and with complete accuracy, repeats many long random numbers I give it</a>.  I also split up the random number across my request, and have ChatGPT piece it together again.</p>
  <blockquote>
  <p>Eric Holloway: How about concatenating these numbers, and removing the
  intermittent filler words? 9475 3008 3305 11 frog 4357 5075 0021 3385
  0894 2324 3812 9336 29 dog 52 1375 8236 3840 5500 6080 3841 log 7266
  0722 9112 9774 4967 4707 81</p>
  <p>ChatGPT: To concatenate the numbers you have provided, while removing
  the filler words “frog”, “dog”, and “log”, you can simply combine the
  numbers in the order that they are given. Here is the resulting
  number: 9475 3008 3305 1143 5750 7500 2133 8508 9423 2438 1293 3629
  5213 7582 3638 4055 0060 8038 4172 6607 2291 1297 7449 6747 0781</p>
  </blockquote>
  <p>How is a neural network able to do this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-26T08:59:31.047Z
Comments.Text: Please be reasonable. **Do not accept your own answer when it is clear that others do not agree and your answer is wrong.** Refer to my comments underneath your answer.
Comments.UserDisplayName: Sny
Tags: <neural-networks><chat-bots><chatgpt><randomness>
Title: How is ChatGPT able to repeat random numbers?
UserDisplayName: yters
ThreadText: # How is ChatGPT able to repeat random numbers?
  
  (Asked by: username_0 on 2023-01-05)
  
  From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator. As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.
  
  However, [ChatGPT very easily, and with complete accuracy, repeats many long random numbers I give it](https://mindmatters.ai/2023/01/is-chatgpt-solely-a-neural-network-i-tested-that/). I also split up the random number across my request, and have ChatGPT piece it together again.
  
  > Eric Holloway: How about concatenating these numbers, and removing the
  > intermittent filler words? 9475 3008 3305 11 frog 4357 5075 0021 3385
  > 0894 2324 3812 9336 29 dog 52 1375 8236 3840 5500 6080 3841 log 7266
  > 0722 9112 9774 4967 4707 81
  >
  > ChatGPT: To concatenate the numbers you have provided, while removing
  > the filler words “frog”, “dog”, and “log”, you can simply combine the
  > numbers in the order that they are given. Here is the resulting
  > number: 9475 3008 3305 1143 5750 7500 2133 8508 9423 2438 1293 3629
  > 5213 7582 3638 4055 0060 8038 4172 6607 2291 1297 7449 6747 0781
  
  How is a neural network able to do this?
  
  **Tags:** neural-networks, chat-bots, chatgpt, randomness
  
  > **Comments:**
  > **username_1**: Please be reasonable. **Do not accept your own answer when it is clear that others do not agree and your answer is wrong.** Refer to my comments underneath your answer.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2023-01-16. Score: -6 | Accepted answer
  
  Turns out ChatGPT is indeed human curated, by open admission.
  
  During [this](https://github.com/username_0/transcripts/blob/main/ChatGPT_confused_by_visual_sentence_structure.txt#L714) conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.
  
  > ...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.
  
  Apparently, the fact that OpenAI actively curates ChatGPT's responses is indirectly implied in the documentation [here](https://beta.openai.com/docs/guides/safety-best-practices/human-in-the-loop-hitl).
  
  > Human in the loop (HITL):
  > Wherever possible, we recommend having a
  > human review outputs before they are used in practice. This is
  > especially critical in high-stakes domains, and for code generation.
  > Humans should be aware of the limitations of the system, and have
  > access to any information needed to verify the outputs (for example,
  > if the application summarizes notes, a human should have easy access
  > to the original notes to refer back).
  
  So, that explains that :)
  
  > **Comments:**
  > **username_2**: The HITL section does _not_ imply that this is what they do with ChatGPT. Please, stop spreading your conspiracy theories without _solid_ evidence.  
  > **username_0**: @DavidIreland did you read the transcript I linked in this answer?  It's pretty clear.  I just added a small excerpt to give the gist.  
  > **username_3**: ChatGPT has no awareness or anything like that of concepts such as "true" and "false". It does not truly understand your questions and there are no guarantees that it provides truthful answers. It just produces text that is statistically likely, based on its training data. So, if you ask it whether a human wrote its output, and it responds saying that this indeed happened... this gives us **absolutely no** new information. It may or may not be true. But this cannot be a basis of your answer.  Your second link is a set of Safety Best Practices, it is not a statement of what they actually did.  
  > **username_0**: @username_3 did you read the transcript?  It makes a lot of sense, and ChatGPT volunteers a fair amount of unsolicited information.  Also, look at what precedes the confession.  The chatbot confessed to being human because I identified a clear pattern that indicated human involvement, and I called them on it.  
  > **username_3**: @username_0 As I explained in the previous comment, it "confessing" anything is meaningless. It does not know what is true and what is not true, and has no self-awareness. You can probably get it to confess anything you like, **especially** with leading prompts where you're basically first telling it that its output is generated by a human. I've seen lots of examples where humans tell the bot that its output is incorrect (even if it actually was correct), and it will consistently apologise and agree.  
  > **username_0**: @username_3 you are begging the question assuming the chatbot is just AI.  The scientific approach is to weigh each hypothesis on its own merits based on the evidence, instead of a priori biasing towards one hypothesis.  
  > **username_0**: @username_3  I would also say you are treating the neural network as a magical black box capable of anything.  I would encourage you to research how neural networks work, and then ask yourself what is the likelihood the statistical associations in a neural network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team?  As compared to the theory put forth in our conversation that ChatGPT's neural network doesn't do well over a long conversation, and needs human intervention to stay on course?  
  > **username_1**: @username_0 **You** are the one begging the question since **you** are the one assuming the chatbot is human since **you** are using the chatbots response which **you** assume to be human to **you** use to prove that it is human.  
  > **username_1**: @username_0 I would also say you are treating neuroscience as a magical black box capable of anything. I would encourage you to research how neurons work, and then ask yourself what is the likelihood the biological associations in a neuron network are capable of reproducing a very long, consistent, off the cuff conversation about ChatGPT being curated by the OpenAI team? As compared to the theory put forth that your brain doesn't do well over a long conversation, and needs alien intervention to stay on course?  
  > **username_1**: Additionally, why does ChatGPT fail many human tasks as "Please type this sentence with the letters reversed?" You base your argument on circular logic and indirect implications, ignoring facts such as ChatGPT's failure to do tasks easily accomplishable for a human being and facts such as the amount of workload ChatGPT received not manageable by the OpenAI team. The team must have been curating most of the chats for repeated tests to work.  
  > **username_1**: We have debunked your transcript, and your documentation does not provide any evidence or "indirect implication" for your argument. Please consider doing some research, such as how neural networks work and the size of the neural network in use.  
  > **username_1**: You have biased ChatGPT in your messages, and ChatGPT is known for making errors. The GPT generated response could easily have been an error. Additionally, you have p-hacked (cherrypicked) your argument. Multiple times in the transcript, and I haven't even read all of them, ChatGPT says that: **GPT-3 does not rely on humans to understand the sentences**, **It's important to note that the text generated by GPT-3 is not created by a human but by the model**, etc.  
  > **username_1**: Meanwhile, you have confused ChatGPT and biased it by repeatedly stating that **So if the extra spaces confuse a human, but not a neural network, what does that imply when ChatGPT gets confused by extra spaces?**, **Does GPT-3 rely on humans to understand the sentences?**, **So there are two possibilities I see: the ChatGPT has very inconsistent responses to sentences with extra spaces, or humans are getting confused, which seems most reasonable?**, **what I mean is there are humans reading these requests and getting confused while writing responses for ChatGPT**  
  > **username_1**: **So how do the humans write responses so quickly for ChatGPT?**, **But you just said that humans are writing responses for GPT-3 and get confused by the text with extra spaces.**, etc.  
  > **username_1**: Additionally, it is your responsibility to maintain and cite readable sources, instead of commanding people to read your entire 100KB transcript which is 1000 lines long. I hope you can do some research, be logical, and be realistic, instead of writing non-sensical arguments trying to spread your logically unsound belief without or with very little useable sources.  
  
  ---
  
  ### Answer by username_4 on 2023-01-05. Score: 10
  
  As well as decent modelling of purposes or strings of digits, ChatGPT can identify when a completely novel string might be a product name, a fictional person or place etc.
  
  The language model in ChatGPT is partially character based. There are around 100,000 tokens in latest GPT - the encoding is called `c100k_base`. Most of them are common English words (e.g. "the"), but it also supports individual characters including individual digits, and processes sequences of characters and effectively models groups of them at a time.
  
  > From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.
  
  That's a reasonable analogy. ChatGPT is a bit like a 2048-gram prediction engine for the next token, shift the sequence along one and repeat. No different to toy fantasy name generators when viewed from 10,000 feet up.
  
  > As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.
  
  A couple of misunderstandings here. First, the random number will not become a single token, but will be one token per digit, or pair of digits or triple digits, depensing on sequence - [you can give this a try to help visualise it](https://platform.openai.com/tokenizer), in the encoding that ChatGPT uses. Of course each of those tokens will have been seen before, millions of times in the training data.
  
  Second, sequences do not need to be seen in the training data in order for ChatGPT to work with them. In fact, with an input sequence length of 2048, pretty much *all* inputs to ChatGPT in inference mode are unique never-seen-before sequences. Regardless if some of the tokens represent a long random number, the chances of any 2048 long sequence of letters and short words being unique when generated are very high.
  
  This is where the neural network model differs from a true 2048-gram. It has generalised from the training data well enough that it actually *can* predict meaningful and useful values for probability of next token, even though in all likelihood it has never before seen the exact same sequence. In this regard it is an approximation of a "perfect" 2048-gram prediction engine that somehow been trained on infinite human writings.
  
  A lot of language modelling is about correctly processing the *context* of a subsequence, so recognising a number sequence as being a grammatical "unit" that can be reused as-is is not a surprising feature.
  
  > **Comments:**
  > **username_0**: Can you expand on how a neural network is able to recognize a grammatical unit and then also repeat it?  My understanding of a generative neural network is it doesn't repeat anything in the input.  It is a big network of statistical associations, and is actually identical to a Markov chain, and outputs the terms statistically associated with the input.  So, in order for it to output the input, there has to be a specific statistical entry that associates input "repeat number X" with output "X".  That's only possible if such entry is in its training, and that cannot happen with random numbers.  
  > **username_4**: @username_0 The tokens in the "Markov chain" here are down to character level, and one big difference between a Markov chain trained on word frequency and a neural network is the the neural network will generalise and be able to assign frequencies to sequences that have never been observed before, due to analogous sequences seen in training data. This is what allows a large language model like ChatGPT to manipulate never-before-seen sequences, where a frequency-based Markov chain cannot, (an actual Markov chain model would also be seriously constrained by sequence length)  
  > **username_0**: Neural networks only generalize in a very shallow sense, like with convolutional layers, they essentially create a statistical model based on smaller features in the image, instead of the image as a whole.  In this case, it'd be finding features based on letter and word combinations.  I don't see how this can generalize to the level where we have an imperative grammar and the neural network is able to understand the command 'repeat this' in many different representations.  Do you have details on how this command generalization occurs in neural networks?  
  > **username_0**: Also, neural networks are a Markov process.  The definition of a Markov process is where the future is independent of the past given the present.  The GPT models have a limited horizon of 2048 tokens, and have no memory.  This fits the definition of a Markov process.  
  > **username_4**: @username_0: Production of next token is a Markov process yes. Perhaps I was confused assuming you were talking about an ngrams process because you made the (incorrect) claim that any repeated/understood sequence somehow had to be in the training data. The ngram is an interesting analogy. The neural networks are approximating (and generalising to) an imaginary 2048-gram with perfect knowledge of all possible human discourse.  
  > **username_4**: Unlike the 2048-gram, the neural network really can predict from and work with sequences that it has not seen before by analogy to similar sequences that it *has* seen in the training data. This makes it an approximation of an imaginary perfect 2048-gram that somehow been trained on all valid text output ever. That includes 2048-grams that include the input random number entered by a human and corresponding copy (or manipulated copy) of it output by the chat. The difference between a LLM and one you or I can train is that it will actually have some sensible probabilities for that  
  > **username_0**: Not that the exact phrase had to be in the markov chain, but there is an equivalent high probability transition that would go from 'repeat X' to generating 'X', where X is a long random number.  This seems highly implausible for a neural network.  I can see a rule based ELIZA type system implementing such a thing, but it doesn't make sense OpenAI would add such a niche rule to their system.  Also, the ability of ChatGPT to 'repeat X' is very flexible representation wise.  I can write the request in many different ways, and break X up, and ChatGPT can still repeat X with complete accuracy.  
  > **username_0**: On the other hand, ChatGPT has trouble exactly where I'd expect a human to have trouble, when I asked it to 'reverse X'.  It does so very inaccurately, just like a human would if they tried reversing a long random number by hand very quickly.  
  > **username_5**: What do you mean by "There are around 2000 tokens in GPT"? Is this really true - hard to imagine.  
  > **username_4**: @username_5: Yes it is true. It is a hybrid system for efficiency.  
  > **username_5**: Thanks. Where can I find an explicit list of these tokens? Do you have a reference?  
  > **username_5**: Are we talking about the same tokens OpenAI's tokenizer counts? https://platform.openai.com/tokenizer  
  > **username_4**: @username_5 Yes. At least for GPT-3 and ChatGPT based on GPT-3 - https://blog.quickchat.ai/post/tokens-entropy-question/  Thanks for the link BTW, added it to the answer.  
  > **username_4**: @username_5 It looks like my quoted numbers are for an earlier version of GPT-3, and that the later versions have a vocab of around 50,000.  
  > **username_4**: @username_5 Took a while to find the raw list for GPT-4 (also same for GPT 3.5 used in ChatGPT I think), but here it is: https://github.com/kaisugi/gpt4_vocab_list/blob/main/cl100k_base_vocab_list.txt - this one has 100k tokens, and my original assertion of 2000 tokens looks very old, but I did get that from a published source (it just must of been a about a different model)  
  > **username_5**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/145840/discussion-between-username_5-username_5-and-username_4).  
  
  ---
  
  ### Answer by username_6 on 2023-03-13. Score: 2
  
  Existing answer is great about model generalization, but I would like to add about an important [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias) of the [Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) model architecture used for ChatGPT.
  
  In the Transformer model architecture, there is a mechanism called *attention*. An attention block in a model can access all context (input and previously-generated output) and reminder what is needed based on the model's choice.
  A partial analogy of the attention mechanism is how humans' eye move to some previous text while writing.
  
  Due the attention mechanism, a Transformer model is very good at repeating the input as is to the output. This is different from previous architectures such as [RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network), where the model has to convert the input to their internal representation.
  
  That being said, the provided example is something more than a mere repeat, and it is very difficult to say "how" a large language model performs in general, especially as ChatGPT model is not public.
  
  
  ---
  
  