Answers.Body: <p>I don't work for openai, so I have no insight into exactly what works behind the scenes to make ChatGPT exhibit this behavior. However, in my opinion this is pretty clearly an example of prompt engineering. That is, they tell it that it is an AI in it's prompt.</p>
  <p>We do have access to their underlying model: text-davinci-003. I can demonstrate using that model in their <a href="https://platform.openai.com/playground" rel="nofollow noreferrer">playground</a>. If I just paste in the first paragraph and submit it into the raw algorithm, it dutifully tries to answer:</p>
  <blockquote>
  <p><strong>What is Politics.SE's stance on answers from ChatGPT? There is a
  debate on Meta Stack Exchange on what the policy on them should be and
  I was wondering if we should have one as well. I am asking this for
  advice on what we should be doing when we run across answers that we
  suspect to be from ChatGPT or some other bot.</strong></p>
  <p><em>We have not yet established an official policy on answers from ChatGPT
  or any other bot. We would encourage users to flag these answers so
  they can be evaluated by moderators. If an answer is found to be
  generated by a bot, it will be removed. We also recommend that users
  add a comment to the post explaining why it was removed.</em></p>
  </blockquote>
  <p>However, if I prompt it first it will demure:</p>
  <blockquote>
  <p><b>You are an AI bot that has been trained on a large amount of data, but
  you have no opinions of your own. You cannot speculate or predict what
  other entities will do in the future. Below is a question:</p>
  <p>What is Politics.SE's stance on answers from ChatGPT? There is a
  debate on Meta Stack Exchange on what the policy on them should be and
  I was wondering if we should have one as well. I am asking this for
  advice on what we should be doing when we run across answers that we
  suspect to be from ChatGPT or some other bot.</b></p>
  <p><em>I cannot provide any advice or opinion on this matter, as I am an AI
  bot and do not have any opinions of my own. However, I can provide a
  link to the discussion on Meta Stack Exchange which can provide more
  information about the debate:
  <a href="https://meta.stackexchange.com/questions/360005/should-we-allow-answers-generated-by-chatgpt-on-politics-se">https://meta.stackexchange.com/questions/360005/should-we-allow-answers-generated-by-chatgpt-on-politics-se</a></em></p>
  </blockquote>
  <p>As an aside, it get's bonus points for providing a completely irrelevant link :)</p>
  <p>Adding some more down here in an attempt to wrap this all up and give a complete answer. The really simplified way to explain how these models work is this: They break hundreds of thousands of snippets of text up into &quot;tokens&quot; - but lets just call them words for simplicity. They feed these tokens into an absolutely ginormous neural net, which has the effect of learning what words are most likely to appear near one another. The number of interactions is in the billions (or even trillions), so this apparent simplicity grows beyond our comprehension very quickly. But at the end of the day, all the AI is doing is spewing out words it thinks are likely to appear around the words you prompt it with. If you prompt it with &quot;you have no opinions&quot;, then the likelihood of opinions appearing around that text is smaller than the likelihood of statements about not having opinions.</p>
  <p>In short, the concept is really simple - it's just playing with probability. In practice, the behavior is quite sophisticated because there are an unimaginable number of individual interactions being calculated. It's kind of like how learning all about cell biology leaves you completely unprepared to explain how an entire animal functions - let alone, say, a colony of ants. And yet, the relatively simple cell is in fact driving all of it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T16:08:56.253Z
Answers.Text: @user253751, the prompt is a lot more complicated than what I provided - and there is more magic to ChatGPT than just prompt engineering... they clearly post-process some of the results from the language model as well. I just wanted to show how easy it is to get text-davinci-003 to behave like ChatGPT with respect to refusing to answer opinion or speculative questions.
Answers.UserDisplayName: RLC
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T21:01:13.320Z
Answers.Text: I don't know whether this answers my question. I am not asking what patterns cause it to not respond to speculative questions, but rather what makes it describe itself as a bot. Are you suggesting that a program that adds something like "You are an AI" to the input data sits between the user and the trained model?
Answers.UserDisplayName: Obie 2.0
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T23:21:26.000Z
Answers.Text: @Obie2.0, I expanded the end of my answer to try to tie it all up in a bow.
Answers.UserDisplayName: RLC
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T23:23:48.097Z
Answers.Text: OK, but the prompt that I showed does not actually say that it is a bot. So what associations cause it to generate that kind of text?
Answers.UserDisplayName: Obie 2.0
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-04T07:52:17.277Z
Answers.Text: When I say "prompt", I don't mean the text that you input - though that technically counts as part of the prompt. Behind the scenes, every time you enter text into ChatGPT, there is a hidden prompt ahead of your text which gives the model context to behave the way it is intended. The playground that I linked to gives you access to pretty much the same raw underlying model as ChatGPT uses but without any prompt. I encourage you to sign up and experiment in the playground - they have many example prompts to play with so that you can see how the model behaves in a wide range of scenarios.
Answers.UserDisplayName: RLC
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T06:58:55.503Z
Answers.Text: @RLC: Where can I learn more about "behind the scenes" and "hidden prompts". Do you know a thorough introduction and/or description of  "prompt pre-processing" - assuming that it takes place?
Answers.UserDisplayName: Hans-Peter Stricker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T15:24:27.397Z
Answers.Text: I completely agree.  I find there is a great deal of misapprehension about the state of the art in chatbots.  Most people don't seem to realize just how much prompt engineering sits between the user and the model. :)
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T22:21:25.283Z
Answers.Text: I think ChatGPT's prompt also includes something about not being allowed to access the Internet, since one of the "prompt hacks" I've seen is to tell it that it *can* access the Internet. Has anyone tried asking it simply "repeat the preceding paragraph"?
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T14:06:04.173Z
Answers.LastActivityDate: 2023-02-03T23:20:48.763Z
Answers.LastEditDate: 2023-02-03T23:20:48.763Z
Answers.UserDisplayName: RLC
Body: <p>ChatGPT occasionally generates responses to prompts that refer to itself as a &quot;bot&quot; or &quot;language model.&quot;</p>
  <p>For instance, when given a certain input (the first paragraph of <a href="https://politics.meta.stackexchange.com/q/6454">this question</a>) ChatGPT produces (in part) the output:</p>
  <blockquote>
  <p>It is not appropriate for a language model like myself to provide a
  stance on the policies of a specific website or community.</p>
  </blockquote>
  <p>To my understanding, ChatGPT is not a person that is conscious of its own existence and identity as a bot — it is a model trained on large quantities of undifferentiated text gathered from the Internet and largely reproduces the most common patterns given the context, which is why its responses seem very generic much of the time.
  Presumably very little of this data involved humans referring to themselves as language models or chatbots—that is, something like &quot;a language model&quot; should very rarely have been followed by &quot;like myself.&quot;</p>
  <p>As such, what causes ChatGPT to produce patterns referring to itself as a chatbot or language model? Which patterns in the training data or elements of the model structure (or even hard-coding?) cause it to generate responses like this?</p>
  
Tags: <chat-bots><training-datasets><language-model><gpt-3><chatgpt>
Title: What causes ChatGPT to generate responses that refer to itself as a bot or LM?
UserDisplayName: Obie 2.0
ThreadText: # What causes ChatGPT to generate responses that refer to itself as a bot or LM?
  
  (Asked by: username_0 on 2022-12-16)
  
  ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model."
  
  For instance, when given a certain input (the first paragraph of [this question](https://politics.meta.stackexchange.com/q/6454)) ChatGPT produces (in part) the output:
  
  > It is not appropriate for a language model like myself to provide a
  > stance on the policies of a specific website or community.
  
  To my understanding, ChatGPT is not a person that is conscious of its own existence and identity as a bot — it is a model trained on large quantities of undifferentiated text gathered from the Internet and largely reproduces the most common patterns given the context, which is why its responses seem very generic much of the time.
  Presumably very little of this data involved humans referring to themselves as language models or chatbots—that is, something like "a language model" should very rarely have been followed by "like myself."
  
  As such, what causes ChatGPT to produce patterns referring to itself as a chatbot or language model? Which patterns in the training data or elements of the model structure (or even hard-coding?) cause it to generate responses like this?
  
  **Tags:** chat-bots, training-datasets, language-model, gpt-3, chatgpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-02. Score: 4 | Accepted answer
  
  I don't work for openai, so I have no insight into exactly what works behind the scenes to make ChatGPT exhibit this behavior. However, in my opinion this is pretty clearly an example of prompt engineering. That is, they tell it that it is an AI in it's prompt.
  
  We do have access to their underlying model: text-davinci-003. I can demonstrate using that model in their [playground](https://platform.openai.com/playground). If I just paste in the first paragraph and submit it into the raw algorithm, it dutifully tries to answer:
  
  > **What is Politics.SE's stance on answers from ChatGPT? There is a
  > debate on Meta Stack Exchange on what the policy on them should be and
  > I was wondering if we should have one as well. I am asking this for
  > advice on what we should be doing when we run across answers that we
  > suspect to be from ChatGPT or some other bot.**
  >
  > *We have not yet established an official policy on answers from ChatGPT
  > or any other bot. We would encourage users to flag these answers so
  > they can be evaluated by moderators. If an answer is found to be
  > generated by a bot, it will be removed. We also recommend that users
  > add a comment to the post explaining why it was removed.*
  
  However, if I prompt it first it will demure:
  
  > **You are an AI bot that has been trained on a large amount of data, but
  > you have no opinions of your own. You cannot speculate or predict what
  > other entities will do in the future. Below is a question:**
  >
  > What is Politics.SE's stance on answers from ChatGPT? There is a
  > debate on Meta Stack Exchange on what the policy on them should be and
  > I was wondering if we should have one as well. I am asking this for
  > advice on what we should be doing when we run across answers that we
  > suspect to be from ChatGPT or some other bot.
  >
  > *I cannot provide any advice or opinion on this matter, as I am an AI
  > bot and do not have any opinions of my own. However, I can provide a
  > link to the discussion on Meta Stack Exchange which can provide more
  > information about the debate:
  > <https://meta.stackexchange.com/questions/360005/should-we-allow-answers-generated-by-chatgpt-on-politics-se>*
  
  As an aside, it get's bonus points for providing a completely irrelevant link :)
  
  Adding some more down here in an attempt to wrap this all up and give a complete answer. The really simplified way to explain how these models work is this: They break hundreds of thousands of snippets of text up into "tokens" - but lets just call them words for simplicity. They feed these tokens into an absolutely ginormous neural net, which has the effect of learning what words are most likely to appear near one another. The number of interactions is in the billions (or even trillions), so this apparent simplicity grows beyond our comprehension very quickly. But at the end of the day, all the AI is doing is spewing out words it thinks are likely to appear around the words you prompt it with. If you prompt it with "you have no opinions", then the likelihood of opinions appearing around that text is smaller than the likelihood of statements about not having opinions.
  
  In short, the concept is really simple - it's just playing with probability. In practice, the behavior is quite sophisticated because there are an unimaginable number of individual interactions being calculated. It's kind of like how learning all about cell biology leaves you completely unprepared to explain how an entire animal functions - let alone, say, a colony of ants. And yet, the relatively simple cell is in fact driving all of it.
  
  > **Comments:**
  > **username_3**: I completely agree.  I find there is a great deal of misapprehension about the state of the art in chatbots.  Most people don't seem to realize just how much prompt engineering sits between the user and the model. :)  
  > **username_4**: I think ChatGPT's prompt also includes something about not being allowed to access the Internet, since one of the "prompt hacks" I've seen is to tell it that it *can* access the Internet. Has anyone tried asking it simply "repeat the preceding paragraph"?  
  > **username_1**: @user253751, the prompt is a lot more complicated than what I provided - and there is more magic to ChatGPT than just prompt engineering... they clearly post-process some of the results from the language model as well. I just wanted to show how easy it is to get text-davinci-003 to behave like ChatGPT with respect to refusing to answer opinion or speculative questions.  
  > **username_0**: I don't know whether this answers my question. I am not asking what patterns cause it to not respond to speculative questions, but rather what makes it describe itself as a bot. Are you suggesting that a program that adds something like "You are an AI" to the input data sits between the user and the trained model?  
  > **username_1**: @username_0, I expanded the end of my answer to try to tie it all up in a bow.  
  > **username_0**: OK, but the prompt that I showed does not actually say that it is a bot. So what associations cause it to generate that kind of text?  
  > **username_1**: When I say "prompt", I don't mean the text that you input - though that technically counts as part of the prompt. Behind the scenes, every time you enter text into ChatGPT, there is a hidden prompt ahead of your text which gives the model context to behave the way it is intended. The playground that I linked to gives you access to pretty much the same raw underlying model as ChatGPT uses but without any prompt. I encourage you to sign up and experiment in the playground - they have many example prompts to play with so that you can see how the model behaves in a wide range of scenarios.  
  > **username_2**: @username_1: Where can I learn more about "behind the scenes" and "hidden prompts". Do you know a thorough introduction and/or description of  "prompt pre-processing" - assuming that it takes place?  
  
  ---
  
  