Answers.Body: <p>Humans, who know things, tend to limit themselves to speaking of things they know, especially in print. Thus any text corpus you may find will have an overpopulation of examples of confidently presented answers.</p>
  <p>Chatbots, which do not know things, cannot impose a similar limit and thus will babble at length on topics about which they know nothing. But because they are trained on those data of people writing about things they know, they will tend to mimic the same confident tone and style of those people.</p>
  <p>It is very important to reiterate here again that chatbots <em>do not know things</em>. By good fortune, they may produce a text which happens to convey factually accurate information. But they may not, and because they do not know things, they do not in advance (or even in retrospect) if they can (or did) produce a factually accurate text or meaningless drivel.</p>
  <p>Thus a chatbot (or its developer by proxy) has only two options available when asked a question. Option 1 is to attempt to generate an answer to every question. Option 2 is to answer every question with &quot;I don't know.&quot; The latter is technically uninteresting and practically useless, so everybody chooses Option 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T01:06:26.327Z
Answers.Text: This is mostly incorrect. Mechanistic interpretability research has proven that transformers store factual information (which can be edited) in their MLP blocks and hence, GPT-based chatbots do actually know things (for references, google "Knowledge Neurons in Pretrained Transformers" and "Moving the Eiffel Tower to ROME"). They also "know" when they generate falsehoods and can be steered to the truth by activation engineering (google "Discovering Latent Knowledge Without Supervision", "The Internal State of an LLM Knows When its Lying" and "Eliciting Truthful Answers from a Language Model").
Answers.UserDisplayName: ain92
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T22:05:20.343Z
Answers.Text: Also, Option 1 vs Option 2 is a false dichotomy. The chatbot could say, "I am not sure but..." and proceed.
Answers.UserDisplayName: Mooing Duck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T18:01:16.143Z
Answers.LastActivityDate: 2023-08-30T18:01:16.143Z
Answers.UserDisplayName: OpenAI was the last straw
Answers.Body: <p>It's quite simple really. LLM based generative AIs don't &quot;know&quot; anything. They're glorified next-word predictors. The only way that they're going to produce an answer to a prompt along the lines of &quot;I don't know&quot; is if their training corpus would indicate that that was the likely response a human would give to the same prompt. Admitting that we don't know something isn't a typically common human trait in the first place, and publishing that online is even less so.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T18:46:17.700Z
Answers.Text: And they don't "feel the need" to act in any way either. They're just optionally (cons)trained in a particular direction.
Answers.UserDisplayName: sehe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T09:29:24.933Z
Answers.LastActivityDate: 2023-09-03T17:54:30.367Z
Answers.LastEditDate: 2023-09-03T17:54:30.367Z
Answers.UserDisplayName: ScottishTapWater
Answers.Body: <p>The only thing the bot &quot;knows&quot; is how to generate sequences of lexical symbols (ie. texts) that mimic what people have written in the corpus it has been trained on.</p>
  <p>The output is based on the prompt using complex rules + some internal state (so that it seems to &quot;remember&quot; the past discussion). It is all just mathematics that is used to choose which symbols (ie. letters and punctuation) to output. It could be implemented just as well with pocket calculators, or even doing the math manually, albeit very slowly and very arduously.</p>
  <p>There is nothing else to the bot. No knowledge, no reasoning, no needs, no goals, no personality. Basically just numbers manipulated in a complex way.</p>
  <p>Whatever else you happen see in the output is always your interpretation. If you spot a need in the text, it is most likely there only because some actual person did write something that reflected their need, and from it, the bot learned such rules that the output mimics something that was written with emotion.</p>
  <p>It is as sentient as a cartoon character in an animated movie. All emotions that you assume to be in the character/text originates from you.</p>
  <p>So, why it looks like they feel the need to be know-it-alls? Because they mimic what people write, and people are like that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T06:07:43.817Z
Answers.Text: Just because the model outputs know-it-all text for a set of queries does not *necessarily* imply that the training dataset is equally know-it-all-esque. (c.f. Most famously, see [the "debate"](https://analyticsindiamag.com/yann-lecun-machine-learning-bias-debate/) on Yann LeCun's [tweet](https://twitter.com/ylecun/status/1274782757907030016?lang=en).)
Answers.UserDisplayName: Mateen Ulhaq
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T06:17:18.347Z
Answers.Text: If we think of humans as very complex models/functions, we may observe that trained humans are prone to certain behaviors (e.g. seeking food/water/shelter, tribalism, groupthink, ...) by nature (i.e. model DNA) in addition to nurture (i.e. training data).
Answers.UserDisplayName: Mateen Ulhaq
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T23:36:00.070Z
Answers.LastActivityDate: 2023-08-31T23:41:56.880Z
Answers.LastEditDate: 2023-08-31T23:41:56.880Z
Answers.UserDisplayName: Jani Miettinen
Answers.Body: <p>The main reason an AI model won't tell you &quot;<em>I don't know</em>&quot; is that the developer doesn't want you to get the impression that the model is incapable of answering your questions.</p>
  <p>Imagine a scenario where you had several difficult or non-trivial questions that the model doesn't have the right answers for. If answers in the form of &quot;<em>I don't know</em>&quot; kept piling up, it will drive you away and make you less confident about getting an answer the next time you have a question.</p>
  <p>So even if it gives an incorrect or inaccurate answer, by just giving an answer, it leaves you under the impression that sometimes it gives correct answers - sometimes it doesn't, and the next time you have a question you might return because you'd think it might give you a correct answer this time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T19:37:38.417Z
Answers.LastActivityDate: 2023-09-01T19:37:38.417Z
Answers.UserDisplayName: machine_1
Answers.Body: <p>The large language models underpinning these &quot;AI&quot; bots have indeed been tweaked to be biased against providing a negative response.</p>
  <p>The simple reason being that the executives of the companies trying to sell &quot;AI&quot; as the Next Big Thing, believe they're less likely to make sales if their product appears to be unable to answer questions. Thus they instruct their engineers to train these models to divert, dodge and dissemble - even to plainly ridiculous lengths - when the model is incapable of coming up with an affirmative answer to the question asked.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-02T18:46:08.687Z
Answers.Text: I f you can provide  a source for that, I'm sure you'll get upvoted. Seems it sounds a bit too conspiracy like for the community...
Answers.UserDisplayName: ben svenssohn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T20:40:29.550Z
Answers.LastActivityDate: 2023-09-01T20:40:29.550Z
Answers.UserDisplayName: Ian Kemp
Answers.Body: <blockquote>
  <p>Have they been trained to always respond as though it were a know-it-all?</p>
  </blockquote>
  <p>Yes, sort of, although it is not in some attempt to upset you or other users.</p>
  <p>The language models used in the chat bots <a href="https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html" rel="noreferrer">have been fine-tuned in conversations with users</a>, and also have been <a href="https://www.reddit.com/r/ChatGPT/comments/10ehfjj/what_is_the_preprompt/" rel="noreferrer">&quot;pre-prompted&quot; to be <em>helpful assistants</em></a>. This impacts the kind of text that they generate.</p>
  <p>Separately to this, the fact that they are often incorrect or generate nonsense is a limitation of the core technology. There is no underlying world model, the model cannot self-assess whether anything it generates is correct. Although the large language models can often surprise users (including the original developers) in the breadth of what they can respond to correctly and give the appearance of understanding, the same models can also fail easily at apparently simple tasks. This can cause some cognitive dissonance for users who expect an AI that appears to communicate like an intelligent human to be able to perform simple maths for example.</p>
  <p>There is probably no global default chat <em>tone</em> that would suit all users. You could however use the raw model more directly and create a chatbot with a different tone to suit you. This would not impact accuracy, and would not add ability to self-assess correctness, but it may address feeling of being talked down to for example. Some services, like Character AI, attempt to give you tools to do just that, although the level of usefulness you get from them will depend on what they focus on (Character AI is more for creative fun, than for functional assistants).</p>
  <p>In limited cases you can also address accuracy with pre-prompts or fine tuning that put a few facts into working memory. This is limited, and cannot address lack of general ability in logic or maths though. Corporate users can do this to give a chat bot some basic product knowledge or correct address for their help desk etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T06:28:17.130Z
Answers.Text: I think adjusted the tone of an ai chatbot won't really change anything because the bot simply doesn't know whether it is knowledgable on a topic or not. OP wants a promp something like 'explain this if you are able to do it but tell me if you are not'. The bot simply can't make this distinction.
Answers.UserDisplayName: quarague
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T06:48:01.853Z
Answers.Text: @quarague It won't change factual accuracy, because the bot cannot realistically introspect about what it is outputting. However, it could remove the "know it all" vibe that they are complaining about.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T06:50:21.907Z
Answers.Text: If it were to start everything with 'I'm not really sure but I believe ..' this doesn't really help OP because it would do that everywhere including on topics where it is 100% perfect spot on.
Answers.UserDisplayName: quarague
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T06:58:03.673Z
Answers.Text: @quarague I am not seeing that the OP has asked for that. I am reading that they are literally complaining about tone not matching the capability. I also cover this difference in the third paragraph, starding with "separately to this", and I make it clear that the model cannot self assess. I am not sure what edit/change to this answer you are suggesting therefore?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-31T14:50:06.710Z
Answers.Text: **Comments have been [moved to chat](https://chat.stackexchange.com/rooms/148227/discussion-on-answer-by-neil-slater-why-do-many-ai-bots-feel-the-need-to-be-know); please do not continue the discussion here.** Before posting a comment below this one, please review the [purposes of comments](/help/privileges/comment). Comments that do not request clarification or suggest improvements usually belong as an [answer](/help/how-to-answer), on [meta], or in [chat]. Comments continuing discussion may be removed.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-29T16:04:20.777Z
Answers.LastActivityDate: 2023-08-30T07:19:21.800Z
Answers.LastEditDate: 2023-08-30T07:19:21.800Z
Answers.UserDisplayName: Neil Slater
Body: <p>Having used various AI bots often over recent months, I noticed that often it will claim to know something, even if it doesn't.
  It would then either explain something which is clearly nonsense, or by rambling on about how the answer isn't known in general.
  Or how, if asked for example- &quot;would you be able to explain X&quot; it wouldn't respond &quot;yes, I could&quot; but rather would elucidate X.
  Have they been trained to always respond as though it were a know-it-all?
  (Google's Bard and ChatGPT specifically, although I'm assuming only open-source AI will be answerable)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-30T14:35:17.750Z
Comments.Text: What nobody wants to admit is that the language used by a large language model naturally mimics the language style and tone of the humans whose utterances were used to build the model. I personally find that ChatGPT (for example) behaves exactly the same as I have observed some humans behave.
Comments.UserDisplayName: user21820
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T06:25:33.833Z
Comments.Text: Humans work the same way. Some humans raise above the mould and can do more self-reflection and cross-analysis of the data they're fed, but that's still a bit out of reach of contemporary AI. Inattentive humans absorb all the data they're fed uncritically and only rarely realize and correct after the fact. Many unscrupulous people exploit that "feature" of human brains - advertising, misinformation and all sorts of other manipulation. Thinking you understand everything (especially something you know little about) is unfortunately very natural to our brains :)
Comments.UserDisplayName: Luaan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T17:03:54.513Z
Comments.Text: These nonsense responses are called [hallucinations](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence))
Comments.UserDisplayName: John Gordon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T17:51:26.440Z
Comments.Text: “often it will claim to know something, even if it doesn't” — are you implying that there are other things an AI _does_ ‘know’?
Comments.UserDisplayName: gidds
Tags: <machine-learning><chatgpt>
Title: Why do many AI bots feel the need to be know-it-alls?
UserDisplayName: ben svenssohn
ThreadText: # Why do many AI bots feel the need to be know-it-alls?
  
  (Asked by: username_0 on 2023-08-29)
  
  Having used various AI bots often over recent months, I noticed that often it will claim to know something, even if it doesn't.
  It would then either explain something which is clearly nonsense, or by rambling on about how the answer isn't known in general.
  Or how, if asked for example- "would you be able to explain X" it wouldn't respond "yes, I could" but rather would elucidate X.
  Have they been trained to always respond as though it were a know-it-all?
  (Google's Bard and ChatGPT specifically, although I'm assuming only open-source AI will be answerable)
  
  **Tags:** machine-learning, chatgpt
  > **Comments:**
  > **username_1**: What nobody wants to admit is that the language used by a large language model naturally mimics the language style and tone of the humans whose utterances were used to build the model. I personally find that ChatGPT (for example) behaves exactly the same as I have observed some humans behave.  
  > **username_2**: Humans work the same way. Some humans raise above the mould and can do more self-reflection and cross-analysis of the data they're fed, but that's still a bit out of reach of contemporary AI. Inattentive humans absorb all the data they're fed uncritically and only rarely realize and correct after the fact. Many unscrupulous people exploit that "feature" of human brains - advertising, misinformation and all sorts of other manipulation. Thinking you understand everything (especially something you know little about) is unfortunately very natural to our brains :)  
  > **username_3**: These nonsense responses are called [hallucinations](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence))  
  > **username_4**: “often it will claim to know something, even if it doesn't” — are you implying that there are other things an AI _does_ ‘know’?  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_14 on 2023-08-29. Score: 25 | Accepted answer
  
  > Have they been trained to always respond as though it were a know-it-all?
  
  Yes, sort of, although it is not in some attempt to upset you or other users.
  
  The language models used in the chat bots [have been fine-tuned in conversations with users](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html), and also have been ["pre-prompted" to be *helpful assistants*](https://www.reddit.com/r/ChatGPT/comments/10ehfjj/what_is_the_preprompt/). This impacts the kind of text that they generate.
  
  Separately to this, the fact that they are often incorrect or generate nonsense is a limitation of the core technology. There is no underlying world model, the model cannot self-assess whether anything it generates is correct. Although the large language models can often surprise users (including the original developers) in the breadth of what they can respond to correctly and give the appearance of understanding, the same models can also fail easily at apparently simple tasks. This can cause some cognitive dissonance for users who expect an AI that appears to communicate like an intelligent human to be able to perform simple maths for example.
  
  There is probably no global default chat *tone* that would suit all users. You could however use the raw model more directly and create a chatbot with a different tone to suit you. This would not impact accuracy, and would not add ability to self-assess correctness, but it may address feeling of being talked down to for example. Some services, like Character AI, attempt to give you tools to do just that, although the level of usefulness you get from them will depend on what they focus on (Character AI is more for creative fun, than for functional assistants).
  
  In limited cases you can also address accuracy with pre-prompts or fine tuning that put a few facts into working memory. This is limited, and cannot address lack of general ability in logic or maths though. Corporate users can do this to give a chat bot some basic product knowledge or correct address for their help desk etc.
  
  > **Comments:**
  > **username_15**: I think adjusted the tone of an ai chatbot won't really change anything because the bot simply doesn't know whether it is knowledgable on a topic or not. OP wants a promp something like 'explain this if you are able to do it but tell me if you are not'. The bot simply can't make this distinction.  
  > **username_14**: @quarague It won't change factual accuracy, because the bot cannot realistically introspect about what it is outputting. However, it could remove the "know it all" vibe that they are complaining about.  
  > **username_15**: If it were to start everything with 'I'm not really sure but I believe ..' this doesn't really help OP because it would do that everywhere including on topics where it is 100% perfect spot on.  
  > **username_14**: @quarague I am not seeing that the OP has asked for that. I am reading that they are literally complaining about tone not matching the capability. I also cover this difference in the third paragraph, starding with "separately to this", and I make it clear that the model cannot self assess. I am not sure what edit/change to this answer you are suggesting therefore?  
  > **username_16**: **Comments have been [moved to chat](https://chat.stackexchange.com/rooms/148227/discussion-on-answer-by-neil-slater-why-do-many-ai-bots-feel-the-need-to-be-know); please do not continue the discussion here.** Before posting a comment below this one, please review the [purposes of comments](/help/privileges/comment). Comments that do not request clarification or suggest improvements usually belong as an [answer](/help/how-to-answer), on [meta], or in [chat]. Comments continuing discussion may be removed.  
  
  ---
  
  ### Answer by username_5 on 2023-08-30. Score: 19
  
  Humans, who know things, tend to limit themselves to speaking of things they know, especially in print. Thus any text corpus you may find will have an overpopulation of examples of confidently presented answers.
  
  Chatbots, which do not know things, cannot impose a similar limit and thus will babble at length on topics about which they know nothing. But because they are trained on those data of people writing about things they know, they will tend to mimic the same confident tone and style of those people.
  
  It is very important to reiterate here again that chatbots *do not know things*. By good fortune, they may produce a text which happens to convey factually accurate information. But they may not, and because they do not know things, they do not in advance (or even in retrospect) if they can (or did) produce a factually accurate text or meaningless drivel.
  
  Thus a chatbot (or its developer by proxy) has only two options available when asked a question. Option 1 is to attempt to generate an answer to every question. Option 2 is to answer every question with "I don't know." The latter is technically uninteresting and practically useless, so everybody chooses Option 1.
  
  > **Comments:**
  > **username_6**: This is mostly incorrect. Mechanistic interpretability research has proven that transformers store factual information (which can be edited) in their MLP blocks and hence, GPT-based chatbots do actually know things (for references, google "Knowledge Neurons in Pretrained Transformers" and "Moving the Eiffel Tower to ROME"). They also "know" when they generate falsehoods and can be steered to the truth by activation engineering (google "Discovering Latent Knowledge Without Supervision", "The Internal State of an LLM Knows When its Lying" and "Eliciting Truthful Answers from a Language Model").  
  > **username_7**: Also, Option 1 vs Option 2 is a false dichotomy. The chatbot could say, "I am not sure but..." and proceed.  
  
  ---
  
  ### Answer by username_8 on 2023-08-31. Score: 14
  
  It's quite simple really. LLM based generative AIs don't "know" anything. They're glorified next-word predictors. The only way that they're going to produce an answer to a prompt along the lines of "I don't know" is if their training corpus would indicate that that was the likely response a human would give to the same prompt. Admitting that we don't know something isn't a typically common human trait in the first place, and publishing that online is even less so.
  
  > **Comments:**
  > **username_9**: And they don't "feel the need" to act in any way either. They're just optionally (cons)trained in a particular direction.  
  
  ---
  
  ### Answer by username_10 on 2023-08-31. Score: 7
  
  The only thing the bot "knows" is how to generate sequences of lexical symbols (ie. texts) that mimic what people have written in the corpus it has been trained on.
  
  The output is based on the prompt using complex rules + some internal state (so that it seems to "remember" the past discussion). It is all just mathematics that is used to choose which symbols (ie. letters and punctuation) to output. It could be implemented just as well with pocket calculators, or even doing the math manually, albeit very slowly and very arduously.
  
  There is nothing else to the bot. No knowledge, no reasoning, no needs, no goals, no personality. Basically just numbers manipulated in a complex way.
  
  Whatever else you happen see in the output is always your interpretation. If you spot a need in the text, it is most likely there only because some actual person did write something that reflected their need, and from it, the bot learned such rules that the output mimics something that was written with emotion.
  
  It is as sentient as a cartoon character in an animated movie. All emotions that you assume to be in the character/text originates from you.
  
  So, why it looks like they feel the need to be know-it-alls? Because they mimic what people write, and people are like that.
  
  > **Comments:**
  > **username_11**: Just because the model outputs know-it-all text for a set of queries does not *necessarily* imply that the training dataset is equally know-it-all-esque. (c.f. Most famously, see [the "debate"](https://analyticsindiamag.com/yann-lecun-machine-learning-bias-debate/) on Yann LeCun's [tweet](https://twitter.com/ylecun/status/1274782757907030016?lang=en).)  
  > **username_11**: If we think of humans as very complex models/functions, we may observe that trained humans are prone to certain behaviors (e.g. seeking food/water/shelter, tribalism, groupthink, ...) by nature (i.e. model DNA) in addition to nurture (i.e. training data).  
  
  ---
  
  ### Answer by username_12 on 2023-09-01. Score: 1
  
  The main reason an AI model won't tell you "*I don't know*" is that the developer doesn't want you to get the impression that the model is incapable of answering your questions.
  
  Imagine a scenario where you had several difficult or non-trivial questions that the model doesn't have the right answers for. If answers in the form of "*I don't know*" kept piling up, it will drive you away and make you less confident about getting an answer the next time you have a question.
  
  So even if it gives an incorrect or inaccurate answer, by just giving an answer, it leaves you under the impression that sometimes it gives correct answers - sometimes it doesn't, and the next time you have a question you might return because you'd think it might give you a correct answer this time.
  
  
  ---
  
  ### Answer by username_13 on 2023-09-01. Score: -2
  
  The large language models underpinning these "AI" bots have indeed been tweaked to be biased against providing a negative response.
  
  The simple reason being that the executives of the companies trying to sell "AI" as the Next Big Thing, believe they're less likely to make sales if their product appears to be unable to answer questions. Thus they instruct their engineers to train these models to divert, dodge and dissemble - even to plainly ridiculous lengths - when the model is incapable of coming up with an affirmative answer to the question asked.
  
  > **Comments:**
  > **username_0**: I f you can provide  a source for that, I'm sure you'll get upvoted. Seems it sounds a bit too conspiracy like for the community...  
  
  ---
  
  