Answers.Body: <p>Although &quot;hallucination&quot; is discussed as if it were a failure mode of token-predicting LLMs (which are the core of all the recent large models), this is not really any different to normal behaviour. LLMs are not designed to output correct information, they are not trained specifically to do so, and there is no loss function or reward signal during training that encourages truth (although some fine tuning or human interaction reinforcement learning applied afterwards may adjust this to some degree).</p>
  <p>LLMs do not have inherent filters for detecting true or accurate text output. Instead they output text according to the probability that the model determines that it should appear, based on training material plus the text so far, including any &quot;pre-prompt&quot; setup plus the text supplied by user and it's own output so far.</p>
  <p>The extent that LLMs output true and accurate statements is limited to:</p>
  <ul>
  <li><p>The probability that a true statement might appear in all similar text from the training data.</p>
  </li>
  <li><p>The ability of the model to generalise from the training data and predict accurately.</p>
  </li>
  <li><p>For some AI systems, ability of added monitoring and control code models to detect and correct issues when they occur. These secondary models are typically far more limited than the LLMs, but may help.</p>
  </li>
  <li><p>For some LLMs there may be fine tuning that helps the core model provide truer or more useful output.  This cannot be done at the same scale as the base training however, because it requires far more human effort per training example.</p>
  </li>
  </ul>
  <p>In all cases, there is limited &quot;understanding&quot; of what the model outputs, in terms of real world relevance and accuracy. Mainly this is achieved because the majority of training data makes logically correct and accurate statements that the model will learn to predict approximately. It cannot refer back to other modes of knowledge though, or have any way to experience the world directly. So sometimes the text predictions produce nonsense that is well written otherwise.</p>
  <p>In my experience, this nonsense happens very easily indeed. Just ask a model an expert level question on any subject. The LLM will produce text that on a first look seems reasonable, but an actual expert in the subject will tell you it makes no sense at all in a &quot;not even wrong&quot; kind of way. This is why LLM answers are not accepted on most Stack Exchange sites.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T18:10:58.970Z
Answers.Text: Interesting, I went to try expert level subject question on Google Bard and you are correct, it started to hallucinate things like I work for Google AI which is total nonsense. However it got some things correct also like the work is available GitHub despite my question was about YouTube video I've made. I'd say it was close to 40/60 percent Bard got right. Pretty impressing I'd say since I asked something about my own work no one else knows.
Answers.UserDisplayName: harism
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T22:23:01.973Z
Answers.Text: @harism: Of course it doesn't "know" anything, it outputs text that has a kind of consistency, and when it comes to guessing things you have done, just like someone writing realistic fiction, sometimes the words match reality.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T17:06:22.513Z
Answers.LastActivityDate: 2023-07-22T17:42:15.863Z
Answers.LastEditDate: 2023-07-22T17:42:15.863Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>It depends on the goal of the model.</p>
  <p>For a model that generates a story plot based on an input prompt, hallucination might even be an welcome feature.</p>
  <p>For a model that is meant to stand in for a subject matter expert and answer topical questions, yes, hallucination is a terrible thing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T11:39:11.507Z
Answers.LastActivityDate: 2023-07-25T11:39:11.507Z
Answers.UserDisplayName: ahron
Body: <p>I've been pondering over this for a while now: is the so-called hallucinating necessarily a problem in LLM or in AI in general? What it stands for anyway, maybe the model is trying to crunch so much information into the answer we just cannot comprehend. How could it be the model's fault since it knows only what we teach them after all?</p>
  <p>I am trying to cover LLM hallucination here as a whole considering it as a general term many of us have read about as a known problem with LLM. For further reading and understanding different kinds of LLM hallucination I recommend to check <a href="https://g.co/bard/share/6f8d1031d317" rel="nofollow noreferrer">this link</a> and <a href="https://chat.openai.com/share/9d1b597e-57f7-4ce2-802e-6d3cf9b6999a" rel="nofollow noreferrer">this link</a>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T22:40:26.483Z
Comments.Text: I'll promote this question with slightly provocative thought; should we call hallucinating concluding instead, if the model is stable and does not add randomness to the data it has been taught with?
Comments.UserDisplayName: harism
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-21T23:27:47.667Z
Comments.Text: For reference was it Facebook who set two LLM chatbots to chat with each other, it didn't take long they created their own language no one understood anymore. Maybe we should.
Comments.UserDisplayName: harism
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-24T08:39:01.257Z
Comments.Text: Hi Harism! Thanks for the question. It would be helpful if you provide a definition for 'hallucination'. This is helpful for (1) clarity and unambiguity of your question and (2) for people who are not very experienced on the topic.
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-24T18:19:48.390Z
Comments.Text: @RobinvanHoorn I will edit my question now but your request for clarity and unambiguity is not necessarily fulfilled but I do my best.
Comments.UserDisplayName: harism
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T11:33:52.530Z
Comments.Text: I wonder if its a bit about intuition for humans. One doesn't have a clear basis for it, but it somehow might make some sense.
Comments.UserDisplayName: NoDataDumpNoContribution
Tags: <chatgpt>
Title: Are LLM hallucination necessarily a bad thing?
UserDisplayName: harism
ThreadText: # Are LLM hallucination necessarily a bad thing?
  
  (Asked by: username_0 on 2023-07-21)
  
  I've been pondering over this for a while now: is the so-called hallucinating necessarily a problem in LLM or in AI in general? What it stands for anyway, maybe the model is trying to crunch so much information into the answer we just cannot comprehend. How could it be the model's fault since it knows only what we teach them after all?
  
  I am trying to cover LLM hallucination here as a whole considering it as a general term many of us have read about as a known problem with LLM. For further reading and understanding different kinds of LLM hallucination I recommend to check [this link](https://g.co/bard/share/6f8d1031d317) and [this link](https://chat.openai.com/share/9d1b597e-57f7-4ce2-802e-6d3cf9b6999a).
  
  **Tags:** chatgpt
  > **Comments:**
  > **username_0**: I'll promote this question with slightly provocative thought; should we call hallucinating concluding instead, if the model is stable and does not add randomness to the data it has been taught with?  
  > **username_0**: For reference was it Facebook who set two LLM chatbots to chat with each other, it didn't take long they created their own language no one understood anymore. Maybe we should.  
  > **username_1**: Hi Harism! Thanks for the question. It would be helpful if you provide a definition for 'hallucination'. This is helpful for (1) clarity and unambiguity of your question and (2) for people who are not very experienced on the topic.  
  > **username_0**: @RobinvanHoorn I will edit my question now but your request for clarity and unambiguity is not necessarily fulfilled but I do my best.  
  > **username_2**: I wonder if its a bit about intuition for humans. One doesn't have a clear basis for it, but it somehow might make some sense.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2023-07-22. Score: 2 | Accepted answer
  
  Although "hallucination" is discussed as if it were a failure mode of token-predicting LLMs (which are the core of all the recent large models), this is not really any different to normal behaviour. LLMs are not designed to output correct information, they are not trained specifically to do so, and there is no loss function or reward signal during training that encourages truth (although some fine tuning or human interaction reinforcement learning applied afterwards may adjust this to some degree).
  
  LLMs do not have inherent filters for detecting true or accurate text output. Instead they output text according to the probability that the model determines that it should appear, based on training material plus the text so far, including any "pre-prompt" setup plus the text supplied by user and it's own output so far.
  
  The extent that LLMs output true and accurate statements is limited to:
  
  * The probability that a true statement might appear in all similar text from the training data.
  * The ability of the model to generalise from the training data and predict accurately.
  * For some AI systems, ability of added monitoring and control code models to detect and correct issues when they occur. These secondary models are typically far more limited than the LLMs, but may help.
  * For some LLMs there may be fine tuning that helps the core model provide truer or more useful output. This cannot be done at the same scale as the base training however, because it requires far more human effort per training example.
  
  In all cases, there is limited "understanding" of what the model outputs, in terms of real world relevance and accuracy. Mainly this is achieved because the majority of training data makes logically correct and accurate statements that the model will learn to predict approximately. It cannot refer back to other modes of knowledge though, or have any way to experience the world directly. So sometimes the text predictions produce nonsense that is well written otherwise.
  
  In my experience, this nonsense happens very easily indeed. Just ask a model an expert level question on any subject. The LLM will produce text that on a first look seems reasonable, but an actual expert in the subject will tell you it makes no sense at all in a "not even wrong" kind of way. This is why LLM answers are not accepted on most Stack Exchange sites.
  
  > **Comments:**
  > **username_0**: Interesting, I went to try expert level subject question on Google Bard and you are correct, it started to hallucinate things like I work for Google AI which is total nonsense. However it got some things correct also like the work is available GitHub despite my question was about YouTube video I've made. I'd say it was close to 40/60 percent Bard got right. Pretty impressing I'd say since I asked something about my own work no one else knows.  
  > **username_3**: @harism: Of course it doesn't "know" anything, it outputs text that has a kind of consistency, and when it comes to guessing things you have done, just like someone writing realistic fiction, sometimes the words match reality.  
  
  ---
  
  ### Answer by username_4 on 2023-07-25. Score: 2
  
  It depends on the goal of the model.
  
  For a model that generates a story plot based on an input prompt, hallucination might even be an welcome feature.
  
  For a model that is meant to stand in for a subject matter expert and answer topical questions, yes, hallucination is a terrible thing.
  
  
  ---
  
  