Answers.Body: <blockquote>
  <p>Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?</p>
  </blockquote>
  <p>Detecting &quot;truth&quot; in a general sense from natural language is not a solved problem by any means. It is not possible to automate this kind of training, except in very limited domains.</p>
  <p>For example, it would be possible to automate production of billions of maths calculations and their correct answers, but not a great use for an LLM, because any pocket calculator is far more efficient at such tasks. Tuning heavily on such constructed data would very likely cause the model to get better at maths calculations at the expense of most of the other text generations.</p>
  <p>Other than such limited generative/combinatorial true-statement engines, the best models for processing text data and assessing truth more generally are the existing LLMs . . .</p>
  <p><a href="https://huyenchip.com/2023/05/02/rlhf.html" rel="nofollow noreferrer">The reinforcement fine tuning used by e.g. ChatGPT can reward for factually correct output</a>, but requires human trainers to assess responses, so is much harder in terms of resources required. In addition, the core technology of LLMs is not really designed to store/recall such factual and reasoning information efficiently, so although this approach is used and does improve the model significantly, it is not a complete answer to the issue, but instead a useful piece of practical fine tuning.</p>
  <p>Scaling up the reinforcement learning stage might improve the LLM model a little practically over where things stand today, but I would suspect that other architectures and mechanisms for world modelling, memory recall and correct reasoning are where more major improvements lie.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T05:23:48.623Z
Answers.Text: But does not factual training in one sphere (like math) reduce made up answers in other spheres? Another sphere is programming, where a testing software can simply run the generated program and check the result. I think, the problem of inventing fake code that only looks real has been solved already. Most LLMs produce if not etirely correct, but at least, sensible code. Similarly, they rarely invent words for natural language.
Answers.UserDisplayName: Anixx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T07:08:30.827Z
Answers.Text: @Anixx: No training in one factual "sphere" does not really impact results in another one, except stylistically. There is no known inherent signal for "The Truth", that can be learned in a cross-domain way. Definitely code generation has got better, but your assertion is false - I can tell you, as a developer who uses ChatGPT for code tasks regularly, that the language model still frequently makes mistakes that invalidate the functionality of the code. The mistakes are rarely with syntax or structure of well-known functions, but they still occur.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-10T21:37:02.970Z
Answers.LastActivityDate: 2024-03-10T21:48:04.897Z
Answers.LastEditDate: 2024-03-10T21:48:04.897Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>You're asking 2 questions here. You should avoid asking more than one question per post.</p>
  <p>They are a big problem because misinformation/disinformation can be harmful.</p>
  <p>For instance, suppose someone uses these models to get medical, military, political, financial, etc., advice, and they get a factually wrong answer. This can have bad consequences. Of course, no intelligent person should blindly use these models, if you care about facts.</p>
  <p>Moreover, these models are not really trained to give you facts, but to give you the most likely answer given the training data. As stated in the other answer, you can fine-tune them with RL to punish them for factually wrong answers, but, as far as I know, there's nothing that prevents models like <code>gpt-4</code> or <code>gpt-3-turbo</code> from sometimes producing factually wrong statements (if there was, it would not really be working). If they give you facts, it's just a byproduct, it's just chance. The next time they could give you wrong info, and this can depend on the input you pass to them, among other parameters, like the temperature.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-10T22:24:15.237Z
Answers.Text: I think it is just one question. "Why are hallucinations a huge problem?" could be restated "Why can we not simply train away hallucinations?" and then the title would match the body. So "a huge problem" not in the sense of damage caused, but in the sense of not being easy to resolve.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-10T23:41:12.377Z
Answers.Text: Ok, I might have misunderstood it then, if that's the case. I can remove this answer if it doesn't really address the question
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T05:17:18.807Z
Answers.Text: Indeed, I asked why it is a huge problem in the sense of resolving it, not regarding damage caused, as @Neil Slater points out.
Answers.UserDisplayName: Anixx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T14:23:42.197Z
Answers.Text: @Anixx Can you please edit your post to clarify that, i.e. change the title to be more specific?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-10T22:11:32.033Z
Answers.LastActivityDate: 2024-03-10T22:17:37.040Z
Answers.LastEditDate: 2024-03-10T22:17:37.040Z
Answers.UserDisplayName: nbro
Body: <p>Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?</p>
  
Tags: <training>
Title: Why are hallucinations a huge problem?
UserDisplayName: Anixx
ThreadText: # Why are hallucinations a huge problem?
  
  (Asked by: username_0 on 2024-03-10)
  
  Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?
  
  **Tags:** training
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-03-10. Score: 3
  
  > Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?
  
  Detecting "truth" in a general sense from natural language is not a solved problem by any means. It is not possible to automate this kind of training, except in very limited domains.
  
  For example, it would be possible to automate production of billions of maths calculations and their correct answers, but not a great use for an LLM, because any pocket calculator is far more efficient at such tasks. Tuning heavily on such constructed data would very likely cause the model to get better at maths calculations at the expense of most of the other text generations.
  
  Other than such limited generative/combinatorial true-statement engines, the best models for processing text data and assessing truth more generally are the existing LLMs . . .
  
  [The reinforcement fine tuning used by e.g. ChatGPT can reward for factually correct output](https://huyenchip.com/2023/05/02/rlhf.html), but requires human trainers to assess responses, so is much harder in terms of resources required. In addition, the core technology of LLMs is not really designed to store/recall such factual and reasoning information efficiently, so although this approach is used and does improve the model significantly, it is not a complete answer to the issue, but instead a useful piece of practical fine tuning.
  
  Scaling up the reinforcement learning stage might improve the LLM model a little practically over where things stand today, but I would suspect that other architectures and mechanisms for world modelling, memory recall and correct reasoning are where more major improvements lie.
  
  > **Comments:**
  > **username_0**: But does not factual training in one sphere (like math) reduce made up answers in other spheres? Another sphere is programming, where a testing software can simply run the generated program and check the result. I think, the problem of inventing fake code that only looks real has been solved already. Most LLMs produce if not etirely correct, but at least, sensible code. Similarly, they rarely invent words for natural language.  
  > **username_1**: @Anixx: No training in one factual "sphere" does not really impact results in another one, except stylistically. There is no known inherent signal for "The Truth", that can be learned in a cross-domain way. Definitely code generation has got better, but your assertion is false - I can tell you, as a developer who uses ChatGPT for code tasks regularly, that the language model still frequently makes mistakes that invalidate the functionality of the code. The mistakes are rarely with syntax or structure of well-known functions, but they still occur.  
  
  ---
  
  ### Answer by username_2 on 2024-03-10. Score: 1
  
  You're asking 2 questions here. You should avoid asking more than one question per post.
  
  They are a big problem because misinformation/disinformation can be harmful.
  
  For instance, suppose someone uses these models to get medical, military, political, financial, etc., advice, and they get a factually wrong answer. This can have bad consequences. Of course, no intelligent person should blindly use these models, if you care about facts.
  
  Moreover, these models are not really trained to give you facts, but to give you the most likely answer given the training data. As stated in the other answer, you can fine-tune them with RL to punish them for factually wrong answers, but, as far as I know, there's nothing that prevents models like `gpt-4` or `gpt-3-turbo` from sometimes producing factually wrong statements (if there was, it would not really be working). If they give you facts, it's just a byproduct, it's just chance. The next time they could give you wrong info, and this can depend on the input you pass to them, among other parameters, like the temperature.
  
  > **Comments:**
  > **username_1**: I think it is just one question. "Why are hallucinations a huge problem?" could be restated "Why can we not simply train away hallucinations?" and then the title would match the body. So "a huge problem" not in the sense of damage caused, but in the sense of not being easy to resolve.  
  > **username_2**: Ok, I might have misunderstood it then, if that's the case. I can remove this answer if it doesn't really address the question  
  > **username_0**: Indeed, I asked why it is a huge problem in the sense of resolving it, not regarding damage caused, as @Neil Slater points out.  
  > **username_2**: @Anixx Can you please edit your post to clarify that, i.e. change the title to be more specific?  
  
  ---
  
  