Answers.Body: <blockquote>
  <p>How sure can one be about the correctness of the models outputs on these kinds of prompts? I do not want it to hallucinate here whatsoever; that would undermine the whole approach of using LLMs in this context?</p>
  </blockquote>
  <p>No way to be 100% sure, e.g. see <a href="https://arxiv.org/abs/2310.01798" rel="nofollow noreferrer">Large Language Models Cannot Self-Correct Reasoning Yet</a>. There can be some tricks to reduce or detect some hallucinations though.</p>
  <blockquote>
  <p>Are there benchmarks specifically made for this task (i.e., providing lots of information in context and then asking for things appearing in this context)?</p>
  </blockquote>
  <p>Yes, e.g. <a href="https://arxiv.org/pdf/2309.08872.pdf" rel="nofollow noreferrer">PDFTriage: Question Answering over Long, Structured Documents</a> but we are waiting for the permission to release data+code.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-31T22:24:55.470Z
Answers.LastActivityDate: 2023-10-31T22:24:55.470Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I know of some benchmarks that LLMs do undergo, but I am no expert whatsoever. I think what I am wondering about is closest to <a href="https://paperswithcode.com/dataset/truthfulqa" rel="nofollow noreferrer">TruthfulQA</a>.
  The question came up when I heard of combining company data with LLMs by providing the internal, sensitive data in prompts à la <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG" rel="nofollow noreferrer">RAG</a> (of course you do not want to fine-tune the model every time, because the data may change daily or even hourly, but instead supply it with the data in prompts).</p>
  <p>So the question is:
  How sure can one be about the correctness of the models outputs on these kinds of prompts? I do not want it to hallucinate here whatsoever; that would undermine the whole approach of using LLMs in this context. Are there benchmarks <em>specifically</em> made for this task (i.e., providing lots of information in context and then asking for things appearing in this context)?</p>
  <p>Thanks a lot!</p>
  
Tags: <large-language-models><prompt><benchmarks>
Title: LLM Hallucinations on In-Context Data
UserDisplayName: Mathy
ThreadText: # LLM Hallucinations on In-Context Data
  
  (Asked by: username_0 on 2023-10-31)
  
  I know of some benchmarks that LLMs do undergo, but I am no expert whatsoever. I think what I am wondering about is closest to [TruthfulQA](https://paperswithcode.com/dataset/truthfulqa).
  The question came up when I heard of combining company data with LLMs by providing the internal, sensitive data in prompts à la [RAG](https://research.ibm.com/blog/retrieval-augmented-generation-RAG) (of course you do not want to fine-tune the model every time, because the data may change daily or even hourly, but instead supply it with the data in prompts).
  
  So the question is:
  How sure can one be about the correctness of the models outputs on these kinds of prompts? I do not want it to hallucinate here whatsoever; that would undermine the whole approach of using LLMs in this context. Are there benchmarks *specifically* made for this task (i.e., providing lots of information in context and then asking for things appearing in this context)?
  
  Thanks a lot!
  
  **Tags:** large-language-models, prompt, benchmarks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-31. Score: 3 | Accepted answer
  
  > How sure can one be about the correctness of the models outputs on these kinds of prompts? I do not want it to hallucinate here whatsoever; that would undermine the whole approach of using LLMs in this context?
  
  No way to be 100% sure, e.g. see [Large Language Models Cannot Self-Correct Reasoning Yet](https://arxiv.org/abs/2310.01798). There can be some tricks to reduce or detect some hallucinations though.
  
  > Are there benchmarks specifically made for this task (i.e., providing lots of information in context and then asking for things appearing in this context)?
  
  Yes, e.g. [PDFTriage: Question Answering over Long, Structured Documents](https://arxiv.org/pdf/2309.08872.pdf) but we are waiting for the permission to release data+code.
  
  
  ---
  
  