Answers.Body: <p>From <a href="https://arxiv.org/abs/2305.11206" rel="nofollow noreferrer">LIMA: Less Is More for Alignment</a> (2023) by
  Chunting Zhou et al.:</p>
  <blockquote>
  <p>These results strongly suggest that <strong>almost all knowledge in large language models is learned during pretraining</strong>, and only limited instruction tuning data is necessary to teach models to produce high quality output.</p>
  </blockquote>
  <p>From <a href="https://arxiv.org/abs/2405.05904" rel="nofollow noreferrer">Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?</a> (2024) by
  <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gekhman,+Z" rel="nofollow noreferrer">Zorik Gekhman</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yona,+G" rel="nofollow noreferrer">Gal Yona</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Aharoni,+R" rel="nofollow noreferrer">Roee Aharoni</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Eyal,+M" rel="nofollow noreferrer">Matan Eyal</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Feder,+A" rel="nofollow noreferrer">Amir Feder</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Reichart,+R" rel="nofollow noreferrer">Roi Reichart</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Herzig,+J" rel="nofollow noreferrer">Jonathan Herzig</a>:</p>
  <blockquote>
  <p>When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. Our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T21:07:52.840Z
Answers.LastActivityDate: 2024-06-06T18:30:31.110Z
Answers.LastEditDate: 2024-06-06T18:30:31.110Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I cannot find a definite answer to this question. Suppose I want to build a QA (question answering) system on a set of personal documents. It looks that RAG (retrieval augmented generation) is the way to go for this task, but I do not understand why some flavour of fine tuning would not work. The LLMs are able to answer factual questions about the models they are trained on (as we see each time we make a query to a LLM), so why should not they be able to answer questions about data they are fine tuned on as well? Is there a good reference or discussion of this issue and some code showcasing this behavior ?</p>
  
Tags: <large-language-models><fine-tuning><question-answering><information-retrieval>
Title: Why fine tuning does not work as well as RAG?
UserDisplayName: Thomas
ThreadText: # Why fine tuning does not work as well as RAG?
  
  (Asked by: username_0 on 2024-05-14)
  
  I cannot find a definite answer to this question. Suppose I want to build a QA (question answering) system on a set of personal documents. It looks that RAG (retrieval augmented generation) is the way to go for this task, but I do not understand why some flavour of fine tuning would not work. The LLMs are able to answer factual questions about the models they are trained on (as we see each time we make a query to a LLM), so why should not they be able to answer questions about data they are fine tuned on as well? Is there a good reference or discussion of this issue and some code showcasing this behavior ?
  
  **Tags:** large-language-models, fine-tuning, question-answering, information-retrieval
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-24. Score: 1
  
  From [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206) (2023) by
  Chunting Zhou et al.:
  
  > These results strongly suggest that **almost all knowledge in large language models is learned during pretraining**, and only limited instruction tuning data is necessary to teach models to produce high quality output.
  
  From [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://arxiv.org/abs/2405.05904) (2024) by
  [Zorik Gekhman](https://arxiv.org/search/cs?searchtype=author&query=Gekhman,+Z), [Gal Yona](https://arxiv.org/search/cs?searchtype=author&query=Yona,+G), [Roee Aharoni](https://arxiv.org/search/cs?searchtype=author&query=Aharoni,+R), [Matan Eyal](https://arxiv.org/search/cs?searchtype=author&query=Eyal,+M), [Amir Feder](https://arxiv.org/search/cs?searchtype=author&query=Feder,+A), [Roi Reichart](https://arxiv.org/search/cs?searchtype=author&query=Reichart,+R), [Jonathan Herzig](https://arxiv.org/search/cs?searchtype=author&query=Herzig,+J):
  
  > When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. Our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.
  
  
  ---
  
  