Answers.Body: <p>Indeed the outputs of instruction-tuned models might be influenced by the specific data used during their fine-tuning process. If the fine-tuning data for a language model includes source code examples where variable or function names are consistently derived from certain patterns like names of mountains, cities, or mythological figures, this could bias the model to generate outputs that reflect these naming conventions because, for example, in RLHF if the <em>reward model</em> or its human annotated <em>ranking process</em> prioritizes patterns from specific fine-tuning data or user feedback over general knowledge, the fine-tuning effect will be stronger. By analyzing the difference in outputs between the pre-trained base model and the fine-tuned model with identical prompts, you can assess how much influence the fine-tuning including RLHF exerts.</p>
  <p>To mitigate such effect, increase diversity in fine-tuning data, use data augmentation techniques such as randomizing variable and function names during pre-processing to ensure the model doesn’t overfit to specific patterns, and apply prompt engineering by specifying naming conventions in the prompt, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-01T03:17:43.300Z
Answers.LastActivityDate: 2024-12-01T03:17:43.300Z
Answers.UserDisplayName: cinch
Body: <p><em>Note: This question was originally posted on GenAI site, but I'm reposting here because I think that's more appropriate here.</em></p>
  <p>Are instruct models' outputs affected by finetuning data? E.g. If the finetuning data of a LLM for instruction following includes names of mountains, cities, mythological figures, etc. as function or variable names in source code examples provided in the finetuning input data, will the model mostly provide these name schemes in the output when asked to write source code?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-02T10:39:28.580Z
Comments.Text: What do you think is the purpose of fine-tuning then? Are you asking what it means to fine-tune?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-02T11:29:05.827Z
Comments.Text: @nbro Just whether the output is affected
Comments.UserDisplayName: Hephaestus
Tags: <fine-tuning>
Title: Is the output of finetuned LLMs (instruct versions) dependent on finetuning data?
UserDisplayName: Hephaestus
ThreadText: # Is the output of finetuned LLMs (instruct versions) dependent on finetuning data?
  
  (Asked by: username_0 on 2024-12-01)
  
  *Note: This question was originally posted on GenAI site, but I'm reposting here because I think that's more appropriate here.*
  
  Are instruct models' outputs affected by finetuning data? E.g. If the finetuning data of a LLM for instruction following includes names of mountains, cities, mythological figures, etc. as function or variable names in source code examples provided in the finetuning input data, will the model mostly provide these name schemes in the output when asked to write source code?
  
  **Tags:** fine-tuning
  > **Comments:**
  > **username_1**: What do you think is the purpose of fine-tuning then? Are you asking what it means to fine-tune?  
  > **username_0**: @nbro Just whether the output is affected  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-01. Score: 0 | Accepted answer
  
  Indeed the outputs of instruction-tuned models might be influenced by the specific data used during their fine-tuning process. If the fine-tuning data for a language model includes source code examples where variable or function names are consistently derived from certain patterns like names of mountains, cities, or mythological figures, this could bias the model to generate outputs that reflect these naming conventions because, for example, in RLHF if the *reward model* or its human annotated *ranking process* prioritizes patterns from specific fine-tuning data or user feedback over general knowledge, the fine-tuning effect will be stronger. By analyzing the difference in outputs between the pre-trained base model and the fine-tuned model with identical prompts, you can assess how much influence the fine-tuning including RLHF exerts.
  
  To mitigate such effect, increase diversity in fine-tuning data, use data augmentation techniques such as randomizing variable and function names during pre-processing to ensure the model doesn’t overfit to specific patterns, and apply prompt engineering by specifying naming conventions in the prompt, etc.
  
  
  ---
  
  