Answers.Body: <p>Ok-Translator-5878 gave you a good reference on Reddit:</p>
  <p><a href="https://aclanthology.org/2023.findings-acl.247/" rel="nofollow noreferrer">Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers</a>
  Damai Dai
  ,
  Yutao Sun
  ,
  Li Dong
  ,
  Yaru Hao
  ,
  Shuming Ma
  ,
  Zhifang Sui
  ,
  Furu Wei
  ACL 2023 | July 2023</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T21:17:14.223Z
Answers.LastActivityDate: 2024-05-24T21:17:14.223Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I came across a Google's blog (<a href="https://research.google/blog/larger-language-models-do-in-context-learning-differently/" rel="nofollow noreferrer">https://research.google/blog/larger-language-models-do-in-context-learning-differently/</a>) discussing large language models (LLMs) and how we can overried their prior knowledge through in-context. Using examples where the labels contradict prior knowledge, they call it flipped-label in-context learning (for example, sentences containing positive sentiment labeled as “negative sentiment”).</p>
  <p><a href="https://i.sstatic.net/MC5fCypB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MC5fCypB.png" alt="enter image description here" /></a></p>
  <p>I'm curious: how does the model is able to even &quot;learn&quot; (overried its priors) without changing its weights? Specifically, how does it adjust its understanding when faced with contradictory labels like positive sentences labeled as negative?</p>
  
Tags: <deep-learning><large-language-models><fine-tuning><google>
Title: How is a LLM able to override its prior knowledge through In-Context Learning?
UserDisplayName: tidealwari91
ThreadText: # How is a LLM able to override its prior knowledge through In-Context Learning?
  
  (Asked by: username_0 on 2024-05-08)
  
  I came across a Google's blog (<https://research.google/blog/larger-language-models-do-in-context-learning-differently/>) discussing large language models (LLMs) and how we can overried their prior knowledge through in-context. Using examples where the labels contradict prior knowledge, they call it flipped-label in-context learning (for example, sentences containing positive sentiment labeled as “negative sentiment”).
  
  [![enter image description here](https://i.sstatic.net/MC5fCypB.png)](https://i.sstatic.net/MC5fCypB.png)
  
  I'm curious: how does the model is able to even "learn" (overried its priors) without changing its weights? Specifically, how does it adjust its understanding when faced with contradictory labels like positive sentences labeled as negative?
  
  **Tags:** deep-learning, large-language-models, fine-tuning, google
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-24. Score: 1
  
  Ok-Translator-5878 gave you a good reference on Reddit:
  
  [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://aclanthology.org/2023.findings-acl.247/)
  Damai Dai
  ,
  Yutao Sun
  ,
  Li Dong
  ,
  Yaru Hao
  ,
  Shuming Ma
  ,
  Zhifang Sui
  ,
  Furu Wei
  ACL 2023 | July 2023
  
  
  ---
  
  