title: Symbol tuning improves in-context learning in language models
summary: We present symbol tuning - finetuning language models on in-context
  input-label pairs where natural language labels (e.g., "positive/negative
  sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol
  tuning leverages the intuition that when a model cannot use instructions or
  natural language labels to figure out a task, it must instead do so by learning
  the input-label mappings.
    We experiment with symbol tuning across Flan-PaLM models up to 540B
  parameters and observe benefits across various settings. First, symbol tuning
  boosts performance on unseen in-context learning tasks and is much more robust
  to underspecified prompts, such as those without instructions or without
  natural language labels. Second, symbol-tuned models are much stronger at
  algorithmic reasoning tasks, with up to 18.2% better performance on the List
  Functions benchmark and up to 15.3% better performance on the Simple Turing
  Concepts benchmark. Finally, symbol-tuned models show large improvements in
  following flipped-labels presented in-context, meaning that they are more
  capable of using in-context information to override prior semantic knowledge.
paper_authors._id: 6462eeed8bd7025ebe857712
paper_authors.name: Jerry Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-10T01:14:08.719Z
paper_authors._id: 60db3b451f1eec955a46d065
paper_authors.avatarUrl: /avatars/10bd161ba6b56717feba9fbf539db4ea.svg
paper_authors.fullname: Jerry Wei
paper_authors.type: user
paper_authors.user: jerrywei
paper_authors._id: 6462eeed8bd7025ebe857713
paper_authors.name: Le Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:04:40.025Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 6462eeed8bd7025ebe857714
paper_authors.name: Andrew Lampinen
paper_authors._id: 6462eeed8bd7025ebe857715
paper_authors.name: Xiangning Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:04:34.385Z
paper_authors._id: 631bc3584aa47d6afbe440bd
paper_authors.avatarUrl: /avatars/cd9a605fbdf5454db9b4f714b28ff3fb.svg
paper_authors.fullname: Xiangning Chen
paper_authors.type: user
paper_authors.user: xiangningc
paper_authors._id: 6462eeed8bd7025ebe857716
paper_authors.name: Da Huang
paper_authors._id: 6462eeed8bd7025ebe857717
paper_authors.name: Yi Tay
paper_authors._id: 6462eeed8bd7025ebe857718
paper_authors.name: Xinyun Chen
paper_authors._id: 6462eeed8bd7025ebe857719
paper_authors.name: Yifeng Lu
paper_authors._id: 6462eeed8bd7025ebe85771a
paper_authors.name: Denny Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T18:28:42.759Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_authors._id: 6462eeed8bd7025ebe85771b
paper_authors.name: Tengyu Ma
paper_authors._id: 6462eeed8bd7025ebe85771c
paper_authors.name: Quoc V. Le
paper_title: Symbol tuning improves in-context learning in language models
paper_summary: We present symbol tuning - finetuning language models on in-context
  input-label pairs where natural language labels (e.g., "positive/negative
  sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol
  tuning leverages the intuition that when a model cannot use instructions or
  natural language labels to figure out a task, it must instead do so by learning
  the input-label mappings.
    We experiment with symbol tuning across Flan-PaLM models up to 540B
  parameters and observe benefits across various settings. First, symbol tuning
  boosts performance on unseen in-context learning tasks and is much more robust
  to underspecified prompts, such as those without instructions or without
  natural language labels. Second, symbol-tuned models are much stronger at
  algorithmic reasoning tasks, with up to 18.2% better performance on the List
  Functions benchmark and up to 15.3% better performance on the Simple Turing
  Concepts benchmark. Finally, symbol-tuned models show large improvements in
  following flipped-labels presented in-context, meaning that they are more
  capable of using in-context information to override prior semantic knowledge.
paper_ai_keywords: symbol tuning in-context input-label pairs algorithmic reasoning tasks List Functions benchmark Simple Turing Concepts benchmark flipped-labels