title: Skills-in-Context Prompting: Unlocking Compositionality in Large
    Language Models
summary: We consider the problem of eliciting compositional generalization
  capabilities in large language models (LLMs) with a novel type of prompting
  strategy. Compositional generalization empowers the LLMs to solve problems that
  are harder than the ones they have seen (i.e., easy-to-hard generalization),
  which is a critical reasoning capability of human-like intelligence. However,
  even the current state-of-the-art LLMs still struggle with this form of
  reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting,
  which instructs LLMs how to compose basic skills to resolve more complex
  problems. We find that it is crucial to demonstrate both the skills and the
  compositional examples within the same prompting context. With as few as two
  examplars, our SKiC prompting initiates strong synergies between skills and
  their composition capabilities. Notably, it empowers LLMs to solve unseen
  problems that require innovative skill compositions, achieving near-perfect
  generalization on a broad range of challenging compositionality tasks.
  Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling
  them to leverage pre-existing internal skills acquired during earlier
  pre-training stages, even when these skills are not explicitly presented in the
  prompting context. This results in the capability of LLMs to solve unseen
  complex problems by activating and composing internal competencies. With such
  prominent features, SKiC prompting is able to achieve state-of-the-art
  performance on challenging mathematical reasoning benchmarks (e.g., MATH).
paper_authors._id: 64c9b91af7f4ccb5ea6573de
paper_authors.name: Jiaao Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-02T07:45:41.838Z
paper_authors._id: 6327c45946ffcfc27cfa1ee3
paper_authors.avatarUrl: /avatars/f64b36e978fc369bfdfebc16057cd67e.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Jiaao
paper_authors._id: 64c9b91af7f4ccb5ea6573df
paper_authors.name: Xiaoman Pan
paper_authors._id: 64c9b91af7f4ccb5ea6573e0
paper_authors.name: Dian Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-20T07:35:16.524Z
paper_authors._id: 62d58fd53bf5e059f7cc3245
paper_authors.avatarUrl: /avatars/7a4f3ee4a37245f67efd26749d66a706.svg
paper_authors.fullname: Dian Yu
paper_authors.type: user
paper_authors.user: yudian
paper_authors._id: 64c9b91af7f4ccb5ea6573e1
paper_authors.name: Kaiqiang Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-07T12:38:53.463Z
paper_authors._id: 63a261f13c003e409322c191
paper_authors.avatarUrl: /avatars/dc4be6ce1983d0d32620275df124fe45.svg
paper_authors.fullname: Kaiqiang Song
paper_authors.type: user
paper_authors.user: kqsong
paper_authors._id: 64c9b91af7f4ccb5ea6573e2
paper_authors.name: Xiaoyang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T08:48:52.155Z
paper_authors._id: 657cd228138b7e391444a65d
paper_authors.avatarUrl: /avatars/c7c984ae483144fab627aa2c54d91d0f.svg
paper_authors.fullname: Xiaoyang Wang
paper_authors.type: user
paper_authors.user: xywang1
paper_authors._id: 64c9b91af7f4ccb5ea6573e3
paper_authors.name: Dong Yu
paper_authors._id: 64c9b91af7f4ccb5ea6573e4
paper_authors.name: Jianshu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-02T07:48:36.355Z
paper_authors._id: 648353887641590704d53894
paper_authors.avatarUrl: /avatars/d0968259531561f8c93f37bddefe2268.svg
paper_authors.fullname: Jianshu Chen
paper_authors.type: user
paper_authors.user: chenjianshu
paper_title: Skills-in-Context Prompting: Unlocking Compositionality in Large
    Language Models
paper_summary: We consider the problem of eliciting compositional generalization
  capabilities in large language models (LLMs) with a novel type of prompting
  strategy. Compositional generalization empowers the LLMs to solve problems that
  are harder than the ones they have seen (i.e., easy-to-hard generalization),
  which is a critical reasoning capability of human-like intelligence. However,
  even the current state-of-the-art LLMs still struggle with this form of
  reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting,
  which instructs LLMs how to compose basic skills to resolve more complex
  problems. We find that it is crucial to demonstrate both the skills and the
  compositional examples within the same prompting context. With as few as two
  examplars, our SKiC prompting initiates strong synergies between skills and
  their composition capabilities. Notably, it empowers LLMs to solve unseen
  problems that require innovative skill compositions, achieving near-perfect
  generalization on a broad range of challenging compositionality tasks.
  Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling
  them to leverage pre-existing internal skills acquired during earlier
  pre-training stages, even when these skills are not explicitly presented in the
  prompting context. This results in the capability of LLMs to solve unseen
  complex problems by activating and composing internal competencies. With such
  prominent features, SKiC prompting is able to achieve state-of-the-art
  performance on challenging mathematical reasoning benchmarks (e.g., MATH).
paper_ai_keywords: compositional generalization large language models (LLMs) testing strategy skills-in-context (SKiC) prompting prompting context skill composition reasoning capability MATH benchmark pre-training stages internal competencies