title: Small Models are Valuable Plug-ins for Large Language Models
summary: Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their
  weights are often publicly unavailable and their immense sizes make the models
  difficult to be tuned with common hardware. As a result, effectively tuning
  these models with large-scale supervised data can be challenging. As an
  alternative, In-Context Learning (ICL) can only use a small number of
  supervised examples due to context length limits. In this paper, we propose
  Super In-Context Learning (SuperICL) which allows black-box LLMs to work with
  locally fine-tuned smaller models, resulting in superior performance on
  supervised tasks. Our experiments demonstrate that SuperICL can improve
  performance beyond state-of-the-art fine-tuned models while addressing the
  instability problem of in-context learning. Furthermore, SuperICL can enhance
  the capabilities of smaller models, such as multilinguality and
  interpretability.
paper_authors._id: 6462de4452193f29576508e5
paper_authors.name: Canwen Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T13:04:48.398Z
paper_authors._id: 5e3c29d8f55e2b62848a5224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618318588221-5e3c29d8f55e2b62848a5224.png
paper_authors.fullname: Canwen Xu
paper_authors.type: user
paper_authors.user: canwenxu
paper_authors._id: 6462de4452193f29576508e6
paper_authors.name: Yichong Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:32:12.579Z
paper_authors._id: 630929395cc6820c2365bac8
paper_authors.avatarUrl: /avatars/a88e4cfdae35570e350a47b3fa1bf9ea.svg
paper_authors.fullname: Yichong
paper_authors.type: user
paper_authors.user: yicxu
paper_authors._id: 6462de4452193f29576508e7
paper_authors.name: Shuohang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:05:01.077Z
paper_authors._id: 6463b2247572c66a8e625a57
paper_authors.avatarUrl: /avatars/7722fb5649d42d966ce1e478946d5f8f.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Shuohang
paper_authors._id: 6462de4452193f29576508e8
paper_authors.name: Yang Liu
paper_authors._id: 6462de4452193f29576508e9
paper_authors.name: Chenguang Zhu
paper_authors._id: 6462de4452193f29576508ea
paper_authors.name: Julian McAuley
paper_title: Small Models are Valuable Plug-ins for Large Language Models
paper_summary: Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their
  weights are often publicly unavailable and their immense sizes make the models
  difficult to be tuned with common hardware. As a result, effectively tuning
  these models with large-scale supervised data can be challenging. As an
  alternative, In-Context Learning (ICL) can only use a small number of
  supervised examples due to context length limits. In this paper, we propose
  Super In-Context Learning (SuperICL) which allows black-box LLMs to work with
  locally fine-tuned smaller models, resulting in superior performance on
  supervised tasks. Our experiments demonstrate that SuperICL can improve
  performance beyond state-of-the-art fine-tuned models while addressing the
  instability problem of in-context learning. Furthermore, SuperICL can enhance
  the capabilities of smaller models, such as multilinguality and
  interpretability.
paper_ai_summary: Super In-Context Learning integrates black-box LLMs with locally fine-tuned smaller models to enhance performance, multilinguality, and interpretability in supervised tasks.
paper_ai_keywords: In-Context Learning Super In-Context Learning