Small Models are Valuable Plug-ins for Large Language Models
Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their
weights are often publicly unavailable and their immense sizes make the models
difficult to be tuned with common hardware. As a result, effectively tuning
these models with large-scale supervised data can be challenging. As an
alternative, In-Context Learning (ICL) can only use a small number of
supervised examples due to context length limits. In this paper, we propose
Super In-Context Learning (SuperICL) which allows black-box LLMs to work with
locally fine-tuned smaller models, resulting in superior performance on
supervised tasks. Our experiments demonstrate that SuperICL can improve
performance beyond state-of-the-art fine-tuned models while addressing the
instability problem of in-context learning. Furthermore, SuperICL can enhance
the capabilities of smaller models, such as multilinguality and
interpretability.
6462de4452193f29576508e5
Canwen Xu
claimed_verified
2023-05-16T13:04:48.398Z
5e3c29d8f55e2b62848a5224
https://cdn-avatars.huggingface.co/v1/production/uploads/1618318588221-5e3c29d8f55e2b62848a5224.png
Canwen Xu
user
canwenxu
6462de4452193f29576508e6
Yichong Xu
claimed_verified
2023-05-16T15:32:12.579Z
630929395cc6820c2365bac8
/avatars/a88e4cfdae35570e350a47b3fa1bf9ea.svg
Yichong
user
yicxu
6462de4452193f29576508e7
Shuohang Wang
claimed_verified
2023-05-17T08:05:01.077Z
6463b2247572c66a8e625a57
/avatars/7722fb5649d42d966ce1e478946d5f8f.svg
Wang
user
Shuohang
6462de4452193f29576508e8
Yang Liu
6462de4452193f29576508e9
Chenguang Zhu
6462de4452193f29576508ea
Julian McAuley
Small Models are Valuable Plug-ins for Large Language Models
Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their
weights are often publicly unavailable and their immense sizes make the models
difficult to be tuned with common hardware. As a result, effectively tuning
these models with large-scale supervised data can be challenging. As an
alternative, In-Context Learning (ICL) can only use a small number of
supervised examples due to context length limits. In this paper, we propose
Super In-Context Learning (SuperICL) which allows black-box LLMs to work with
locally fine-tuned smaller models, resulting in superior performance on
supervised tasks. Our experiments demonstrate that SuperICL can improve
performance beyond state-of-the-art fine-tuned models while addressing the
instability problem of in-context learning. Furthermore, SuperICL can enhance
the capabilities of smaller models, such as multilinguality and
interpretability.
Super In-Context Learning integrates black-box LLMs with locally fine-tuned smaller models to enhance performance, multilinguality, and interpretability in supervised tasks.
In-Context Learning Super In-Context Learning