Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small
  Scorer
Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in
multi-tasking under a unified instruction-following paradigm, where they also
exhibit remarkable generalization abilities to unseen tasks. Despite their
impressive performance, these LLMs, with sizes ranging from several billion to
hundreds of billions of parameters, demand substantial computational resources,
making their training and inference expensive and inefficient. Furthermore,
adapting these models to downstream applications, particularly complex tasks,
is often unfeasible due to the extensive hardware requirements for finetuning,
even when utilizing parameter-efficient approaches such as prompt tuning.
Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and
FLAN-PaLM-540B, are not publicly accessible, severely limiting their
customization potential. To address these challenges, we introduce a pretrained
small scorer, Cappy, designed to enhance the performance and efficiency of
multi-task LLMs. With merely 360 million parameters, Cappy functions either
independently on classification tasks or serve as an auxiliary component for
LLMs, boosting their performance. Moreover, Cappy enables efficiently
integrating downstream supervision without requiring LLM finetuning nor the
access to their parameters. Our experiments demonstrate that, when working
independently on 11 language understanding tasks from PromptSource, Cappy
outperforms LLMs that are several orders of magnitude larger. Besides, on 45
complex tasks from BIG-Bench, Cappy boosts the performance of the advanced
multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to
cooperate with other LLM adaptations, including finetuning and in-context
learning, offering additional performance enhancement.
6552ed5ccfd819e1ad32669b
Bowen Tan
claimed_verified
2023-12-07T08:30:18.595Z
64c52182d43e4dee5177e8b7
/avatars/d13edf5fe3f7f21bb2f22363378b0d9d.svg
Bowen Tan
user
btan2
6552ed5ccfd819e1ad32669c
Yun Zhu
6552ed5ccfd819e1ad32669d
Lijuan Liu
admin_assigned
2023-11-14T11:03:32.156Z
634390fb07fa6ff0c228933c
/avatars/a9ccb879c6f2d6dc05e77ab88818d753.svg
LIU LIJUAN
user
liulj
6552ed5ccfd819e1ad32669e
Eric Xing
admin_assigned
2023-11-14T11:03:40.174Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
6552ed5ccfd819e1ad32669f
Zhiting Hu
6552ed5ccfd819e1ad3266a0
Jindong Chen
admin_assigned
2023-11-14T11:04:01.026Z
635242b8b3678a43742baec6
/avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
chenjindong
user
chenjindong
Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small
  Scorer
Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in
multi-tasking under a unified instruction-following paradigm, where they also
exhibit remarkable generalization abilities to unseen tasks. Despite their
impressive performance, these LLMs, with sizes ranging from several billion to
hundreds of billions of parameters, demand substantial computational resources,
making their training and inference expensive and inefficient. Furthermore,
adapting these models to downstream applications, particularly complex tasks,
is often unfeasible due to the extensive hardware requirements for finetuning,
even when utilizing parameter-efficient approaches such as prompt tuning.
Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and
FLAN-PaLM-540B, are not publicly accessible, severely limiting their
customization potential. To address these challenges, we introduce a pretrained
small scorer, Cappy, designed to enhance the performance and efficiency of
multi-task LLMs. With merely 360 million parameters, Cappy functions either
independently on classification tasks or serve as an auxiliary component for
LLMs, boosting their performance. Moreover, Cappy enables efficiently
integrating downstream supervision without requiring LLM finetuning nor the
access to their parameters. Our experiments demonstrate that, when working
independently on 11 language understanding tasks from PromptSource, Cappy
outperforms LLMs that are several orders of magnitude larger. Besides, on 45
complex tasks from BIG-Bench, Cappy boosts the performance of the advanced
multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to
cooperate with other LLM adaptations, including finetuning and in-context
learning, offering additional performance enhancement.
A smaller scorer model called Cappy enhances the performance and efficiency of large language models on various tasks without requiring extensive finetuning or access to the LLM parameters.
large language models multi-tasking instruction-following paradigm parameter-efficient prompt tuning pretrained small scorer auxiliary component downstream supervision classification tasks PromptSource BIG-Bench FLAN-T5 in-context learning