title: Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small
    Scorer
summary: Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in
  multi-tasking under a unified instruction-following paradigm, where they also
  exhibit remarkable generalization abilities to unseen tasks. Despite their
  impressive performance, these LLMs, with sizes ranging from several billion to
  hundreds of billions of parameters, demand substantial computational resources,
  making their training and inference expensive and inefficient. Furthermore,
  adapting these models to downstream applications, particularly complex tasks,
  is often unfeasible due to the extensive hardware requirements for finetuning,
  even when utilizing parameter-efficient approaches such as prompt tuning.
  Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and
  FLAN-PaLM-540B, are not publicly accessible, severely limiting their
  customization potential. To address these challenges, we introduce a pretrained
  small scorer, Cappy, designed to enhance the performance and efficiency of
  multi-task LLMs. With merely 360 million parameters, Cappy functions either
  independently on classification tasks or serve as an auxiliary component for
  LLMs, boosting their performance. Moreover, Cappy enables efficiently
  integrating downstream supervision without requiring LLM finetuning nor the
  access to their parameters. Our experiments demonstrate that, when working
  independently on 11 language understanding tasks from PromptSource, Cappy
  outperforms LLMs that are several orders of magnitude larger. Besides, on 45
  complex tasks from BIG-Bench, Cappy boosts the performance of the advanced
  multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to
  cooperate with other LLM adaptations, including finetuning and in-context
  learning, offering additional performance enhancement.
paper_authors._id: 6552ed5ccfd819e1ad32669b
paper_authors.name: Bowen Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:30:18.595Z
paper_authors._id: 64c52182d43e4dee5177e8b7
paper_authors.avatarUrl: /avatars/d13edf5fe3f7f21bb2f22363378b0d9d.svg
paper_authors.fullname: Bowen Tan
paper_authors.type: user
paper_authors.user: btan2
paper_authors._id: 6552ed5ccfd819e1ad32669c
paper_authors.name: Yun Zhu
paper_authors._id: 6552ed5ccfd819e1ad32669d
paper_authors.name: Lijuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:03:32.156Z
paper_authors._id: 634390fb07fa6ff0c228933c
paper_authors.avatarUrl: /avatars/a9ccb879c6f2d6dc05e77ab88818d753.svg
paper_authors.fullname: LIU LIJUAN
paper_authors.type: user
paper_authors.user: liulj
paper_authors._id: 6552ed5ccfd819e1ad32669e
paper_authors.name: Eric Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:03:40.174Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 6552ed5ccfd819e1ad32669f
paper_authors.name: Zhiting Hu
paper_authors._id: 6552ed5ccfd819e1ad3266a0
paper_authors.name: Jindong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T11:04:01.026Z
paper_authors._id: 635242b8b3678a43742baec6
paper_authors.avatarUrl: /avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
paper_authors.fullname: chenjindong
paper_authors.type: user
paper_authors.user: chenjindong
paper_title: Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small
    Scorer
paper_summary: Large language models (LLMs) such as T0, FLAN, and OPT-IML, excel in
  multi-tasking under a unified instruction-following paradigm, where they also
  exhibit remarkable generalization abilities to unseen tasks. Despite their
  impressive performance, these LLMs, with sizes ranging from several billion to
  hundreds of billions of parameters, demand substantial computational resources,
  making their training and inference expensive and inefficient. Furthermore,
  adapting these models to downstream applications, particularly complex tasks,
  is often unfeasible due to the extensive hardware requirements for finetuning,
  even when utilizing parameter-efficient approaches such as prompt tuning.
  Additionally, the most powerful multi-task LLMs, such as OPT-IML-175B and
  FLAN-PaLM-540B, are not publicly accessible, severely limiting their
  customization potential. To address these challenges, we introduce a pretrained
  small scorer, Cappy, designed to enhance the performance and efficiency of
  multi-task LLMs. With merely 360 million parameters, Cappy functions either
  independently on classification tasks or serve as an auxiliary component for
  LLMs, boosting their performance. Moreover, Cappy enables efficiently
  integrating downstream supervision without requiring LLM finetuning nor the
  access to their parameters. Our experiments demonstrate that, when working
  independently on 11 language understanding tasks from PromptSource, Cappy
  outperforms LLMs that are several orders of magnitude larger. Besides, on 45
  complex tasks from BIG-Bench, Cappy boosts the performance of the advanced
  multi-task LLM, FLAN-T5, by a large margin. Furthermore, Cappy is flexible to
  cooperate with other LLM adaptations, including finetuning and in-context
  learning, offering additional performance enhancement.
paper_ai_summary: A smaller scorer model called Cappy enhances the performance and efficiency of large language models on various tasks without requiring extensive finetuning or access to the LLM parameters.
paper_ai_keywords: large language models multi-tasking instruction-following paradigm parameter-efficient prompt tuning pretrained small scorer auxiliary component downstream supervision classification tasks PromptSource BIG-Bench FLAN-T5 in-context learning