title: FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning
summary: Learning paradigms for large language models (LLMs) currently tend to fall
  within either in-context learning (ICL) or full fine-tuning. Each of these
  comes with their own trade-offs based on available data, model size, compute
  cost, ease-of-use, and final quality with neither solution performing well
  across-the-board. In this article, we first describe ICL and fine-tuning
  paradigms in a way that highlights their natural connections. Based on these
  connections, we propose a new learning paradigm called FIAT that fuses the best
  of these paradigms together, enabling prompt-engineered instructions and
  chain-of-thought reasoning with the very largest models while also using
  similar methods to perform parameter updates on a modestly-sized LLM with
  parameter-efficient tuning. We evaluate FIAT's effectiveness on a variety of
  multilingual tasks and observe that FIAT performs better than both ICL and
  fine-tuning at scales ranging from 100-10,000 training examples. We hope that
  FIAT provides a practical way of harnessing the full potential of LLMs without
  needing to make a hard choice between learning paradigms.
paper_authors._id: 64ffc9535d763775629dad3d
paper_authors.name: Xinyi Wang
paper_authors._id: 64ffc9535d763775629dad3e
paper_authors.name: John Wieting
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:21:03.929Z
paper_authors._id: 623d449d38540758f05ccfc4
paper_authors.avatarUrl: /avatars/d79808850b1a2ef5fe23e08eb6f22d57.svg
paper_authors.fullname: John Wieting
paper_authors.type: user
paper_authors.user: jwieting
paper_authors._id: 64ffc9535d763775629dad3f
paper_authors.name: Jonathan H. Clark
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:21:29.174Z
paper_authors._id: 6244a009a37b7efa7d335e47
paper_authors.avatarUrl: /avatars/465c97aa99e0377b09e2f43813da2701.svg
paper_authors.fullname: Jonathan Clark
paper_authors.type: user
paper_authors.user: jhclark
paper_title: FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning
paper_summary: Learning paradigms for large language models (LLMs) currently tend to fall
  within either in-context learning (ICL) or full fine-tuning. Each of these
  comes with their own trade-offs based on available data, model size, compute
  cost, ease-of-use, and final quality with neither solution performing well
  across-the-board. In this article, we first describe ICL and fine-tuning
  paradigms in a way that highlights their natural connections. Based on these
  connections, we propose a new learning paradigm called FIAT that fuses the best
  of these paradigms together, enabling prompt-engineered instructions and
  chain-of-thought reasoning with the very largest models while also using
  similar methods to perform parameter updates on a modestly-sized LLM with
  parameter-efficient tuning. We evaluate FIAT's effectiveness on a variety of
  multilingual tasks and observe that FIAT performs better than both ICL and
  fine-tuning at scales ranging from 100-10,000 training examples. We hope that
  FIAT provides a practical way of harnessing the full potential of LLMs without
  needing to make a hard choice between learning paradigms.
paper_ai_summary: FIAT, a new learning paradigm, combines in-context learning and fine-tuning to improve performance across multilingual tasks using different model sizes.
paper_ai_keywords: in-context learning fine-tuning FIAT prompt-engineering chain-of-thought reasoning parameter updates parameter-efficient tuning multilingual tasks