title: FrugalGPT: How to Use Large Language Models While Reducing Cost and
    Improving Performance
summary: There is a rapidly growing number of large language models (LLMs) that users
  can query for a fee. We review the cost associated with querying popular LLM
  APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have
  heterogeneous pricing structures, with fees that can differ by two orders of
  magnitude. In particular, using LLMs on large collections of queries and text
  can be expensive. Motivated by this, we outline and discuss three types of
  strategies that users can exploit to reduce the inference cost associated with
  using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As
  an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM
  cascade which learns which combinations of LLMs to use for different queries in
  order to reduce cost and improve accuracy. Our experiments show that FrugalGPT
  can match the performance of the best individual LLM (e.g. GPT-4) with up to
  98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.
  The ideas and findings presented here lay a foundation for using LLMs
  sustainably and efficiently.
paper_authors._id: 645aec70c266796265bbe696
paper_authors.name: Lingjiao Chen
paper_authors._id: 645aec70c266796265bbe697
paper_authors.name: Matei Zaharia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-11T11:45:47.019Z
paper_authors._id: 642509c6d476e4ad5568dd31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642509c6d476e4ad5568dd31/dPiEV70hMeuY-cveLNPQg.jpeg
paper_authors.fullname: Matei Zaharia
paper_authors.type: user
paper_authors.user: mateiz
paper_authors._id: 645aec70c266796265bbe698
paper_authors.name: James Zou
paper_title: FrugalGPT: How to Use Large Language Models While Reducing Cost and
    Improving Performance
paper_summary: There is a rapidly growing number of large language models (LLMs) that users
  can query for a fee. We review the cost associated with querying popular LLM
  APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have
  heterogeneous pricing structures, with fees that can differ by two orders of
  magnitude. In particular, using LLMs on large collections of queries and text
  can be expensive. Motivated by this, we outline and discuss three types of
  strategies that users can exploit to reduce the inference cost associated with
  using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As
  an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM
  cascade which learns which combinations of LLMs to use for different queries in
  order to reduce cost and improve accuracy. Our experiments show that FrugalGPT
  can match the performance of the best individual LLM (e.g. GPT-4) with up to
  98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.
  The ideas and findings presented here lay a foundation for using LLMs
  sustainably and efficiently.
paper_ai_summary: Strategies including prompt adaptation, LLM approximation, and LLM cascade are proposed to reduce costs and maintain or improve performance when using large language models.
paper_ai_keywords: large language models (LLMs) prompt adaptation LLM approximation LLM cascade FrugalGPT