title: EcoAssistant: Using LLM Assistant More Affordably and Accurately
summary: Today, users ask Large language models (LLMs) as assistants to answer queries
  that require external knowledge; they ask about the weather in a specific city,
  about stock prices, and even about where specific locations are within their
  neighborhood. These queries require the LLM to produce code that invokes
  external APIs to answer the user's question, yet LLMs rarely produce correct
  code on the first try, requiring iterative code refinement upon execution
  results. In addition, using LLM assistants to support high query volumes can be
  expensive. In this work, we contribute a framework, EcoAssistant, that enables
  LLMs to answer code-driven queries more affordably and accurately. EcoAssistant
  contains three components. First, it allows the LLM assistants to converse with
  an automatic code executor to iteratively refine code or to produce answers
  based on the execution results. Second, we use a hierarchy of LLM assistants,
  which attempts to answer the query with weaker, cheaper LLMs before backing off
  to stronger, expensive ones. Third, we retrieve solutions from past successful
  queries as in-context demonstrations to help subsequent queries. Empirically,
  we show that EcoAssistant offers distinct advantages for affordability and
  accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of
  GPT-4's cost.
paper_authors._id: 651f60545ff73c3cec5678ee
paper_authors.name: Jieyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-09T10:02:53.745Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 651f60545ff73c3cec5678ef
paper_authors.name: Ranjay Krishna
paper_authors._id: 651f60545ff73c3cec5678f0
paper_authors.name: Ahmed H. Awadallah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:28:33.399Z
paper_authors._id: 6398c705cc8635e998d3c874
paper_authors.avatarUrl: /avatars/3f3f56021cd74c561bde118d6ad7439d.svg
paper_authors.fullname: Ahmed Awadallah
paper_authors.type: user
paper_authors.user: aawadall
paper_authors._id: 651f60545ff73c3cec5678f1
paper_authors.name: Chi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:29:15.329Z
paper_authors._id: 640baf807047b6c332a912e1
paper_authors.avatarUrl: /avatars/830fd35aff114eedf68c4f9eeb9fe944.svg
paper_authors.fullname: Chi Wang
paper_authors.type: user
paper_authors.user: sonichi
paper_title: EcoAssistant: Using LLM Assistant More Affordably and Accurately
paper_summary: Today, users ask Large language models (LLMs) as assistants to answer queries
  that require external knowledge; they ask about the weather in a specific city,
  about stock prices, and even about where specific locations are within their
  neighborhood. These queries require the LLM to produce code that invokes
  external APIs to answer the user's question, yet LLMs rarely produce correct
  code on the first try, requiring iterative code refinement upon execution
  results. In addition, using LLM assistants to support high query volumes can be
  expensive. In this work, we contribute a framework, EcoAssistant, that enables
  LLMs to answer code-driven queries more affordably and accurately. EcoAssistant
  contains three components. First, it allows the LLM assistants to converse with
  an automatic code executor to iteratively refine code or to produce answers
  based on the execution results. Second, we use a hierarchy of LLM assistants,
  which attempts to answer the query with weaker, cheaper LLMs before backing off
  to stronger, expensive ones. Third, we retrieve solutions from past successful
  queries as in-context demonstrations to help subsequent queries. Empirically,
  we show that EcoAssistant offers distinct advantages for affordability and
  accuracy, surpassing GPT-4 by 10 points of success rate with less than 50% of
  GPT-4's cost.
paper_ai_summary: EcoAssistant framework enables more affordable and accurate responses from LLMs through iterative code refinement, LLM hierarchy, and solution retrieval.
paper_ai_keywords: Large language models LLMs automatic code executor hierarchical LLMs in-context demonstrations