title: Octopus v2: On-device language model for super agent
summary: Language models have shown effectiveness in a variety of software
  applications, particularly in tasks related to automatic workflow. These models
  possess the crucial ability to call functions, which is essential in creating
  AI agents. Despite the high performance of large-scale language models in cloud
  environments, they are often associated with concerns over privacy and cost.
  Current on-device models for function calling face issues with latency and
  accuracy. Our research presents a new method that empowers an on-device model
  with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
  and latency, and decrease the context length by 95\%. When compared to Llama-7B
  with a RAG-based function calling mechanism, our method enhances latency by
  35-fold. This method reduces the latency to levels deemed suitable for
  deployment across a variety of edge devices in production environments,
  aligning with the performance requisites for real-world applications.
paper_authors._id: 660cb8f527b5ec27265bc10c
paper_authors.name: Wei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T20:47:27.720Z
paper_authors._id: 633a0c23c0b7fd23493d21cf
paper_authors.avatarUrl: /avatars/585498c2b63d1f46169e864999c37fc6.svg
paper_authors.fullname: Wei Chen
paper_authors.type: user
paper_authors.user: NexaAI
paper_authors._id: 660cb8f527b5ec27265bc10d
paper_authors.name: Zhiyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-15T06:45:39.282Z
paper_authors._id: 64a07bb3d5c3d57173da0103
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a07bb3d5c3d57173da0103/F0GsjtPbwNMfVZ3bohadR.jpeg
paper_authors.fullname: Zack Zhiyuan Li
paper_authors.type: user
paper_authors.user: zack4ai
paper_title: Octopus v2: On-device language model for super agent
paper_summary: Language models have shown effectiveness in a variety of software
  applications, particularly in tasks related to automatic workflow. These models
  possess the crucial ability to call functions, which is essential in creating
  AI agents. Despite the high performance of large-scale language models in cloud
  environments, they are often associated with concerns over privacy and cost.
  Current on-device models for function calling face issues with latency and
  accuracy. Our research presents a new method that empowers an on-device model
  with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
  and latency, and decrease the context length by 95\%. When compared to Llama-7B
  with a RAG-based function calling mechanism, our method enhances latency by
  35-fold. This method reduces the latency to levels deemed suitable for
  deployment across a variety of edge devices in production environments,
  aligning with the performance requisites for real-world applications.
paper_ai_keywords: function calling AI agents language models large-scale encoder multi-scale GPT-4 Llama-7B RAG-based edge devices