Octopus v2: On-device language model for super agent
Language models have shown effectiveness in a variety of software
applications, particularly in tasks related to automatic workflow. These models
possess the crucial ability to call functions, which is essential in creating
AI agents. Despite the high performance of large-scale language models in cloud
environments, they are often associated with concerns over privacy and cost.
Current on-device models for function calling face issues with latency and
accuracy. Our research presents a new method that empowers an on-device model
with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
and latency, and decrease the context length by 95\%. When compared to Llama-7B
with a RAG-based function calling mechanism, our method enhances latency by
35-fold. This method reduces the latency to levels deemed suitable for
deployment across a variety of edge devices in production environments,
aligning with the performance requisites for real-world applications.
660cb8f527b5ec27265bc10c
Wei Chen
claimed_verified
2024-04-11T20:47:27.720Z
633a0c23c0b7fd23493d21cf
/avatars/585498c2b63d1f46169e864999c37fc6.svg
Wei Chen
user
NexaAIalex
660cb8f527b5ec27265bc10d
Zhiyuan Li
claimed_verified
2024-04-15T06:45:39.282Z
64a07bb3d5c3d57173da0103
https://cdn-avatars.huggingface.co/v1/production/uploads/64a07bb3d5c3d57173da0103/F0GsjtPbwNMfVZ3bohadR.jpeg
Zack Zhiyuan Li
user
zack4ai
Octopus v2: On-device language model for super agent
Language models have shown effectiveness in a variety of software
applications, particularly in tasks related to automatic workflow. These models
possess the crucial ability to call functions, which is essential in creating
AI agents. Despite the high performance of large-scale language models in cloud
environments, they are often associated with concerns over privacy and cost.
Current on-device models for function calling face issues with latency and
accuracy. Our research presents a new method that empowers an on-device model
with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
and latency, and decrease the context length by 95\%. When compared to Llama-7B
with a RAG-based function calling mechanism, our method enhances latency by
35-fold. This method reduces the latency to levels deemed suitable for
deployment across a variety of edge devices in production environments,
aligning with the performance requisites for real-world applications.
A new on-device method improves the performance of a large language model, matching GPT-4's accuracy and latency while reducing context length and enhancing deployment on edge devices.
function calling large-scale language models on-device models GPT-4 RAG-based function calling mechanism context length edge devices