Parrot: Efficient Serving of LLM-based Applications with Semantic
  Variable
The rise of large language models (LLMs) has enabled LLM-based applications
(a.k.a. AI agents or co-pilots), a new software paradigm that combines the
strength of LLM and conventional software. Diverse LLM applications from
different tenants could design complex workflows using multiple LLM requests to
accomplish one task. However, they have to use the over-simplified
request-level API provided by today's public LLM services, losing essential
application-level information. Public LLM services have to blindly optimize
individual LLM requests, leading to sub-optimal end-to-end performance of LLM
applications.
  This paper introduces Parrot, an LLM service system that focuses on the
end-to-end experience of LLM-based applications. Parrot proposes Semantic
Variable, a unified abstraction to expose application-level knowledge to public
LLM services. A Semantic Variable annotates an input/output variable in the
prompt of a request, and creates the data pipeline when connecting multiple LLM
requests, providing a natural way to program LLM applications. Exposing
Semantic Variables to the public LLM service allows it to perform conventional
data flow analysis to uncover the correlation across multiple LLM requests.
This correlation opens a brand-new optimization space for the end-to-end
performance of LLM-based applications. Extensive evaluations demonstrate that
Parrot can achieve up to an order-of-magnitude improvement for popular and
practical use cases of LLM applications.
6659ecd1a6fb89884bb3c66f
Chaofan Lin
admin_assigned
2024-05-31T19:47:00.214Z
644200d95d600fb09520de53
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/prs0wIjQx7PE4-IYkXDvw.jpeg
Chaofan Lin
user
siriusneo
6659ecd1a6fb89884bb3c670
Zhenhua Han
admin_assigned
2024-05-31T19:47:06.456Z
64cc9b96e60d2cddfadca2c8
/avatars/a07755847ec8d05052221d351a3ae20f.svg
Zhenhua Han
user
hzhua
6659ecd1a6fb89884bb3c671
Chengruidong Zhang
admin_assigned
2024-05-31T19:47:15.481Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
6659ecd1a6fb89884bb3c672
Yuqing Yang
6659ecd1a6fb89884bb3c673
Fan Yang
6659ecd1a6fb89884bb3c674
Chen Chen
6659ecd1a6fb89884bb3c675
Lili Qiu
Parrot: Efficient Serving of LLM-based Applications with Semantic
  Variable
The rise of large language models (LLMs) has enabled LLM-based applications
(a.k.a. AI agents or co-pilots), a new software paradigm that combines the
strength of LLM and conventional software. Diverse LLM applications from
different tenants could design complex workflows using multiple LLM requests to
accomplish one task. However, they have to use the over-simplified
request-level API provided by today's public LLM services, losing essential
application-level information. Public LLM services have to blindly optimize
individual LLM requests, leading to sub-optimal end-to-end performance of LLM
applications.
  This paper introduces Parrot, an LLM service system that focuses on the
end-to-end experience of LLM-based applications. Parrot proposes Semantic
Variable, a unified abstraction to expose application-level knowledge to public
LLM services. A Semantic Variable annotates an input/output variable in the
prompt of a request, and creates the data pipeline when connecting multiple LLM
requests, providing a natural way to program LLM applications. Exposing
Semantic Variables to the public LLM service allows it to perform conventional
data flow analysis to uncover the correlation across multiple LLM requests.
This correlation opens a brand-new optimization space for the end-to-end
performance of LLM-based applications. Extensive evaluations demonstrate that
Parrot can achieve up to an order-of-magnitude improvement for popular and
practical use cases of LLM applications.
Parrot, an LLM service system, improves end-to-end performance of LLM applications by introducing Semantic Variables for data pipeline creation and correlation analysis.
large language models LLM AI agents co-pilots LLM-based applications request-level API application-level information end-to-end performance Semantic Variable prompt data pipeline data flow analysis