title: Parrot: Efficient Serving of LLM-based Applications with Semantic
    Variable
summary: The rise of large language models (LLMs) has enabled LLM-based applications
  (a.k.a. AI agents or co-pilots), a new software paradigm that combines the
  strength of LLM and conventional software. Diverse LLM applications from
  different tenants could design complex workflows using multiple LLM requests to
  accomplish one task. However, they have to use the over-simplified
  request-level API provided by today's public LLM services, losing essential
  application-level information. Public LLM services have to blindly optimize
  individual LLM requests, leading to sub-optimal end-to-end performance of LLM
  applications.
    This paper introduces Parrot, an LLM service system that focuses on the
  end-to-end experience of LLM-based applications. Parrot proposes Semantic
  Variable, a unified abstraction to expose application-level knowledge to public
  LLM services. A Semantic Variable annotates an input/output variable in the
  prompt of a request, and creates the data pipeline when connecting multiple LLM
  requests, providing a natural way to program LLM applications. Exposing
  Semantic Variables to the public LLM service allows it to perform conventional
  data flow analysis to uncover the correlation across multiple LLM requests.
  This correlation opens a brand-new optimization space for the end-to-end
  performance of LLM-based applications. Extensive evaluations demonstrate that
  Parrot can achieve up to an order-of-magnitude improvement for popular and
  practical use cases of LLM applications.
paper_authors._id: 6659ecd1a6fb89884bb3c66f
paper_authors.name: Chaofan Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:47:00.214Z
paper_authors._id: 644200d95d600fb09520de53
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/prs0wIjQx7PE4-IYkXDvw.jpeg
paper_authors.fullname: Chaofan Lin
paper_authors.type: user
paper_authors.user: siriusneo
paper_authors._id: 6659ecd1a6fb89884bb3c670
paper_authors.name: Zhenhua Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:47:06.456Z
paper_authors._id: 64cc9b96e60d2cddfadca2c8
paper_authors.avatarUrl: /avatars/a07755847ec8d05052221d351a3ae20f.svg
paper_authors.fullname: Zhenhua Han
paper_authors.type: user
paper_authors.user: hzhua
paper_authors._id: 6659ecd1a6fb89884bb3c671
paper_authors.name: Chengruidong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:47:15.481Z
paper_authors._id: 64646896884f2e3e1ced3cd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chengruidong
paper_authors._id: 6659ecd1a6fb89884bb3c672
paper_authors.name: Yuqing Yang
paper_authors._id: 6659ecd1a6fb89884bb3c673
paper_authors.name: Fan Yang
paper_authors._id: 6659ecd1a6fb89884bb3c674
paper_authors.name: Chen Chen
paper_authors._id: 6659ecd1a6fb89884bb3c675
paper_authors.name: Lili Qiu
paper_title: Parrot: Efficient Serving of LLM-based Applications with Semantic
    Variable
paper_summary: The rise of large language models (LLMs) has enabled LLM-based applications
  (a.k.a. AI agents or co-pilots), a new software paradigm that combines the
  strength of LLM and conventional software. Diverse LLM applications from
  different tenants could design complex workflows using multiple LLM requests to
  accomplish one task. However, they have to use the over-simplified
  request-level API provided by today's public LLM services, losing essential
  application-level information. Public LLM services have to blindly optimize
  individual LLM requests, leading to sub-optimal end-to-end performance of LLM
  applications.
    This paper introduces Parrot, an LLM service system that focuses on the
  end-to-end experience of LLM-based applications. Parrot proposes Semantic
  Variable, a unified abstraction to expose application-level knowledge to public
  LLM services. A Semantic Variable annotates an input/output variable in the
  prompt of a request, and creates the data pipeline when connecting multiple LLM
  requests, providing a natural way to program LLM applications. Exposing
  Semantic Variables to the public LLM service allows it to perform conventional
  data flow analysis to uncover the correlation across multiple LLM requests.
  This correlation opens a brand-new optimization space for the end-to-end
  performance of LLM-based applications. Extensive evaluations demonstrate that
  Parrot can achieve up to an order-of-magnitude improvement for popular and
  practical use cases of LLM applications.
paper_ai_summary: Parrot, an LLM service system, improves end-to-end performance of LLM applications by introducing Semantic Variables for data pipeline creation and correlation analysis.
paper_ai_keywords: large language models LLM AI agents co-pilots LLM-based applications request-level API application-level information end-to-end performance Semantic Variable prompt data pipeline data flow analysis