title: LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to
    Small-Scale Local LLMs
summary: The widespread adoption of cloud-based proprietary large language models
  (LLMs) has introduced significant challenges, including operational
  dependencies, privacy concerns, and the necessity of continuous internet
  connectivity. In this work, we introduce an LLMOps pipeline, "LlamaDuo", for
  the seamless migration of knowledge and abilities from service-oriented LLMs to
  smaller, locally manageable models. This pipeline is crucial for ensuring
  service continuity in the presence of operational failures, strict privacy
  policies, or offline requirements. Our LlamaDuo involves fine-tuning a small
  language model against the service LLM using a synthetic dataset generated by
  the latter. If the performance of the fine-tuned model falls short of
  expectations, it is enhanced by further fine-tuning with additional similar
  data created by the service LLM. This iterative process guarantees that the
  smaller model can eventually match or even surpass the service LLM's
  capabilities in specific downstream tasks, offering a practical and scalable
  solution for managing AI deployments in constrained environments. Extensive
  experiments with leading edge LLMs are conducted to demonstrate the
  effectiveness, adaptability, and affordability of LlamaDuo across various
  downstream tasks. Our pipeline implementation is available at
  https://github.com/deep-diver/llamaduo.
paper_authors._id: 66cd31e4a9107693c2dbafbb
paper_authors.name: Chansung Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:38:48.610Z
paper_authors._id: 60d3b57ad7b174177faabd6e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659971187637-60d3b57ad7b174177faabd6e.jpeg
paper_authors.fullname: chansung park
paper_authors.type: user
paper_authors.user: chansung
paper_authors._id: 66cd31e4a9107693c2dbafbc
paper_authors.name: Juyong Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:48:26.456Z
paper_authors._id: 64b63712b3b69063d9aff341
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b63712b3b69063d9aff341/nP6GyBKBtnWWoA0TvQF3K.jpeg
paper_authors.fullname: John
paper_authors.type: user
paper_authors.user: juyongjiang
paper_authors._id: 66cd31e4a9107693c2dbafbd
paper_authors.name: Fan Wang
paper_authors._id: 66cd31e4a9107693c2dbafbe
paper_authors.name: Sayak Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T12:50:17.512Z
paper_authors._id: 5f7fbd813e94f16a85448745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
paper_authors.fullname: Sayak Paul
paper_authors.type: user
paper_authors.user: sayakpaul
paper_authors._id: 66cd31e4a9107693c2dbafbf
paper_authors.name: Jing Tang
paper_authors._id: 66cd31e4a9107693c2dbafc0
paper_authors.name: Sunghun Kim
paper_title: LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to
    Small-Scale Local LLMs
paper_summary: The widespread adoption of cloud-based proprietary large language models
  (LLMs) has introduced significant challenges, including operational
  dependencies, privacy concerns, and the necessity of continuous internet
  connectivity. In this work, we introduce an LLMOps pipeline, "LlamaDuo", for
  the seamless migration of knowledge and abilities from service-oriented LLMs to
  smaller, locally manageable models. This pipeline is crucial for ensuring
  service continuity in the presence of operational failures, strict privacy
  policies, or offline requirements. Our LlamaDuo involves fine-tuning a small
  language model against the service LLM using a synthetic dataset generated by
  the latter. If the performance of the fine-tuned model falls short of
  expectations, it is enhanced by further fine-tuning with additional similar
  data created by the service LLM. This iterative process guarantees that the
  smaller model can eventually match or even surpass the service LLM's
  capabilities in specific downstream tasks, offering a practical and scalable
  solution for managing AI deployments in constrained environments. Extensive
  experiments with leading edge LLMs are conducted to demonstrate the
  effectiveness, adaptability, and affordability of LlamaDuo across various
  downstream tasks. Our pipeline implementation is available at
  https://github.com/deep-diver/llamaduo.
paper_ai_keywords: LLMOps LlamaDuo fine-tuning synthetic dataset locally manageable models service LLM downstream tasks