title: Distilling LLM Agent into Small Models with Retrieval and Code Tools
summary: Large language models (LLMs) excel at complex reasoning tasks but remain
  computationally expensive, limiting their practical deployment. To address
  this, recent works have focused on distilling reasoning capabilities into
  smaller language models (sLMs) using chain-of-thought (CoT) traces from teacher
  LLMs. However, this approach struggles in scenarios requiring rare factual
  knowledge or precise computation, where sLMs often hallucinate due to limited
  capability. In this work, we propose Agent Distillation, a framework for
  transferring not only reasoning capability but full task-solving behavior from
  LLM-based agents into sLMs with retrieval and code tools. We improve agent
  distillation along two complementary axes: (1) we introduce a prompting method
  called first-thought prefix to enhance the quality of teacher-generated
  trajectories; and (2) we propose a self-consistent action generation for
  improving test-time robustness of small agents. We evaluate our method on eight
  reasoning tasks across factual and mathematical domains, covering both
  in-domain and out-of-domain generalization. Our results show that sLMs as small
  as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next-tier
  larger 1.5B, 3B, 7B models fine-tuned using CoT distillation, demonstrating the
  potential of agent distillation for building practical, tool-using small
  agents. Our code is available at https://github.com/Nardien/agent-distillation.
paper_authors._id: 6833c9fd298a7bec9c3da3b0
paper_authors.name: Minki Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-26T14:42:07.255Z
paper_authors._id: 64b74920fe6a108d03fed767
paper_authors.avatarUrl: /avatars/a2c05b809c36fa5fab8e1a43b3e67051.svg
paper_authors.fullname: Minki Kang
paper_authors.type: user
paper_authors.user: Nardien
paper_authors._id: 6833c9fd298a7bec9c3da3b1
paper_authors.name: Jongwon Jeong
paper_authors._id: 6833c9fd298a7bec9c3da3b2
paper_authors.name: Seanie Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T14:07:42.794Z
paper_authors._id: 64ad5f59b7e4b2c1ce47eb43
paper_authors.avatarUrl: /avatars/1f13ebe21a90d8c99920aa2c8cd9ac45.svg
paper_authors.fullname: Seanie Lee
paper_authors.type: user
paper_authors.user: Seanie-lee
paper_authors._id: 6833c9fd298a7bec9c3da3b3
paper_authors.name: Jaewoong Cho
paper_authors._id: 6833c9fd298a7bec9c3da3b4
paper_authors.name: Sung Ju Hwang
paper_title: Distilling LLM Agent into Small Models with Retrieval and Code Tools
paper_summary: Large language models (LLMs) excel at complex reasoning tasks but remain
  computationally expensive, limiting their practical deployment. To address
  this, recent works have focused on distilling reasoning capabilities into
  smaller language models (sLMs) using chain-of-thought (CoT) traces from teacher
  LLMs. However, this approach struggles in scenarios requiring rare factual
  knowledge or precise computation, where sLMs often hallucinate due to limited
  capability. In this work, we propose Agent Distillation, a framework for
  transferring not only reasoning capability but full task-solving behavior from
  LLM-based agents into sLMs with retrieval and code tools. We improve agent
  distillation along two complementary axes: (1) we introduce a prompting method
  called first-thought prefix to enhance the quality of teacher-generated
  trajectories; and (2) we propose a self-consistent action generation for
  improving test-time robustness of small agents. We evaluate our method on eight
  reasoning tasks across factual and mathematical domains, covering both
  in-domain and out-of-domain generalization. Our results show that sLMs as small
  as 0.5B, 1.5B, 3B parameters can achieve performance competitive with next-tier
  larger 1.5B, 3B, 7B models fine-tuned using CoT distillation, demonstrating the
  potential of agent distillation for building practical, tool-using small
  agents. Our code is available at https://github.com/Nardien/agent-distillation.
paper_ai_summary: Agent Distillation transfers reasoning and task-solving capabilities from large language models to smaller models using enhanced prompts and self-consistent actions, matching performance of larger models on various reasoning tasks.
paper_ai_keywords: Large language models small language models chain-of-thought agent distillation prompting method first-thought prefix self-consistent action generation task-solving behavior retrieval tools code tools in-domain generalization out-of-domain generalization