title: Hephaestus: Improving Fundamental Agent Capabilities of Large Language
    Models through Continual Pre-Training
summary: Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous
  agents typically rely on complex prompting or extensive fine-tuning, which
  often fails to introduce new capabilities while preserving strong
  generalizability. We introduce Hephaestus-Forge, the first large-scale
  pre-training corpus designed to enhance the fundamental capabilities of LLM
  agents in API function calling, intrinsic reasoning and planning, and adapting
  to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data
  encompassing 76,537 APIs, including both tool documentation to introduce
  knowledge of API functions and function calling trajectories to strengthen
  intrinsic reasoning. To explore effective training protocols, we investigate
  scaling laws to identify the optimal recipe in data mixing ratios. By continual
  pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale
  open-source LLMs and rivals commercial LLMs on three agent benchmarks,
  demonstrating the effectiveness of our pre-training corpus in enhancing
  fundamental agentic capabilities and generalization of LLMs to new tasks or
  environments.
paper_authors._id: 67ac1d45e6f1e95ccf6de3b7
paper_authors.name: Yuchen Zhuang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-12T04:02:14.866Z
paper_authors._id: 6471bddd609ae9f56368f132
paper_authors.avatarUrl: /avatars/71a80127a01e662ab2790de0511326b6.svg
paper_authors.fullname: Yuchen Zhuang
paper_authors.type: user
paper_authors.user: yczhuang
paper_authors._id: 67ac1d45e6f1e95ccf6de3b8
paper_authors.name: Jingfeng Yang
paper_authors._id: 67ac1d45e6f1e95ccf6de3b9
paper_authors.name: Haoming Jiang
paper_authors._id: 67ac1d45e6f1e95ccf6de3ba
paper_authors.name: Xin Liu
paper_authors._id: 67ac1d45e6f1e95ccf6de3bb
paper_authors.name: Kewei Cheng
paper_authors._id: 67ac1d45e6f1e95ccf6de3bc
paper_authors.name: Sanket Lokegaonkar
paper_authors._id: 67ac1d45e6f1e95ccf6de3bd
paper_authors.name: Yifan Gao
paper_authors._id: 67ac1d45e6f1e95ccf6de3be
paper_authors.name: Qing Ping
paper_authors._id: 67ac1d45e6f1e95ccf6de3bf
paper_authors.name: Tianyi Liu
paper_authors._id: 67ac1d45e6f1e95ccf6de3c0
paper_authors.name: Binxuan Huang
paper_authors._id: 67ac1d45e6f1e95ccf6de3c1
paper_authors.name: Zheng Li
paper_authors._id: 67ac1d45e6f1e95ccf6de3c2
paper_authors.name: Zhengyang Wang
paper_authors._id: 67ac1d45e6f1e95ccf6de3c3
paper_authors.name: Pei Chen
paper_authors._id: 67ac1d45e6f1e95ccf6de3c4
paper_authors.name: Ruijie Wang
paper_authors._id: 67ac1d45e6f1e95ccf6de3c5
paper_authors.name: Rongzhi Zhang
paper_authors._id: 67ac1d45e6f1e95ccf6de3c6
paper_authors.name: Nasser Zalmout
paper_authors._id: 67ac1d45e6f1e95ccf6de3c7
paper_authors.name: Priyanka Nigam
paper_authors._id: 67ac1d45e6f1e95ccf6de3c8
paper_authors.name: Bing Yin
paper_authors._id: 67ac1d45e6f1e95ccf6de3c9
paper_authors.name: Chao Zhang
paper_title: Hephaestus: Improving Fundamental Agent Capabilities of Large Language
    Models through Continual Pre-Training
paper_summary: Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous
  agents typically rely on complex prompting or extensive fine-tuning, which
  often fails to introduce new capabilities while preserving strong
  generalizability. We introduce Hephaestus-Forge, the first large-scale
  pre-training corpus designed to enhance the fundamental capabilities of LLM
  agents in API function calling, intrinsic reasoning and planning, and adapting
  to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data
  encompassing 76,537 APIs, including both tool documentation to introduce
  knowledge of API functions and function calling trajectories to strengthen
  intrinsic reasoning. To explore effective training protocols, we investigate
  scaling laws to identify the optimal recipe in data mixing ratios. By continual
  pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale
  open-source LLMs and rivals commercial LLMs on three agent benchmarks,
  demonstrating the effectiveness of our pre-training corpus in enhancing
  fundamental agentic capabilities and generalization of LLMs to new tasks or
  environments.
paper_ai_keywords: agent-oriented pre-training data LLM-based autonomous agents prompting fine-tuning Hephaestus-Forge large-scale pre-training corpus API function calling intrinsic reasoning planning environmental feedback tool documentation function calling trajectories scaling laws data mixing ratios continual pre-training open-source LLMs agent benchmarks fundamental agentic capabilities generalization of LLMs