Hephaestus: Improving Fundamental Agent Capabilities of Large Language
  Models through Continual Pre-Training
Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous
agents typically rely on complex prompting or extensive fine-tuning, which
often fails to introduce new capabilities while preserving strong
generalizability. We introduce Hephaestus-Forge, the first large-scale
pre-training corpus designed to enhance the fundamental capabilities of LLM
agents in API function calling, intrinsic reasoning and planning, and adapting
to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data
encompassing 76,537 APIs, including both tool documentation to introduce
knowledge of API functions and function calling trajectories to strengthen
intrinsic reasoning. To explore effective training protocols, we investigate
scaling laws to identify the optimal recipe in data mixing ratios. By continual
pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale
open-source LLMs and rivals commercial LLMs on three agent benchmarks,
demonstrating the effectiveness of our pre-training corpus in enhancing
fundamental agentic capabilities and generalization of LLMs to new tasks or
environments.
67ac1d45e6f1e95ccf6de3b7
Yuchen Zhuang
extracted_pending
2025-02-12T04:02:14.866Z
6471bddd609ae9f56368f132
https://cdn-avatars.huggingface.co/v1/production/uploads/6471bddd609ae9f56368f132/G91Q4iCGN2dy3oMaz-LrO.jpeg
Yuchen Zhuang
user
yczhuang
67ac1d45e6f1e95ccf6de3b8
Jingfeng Yang
67ac1d45e6f1e95ccf6de3b9
Haoming Jiang
67ac1d45e6f1e95ccf6de3ba
Xin Liu
67ac1d45e6f1e95ccf6de3bb
Kewei Cheng
67ac1d45e6f1e95ccf6de3bc
Sanket Lokegaonkar
67ac1d45e6f1e95ccf6de3bd
Yifan Gao
67ac1d45e6f1e95ccf6de3be
Qing Ping
67ac1d45e6f1e95ccf6de3bf
Tianyi Liu
67ac1d45e6f1e95ccf6de3c0
Binxuan Huang
67ac1d45e6f1e95ccf6de3c1
Zheng Li
67ac1d45e6f1e95ccf6de3c2
Zhengyang Wang
67ac1d45e6f1e95ccf6de3c3
Pei Chen
67ac1d45e6f1e95ccf6de3c4
Ruijie Wang
67ac1d45e6f1e95ccf6de3c5
Rongzhi Zhang
claimed_verified
2025-05-16T07:13:19.324Z
644a2c4e9a1c5faef7a5dbd8
/avatars/fbbbc1347f8e423b2477e2506fdb43d9.svg
Rongzhi Zhang
user
Solute
67ac1d45e6f1e95ccf6de3c6
Nasser Zalmout
67ac1d45e6f1e95ccf6de3c7
Priyanka Nigam
67ac1d45e6f1e95ccf6de3c8
Bing Yin
67ac1d45e6f1e95ccf6de3c9
Chao Zhang
Hephaestus: Improving Fundamental Agent Capabilities of Large Language
  Models through Continual Pre-Training
Due to the scarcity of agent-oriented pre-training data, LLM-based autonomous
agents typically rely on complex prompting or extensive fine-tuning, which
often fails to introduce new capabilities while preserving strong
generalizability. We introduce Hephaestus-Forge, the first large-scale
pre-training corpus designed to enhance the fundamental capabilities of LLM
agents in API function calling, intrinsic reasoning and planning, and adapting
to environmental feedback. Hephaestus-Forge comprises 103B agent-specific data
encompassing 76,537 APIs, including both tool documentation to introduce
knowledge of API functions and function calling trajectories to strengthen
intrinsic reasoning. To explore effective training protocols, we investigate
scaling laws to identify the optimal recipe in data mixing ratios. By continual
pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale
open-source LLMs and rivals commercial LLMs on three agent benchmarks,
demonstrating the effectiveness of our pre-training corpus in enhancing
fundamental agentic capabilities and generalization of LLMs to new tasks or
environments.
Hephaestus-Forge, a large-scale pre-training corpus for LLM agents, enhances API function calling, reasoning, and environmental adaptation, outperforming existing models in agentic capabilities and generalization.
LLM-based autonomous agents Hephaestus-Forge API function calling intrinsic reasoning environmental feedback agent benchmarks continual pre-training