Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs
  More Realistic and Less Risky
Large language models (LLMs) are increasingly tasked with invoking enterprise
APIs, yet they routinely falter when near-duplicate tools vie for the same user
intent or when required arguments are left underspecified. We introduce
DiaFORGE (Dialogue Framework for Organic Response Generation & Evaluation), a
disambiguation-centric, three-stage pipeline that (i) synthesizes
persona-driven, multi-turn dialogues in which the assistant must distinguish
among highly similar tools, (ii) performs supervised fine-tuning of open-source
models with reasoning traces across 3B - 70B parameters, and (iii) evaluates
real-world readiness via a dynamic suite that redeploys each model in a live
agentic loop and reports end-to-end goal completion alongside conventional
static metrics. On our dynamic benchmark DiaBENCH, models trained with DiaFORGE
raise tool-invocation success by 27 pp over GPT-4o and by 49 pp over
Claude-3.5-Sonnet, both under optimized prompting. To spur further research, we
release an open corpus of 5000 production-grade enterprise API specifications
paired with rigorously validated, disambiguation-focused dialogues, offering a
practical blueprint for building reliable, enterprise-ready tool-calling
agents.
686c67e2364e2ad167eb5314
Ashutosh Hathidara
claimed_verified
2025-07-08T08:06:34.896Z
637859f98f288aba3d01f588
https://cdn-avatars.huggingface.co/v1/production/uploads/637859f98f288aba3d01f588/eP8YNMOtxTvxH-rYEYn4f.png
Ashutosh Hathidara
user
ashutosh1919
686c67e2364e2ad167eb5315
Julien Yu
686c67e2364e2ad167eb5316
Sebastian Schreiber
Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs
  More Realistic and Less Risky
Large language models (LLMs) are increasingly tasked with invoking enterprise
APIs, yet they routinely falter when near-duplicate tools vie for the same user
intent or when required arguments are left underspecified. We introduce
DiaFORGE (Dialogue Framework for Organic Response Generation & Evaluation), a
disambiguation-centric, three-stage pipeline that (i) synthesizes
persona-driven, multi-turn dialogues in which the assistant must distinguish
among highly similar tools, (ii) performs supervised fine-tuning of open-source
models with reasoning traces across 3B - 70B parameters, and (iii) evaluates
real-world readiness via a dynamic suite that redeploys each model in a live
agentic loop and reports end-to-end goal completion alongside conventional
static metrics. On our dynamic benchmark DiaBENCH, models trained with DiaFORGE
raise tool-invocation success by 27 pp over GPT-4o and by 49 pp over
Claude-3.5-Sonnet, both under optimized prompting. To spur further research, we
release an open corpus of 5000 production-grade enterprise API specifications
paired with rigorously validated, disambiguation-focused dialogues, offering a
practical blueprint for building reliable, enterprise-ready tool-calling
agents.
DiaFORGE, a three-stage pipeline, enhances tool invocation success in LLMs by synthesizing dialogues, fine-tuning models, and evaluating them in a live environment.
LLMs enterprise APIs DiaFORGE persona-driven multi-turn dialogues supervised fine-tuning reasoning traces dynamic benchmark DiaBENCH tool-invocation success GPT-4o Claude-3.5-Sonnet production-grade enterprise API specifications disambiguation-focused dialogues