title: DiaTool-DPO: Multi-Turn Direct Preference Optimization for
    Tool-Augmented Large Language Models
summary: Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in
  real-world applications, but face challenges in handling incomplete queries and
  out-of-scope requests. While existing approaches rely mainly on Supervised
  Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method
  that enhances TA-LLM's dialogue capabilities through Direct Preference
  Optimization. We model TA-LLM interactions as a Markov Decision Process with 5
  distinct dialogue states and categorize user queries into 3 types based on
  their state transition trajectories. We automatically construct paired
  trajectory datasets of correct and incorrect dialogue flows and introduce a
  specialized objective loss for dialogue control. Our comprehensive evaluation
  demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in
  information gathering, 91% in tool call rejection) with substantial
  improvements over baseline (44% and 9.6% respectively) while maintaining core
  functionality. Our approach opens new possibilities for developing TA-LLMs that
  can handle diverse real-world scenarios without requiring additional expert
  demonstrations or human labeling.
paper_authors._id: 67f4d15a8f00d281c155880f
paper_authors.name: Sunghee Jung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T10:41:40.756Z
paper_authors._id: 65e30342e8b017ee1384824c
paper_authors.avatarUrl: /avatars/e5d07b037f611ccfaf719959d971d102.svg
paper_authors.fullname: Sunghee Jung
paper_authors.type: user
paper_authors.user: hash2430
paper_authors._id: 67f4d15a8f00d281c1558810
paper_authors.name: Donghun Lee
paper_authors._id: 67f4d15a8f00d281c1558811
paper_authors.name: Shinbok Lee
paper_authors._id: 67f4d15a8f00d281c1558812
paper_authors.name: Gaeun Seo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:42:12.004Z
paper_authors._id: 63148a8f5f47a18962765802
paper_authors.avatarUrl: /avatars/bc58a863727794006dddf758efa09411.svg
paper_authors.fullname: gaeunseo
paper_authors.type: user
paper_authors.user: gaeunseo
paper_authors._id: 67f4d15a8f00d281c1558813
paper_authors.name: Daniel Lee
paper_authors._id: 67f4d15a8f00d281c1558814
paper_authors.name: Byeongil Ko
paper_authors._id: 67f4d15a8f00d281c1558815
paper_authors.name: Junrae Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:42:51.801Z
paper_authors._id: 6464d7e4f4f3102bb6c79268
paper_authors.avatarUrl: /avatars/177e0c0f49728f8554173f63cc52a859.svg
paper_authors.fullname: cho
paper_authors.type: user
paper_authors.user: junrae
paper_authors._id: 67f4d15a8f00d281c1558816
paper_authors.name: Kihyun Kim
paper_authors._id: 67f4d15a8f00d281c1558817
paper_authors.name: Eunggyun Kim
paper_authors._id: 67f4d15a8f00d281c1558818
paper_authors.name: Myeongcheol Shin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:43:26.136Z
paper_authors._id: 60c89e7fd9d94022e7eac9c4
paper_authors.avatarUrl: /avatars/0a5dedfffcfb9c9ca7034c3e734c120d.svg
paper_authors.fullname: Myeongcheol Shin
paper_authors.type: user
paper_authors.user: dsindex
paper_title: DiaTool-DPO: Multi-Turn Direct Preference Optimization for
    Tool-Augmented Large Language Models
paper_summary: Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in
  real-world applications, but face challenges in handling incomplete queries and
  out-of-scope requests. While existing approaches rely mainly on Supervised
  Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method
  that enhances TA-LLM's dialogue capabilities through Direct Preference
  Optimization. We model TA-LLM interactions as a Markov Decision Process with 5
  distinct dialogue states and categorize user queries into 3 types based on
  their state transition trajectories. We automatically construct paired
  trajectory datasets of correct and incorrect dialogue flows and introduce a
  specialized objective loss for dialogue control. Our comprehensive evaluation
  demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in
  information gathering, 91% in tool call rejection) with substantial
  improvements over baseline (44% and 9.6% respectively) while maintaining core
  functionality. Our approach opens new possibilities for developing TA-LLMs that
  can handle diverse real-world scenarios without requiring additional expert
  demonstrations or human labeling.
paper_ai_summary: DiaTool-DPO improves TA-LLM dialogue capabilities through Direct Preference Optimization, enhancing performance in information gathering and tool call rejection compared to baselines.
paper_ai_keywords: Supervised Fine-Tuning Direct Preference Optimization Markov Decision Process dialogue states user queries dialogue control information gathering tool call rejection TA-LLMs