title: SDPO: Segment-Level Direct Preference Optimization for Social Agents
summary: Social agents powered by large language models (LLMs) can simulate human
  social behaviors but fall short in handling complex goal-oriented social
  dialogues. Direct Preference Optimization (DPO) has proven effective in
  aligning LLM behavior with human preferences across a variety of agent tasks.
  Existing DPO-based approaches for multi-turn interactions are divided into
  turn-level and session-level methods. The turn-level method is overly
  fine-grained, focusing exclusively on individual turns, while session-level
  methods are too coarse-grained, often introducing training noise. To address
  these limitations, we propose Segment-Level Direct Preference Optimization
  (SDPO), which focuses on specific key segments within interactions to optimize
  multi-turn agent behavior while minimizing training noise. Evaluations on the
  SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform
  both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring
  SDPO's potential to advance the social intelligence of LLM-based agents. We
  release our code and data at
  https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.
paper_authors._id: 677b977e08c75e8046c38449
paper_authors.name: Aobo Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-06T12:37:42.697Z
paper_authors._id: 649974df6a49e761b2944145
paper_authors.avatarUrl: /avatars/5f149f87df89786807f3185a1e538362.svg
paper_authors.fullname: Aobo Kong
paper_authors.type: user
paper_authors.user: KAB1314
paper_authors._id: 677b977e08c75e8046c3844a
paper_authors.name: Wentao Ma
paper_authors._id: 677b977e08c75e8046c3844b
paper_authors.name: Shiwan Zhao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-06T12:16:57.707Z
paper_authors._id: 62f4b84f2f63f904a0c5b355
paper_authors.avatarUrl: /avatars/aa70c9709c736745246cadb7286c1e62.svg
paper_authors.fullname: Shiwan Zhao
paper_authors.type: user
paper_authors.user: shiwan
paper_authors._id: 677b977e08c75e8046c3844c
paper_authors.name: Yongbin Li
paper_authors._id: 677b977e08c75e8046c3844d
paper_authors.name: Yuchuan Wu
paper_authors._id: 677b977e08c75e8046c3844e
paper_authors.name: Ke Wang
paper_authors._id: 677b977e08c75e8046c3844f
paper_authors.name: Xiaoqian Liu
paper_authors._id: 677b977e08c75e8046c38450
paper_authors.name: Qicheng Li
paper_authors._id: 677b977e08c75e8046c38451
paper_authors.name: Yong Qin
paper_authors._id: 677b977e08c75e8046c38452
paper_authors.name: Fei Huang
paper_title: SDPO: Segment-Level Direct Preference Optimization for Social Agents
paper_summary: Social agents powered by large language models (LLMs) can simulate human
  social behaviors but fall short in handling complex goal-oriented social
  dialogues. Direct Preference Optimization (DPO) has proven effective in
  aligning LLM behavior with human preferences across a variety of agent tasks.
  Existing DPO-based approaches for multi-turn interactions are divided into
  turn-level and session-level methods. The turn-level method is overly
  fine-grained, focusing exclusively on individual turns, while session-level
  methods are too coarse-grained, often introducing training noise. To address
  these limitations, we propose Segment-Level Direct Preference Optimization
  (SDPO), which focuses on specific key segments within interactions to optimize
  multi-turn agent behavior while minimizing training noise. Evaluations on the
  SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform
  both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring
  SDPO's potential to advance the social intelligence of LLM-based agents. We
  release our code and data at
  https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.
paper_ai_summary: Segment-Level Direct Preference Optimization (SDPO) enhances complex goal-oriented social dialogues in LLM-powered social agents by focusing on key segments, outperforming existing DPO methods.
paper_ai_keywords: large language models (LLMs) direct preference optimization (DPO) turn-level session-level segment-level SOTOPIA benchmark GPT-4o