title: StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction
    Following
summary: Multi-turn instruction following capability constitutes a core competency of
  large language models (LLMs) in real-world applications. Existing evaluation
  benchmarks predominantly focus on fine-grained constraint satisfaction and
  domain-specific capability assessment, yet overlook the crucial structural
  dependency between dialogue turns that distinguishes multi-turn from
  single-turn interactions. This structural dependency not only reflects user
  intent but also establishes a second dimension for instruction following
  evaluation beyond constraint satisfaction. To address this gap, we propose
  StructFlowBench, a multi-turn instruction following benchmark with structural
  flow modeling. The benchmark innovatively defines a structural flow framework
  comprising six fundamental inter-turn relationships, which not only introduces
  novel structural constraints for model evaluation but also serves as generation
  parameters for creating customized dialogue flows tailored to specific
  scenarios. Adopting established LLM-based automatic evaluation methodologies,
  we conduct systematic evaluations of 13 leading open-source and closed-source
  LLMs. Experimental results reveal significant deficiencies in current models'
  comprehension of multi-turn dialogue structures. The code is available at
  https://github.com/MLGroupJLU/StructFlowBench.
paper_authors._id: 67b9dda03593f69f41cdb5d3
paper_authors.name: Jinnan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:24:42.884Z
paper_authors._id: 6434f5a5a4c9c55871ae888f
paper_authors.avatarUrl: /avatars/058389c773a67b2b03d44556f0ee43d1.svg
paper_authors.fullname: Jinnan Li
paper_authors.type: user
paper_authors.user: Jinnan
paper_authors._id: 67b9dda03593f69f41cdb5d4
paper_authors.name: Jinzhe Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:24:50.056Z
paper_authors._id: 67658bd7f7ac7e978ab6f957
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/c8VBgFckkZNUGeqUyotwq.png
paper_authors.fullname: Jinzhe Li
paper_authors.type: user
paper_authors.user: JinzheFudan
paper_authors._id: 67b9dda03593f69f41cdb5d5
paper_authors.name: Yue Wang
paper_authors._id: 67b9dda03593f69f41cdb5d6
paper_authors.name: Yi Chang
paper_authors._id: 67b9dda03593f69f41cdb5d7
paper_authors.name: Yuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T15:47:33.508Z
paper_authors._id: 670e57b3391f1a7021182bff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
paper_authors.fullname: Yuan Wu
paper_authors.type: user
paper_authors.user: WhiteCatY
paper_title: StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction
    Following
paper_summary: Multi-turn instruction following capability constitutes a core competency of
  large language models (LLMs) in real-world applications. Existing evaluation
  benchmarks predominantly focus on fine-grained constraint satisfaction and
  domain-specific capability assessment, yet overlook the crucial structural
  dependency between dialogue turns that distinguishes multi-turn from
  single-turn interactions. This structural dependency not only reflects user
  intent but also establishes a second dimension for instruction following
  evaluation beyond constraint satisfaction. To address this gap, we propose
  StructFlowBench, a multi-turn instruction following benchmark with structural
  flow modeling. The benchmark innovatively defines a structural flow framework
  comprising six fundamental inter-turn relationships, which not only introduces
  novel structural constraints for model evaluation but also serves as generation
  parameters for creating customized dialogue flows tailored to specific
  scenarios. Adopting established LLM-based automatic evaluation methodologies,
  we conduct systematic evaluations of 13 leading open-source and closed-source
  LLMs. Experimental results reveal significant deficiencies in current models'
  comprehension of multi-turn dialogue structures. The code is available at
  https://github.com/MLGroupJLU/StructFlowBench.
paper_ai_summary: StructFlowBench addresses the lack of structural dependency evaluation in multi-turn instruction following by introducing a benchmark with six fundamental inter-turn relationships.
paper_ai_keywords: multi-turn instruction following dialogue turns structural dependency user intent constraint satisfaction StructFlowBench structural flow framework LLM-based automatic evaluation multi-turn dialogue structures