StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction
  Following
Multi-turn instruction following capability constitutes a core competency of
large language models (LLMs) in real-world applications. Existing evaluation
benchmarks predominantly focus on fine-grained constraint satisfaction and
domain-specific capability assessment, yet overlook the crucial structural
dependency between dialogue turns that distinguishes multi-turn from
single-turn interactions. This structural dependency not only reflects user
intent but also establishes a second dimension for instruction following
evaluation beyond constraint satisfaction. To address this gap, we propose
StructFlowBench, a multi-turn instruction following benchmark with structural
flow modeling. The benchmark innovatively defines a structural flow framework
comprising six fundamental inter-turn relationships, which not only introduces
novel structural constraints for model evaluation but also serves as generation
parameters for creating customized dialogue flows tailored to specific
scenarios. Adopting established LLM-based automatic evaluation methodologies,
we conduct systematic evaluations of 13 leading open-source and closed-source
LLMs. Experimental results reveal significant deficiencies in current models'
comprehension of multi-turn dialogue structures. The code is available at
https://github.com/MLGroupJLU/StructFlowBench.
67b9dda03593f69f41cdb5d3
Jinnan Li
admin_assigned
2025-02-24T13:24:42.884Z
6434f5a5a4c9c55871ae888f
/avatars/058389c773a67b2b03d44556f0ee43d1.svg
Jinnan Li
user
Jinnan
67b9dda03593f69f41cdb5d4
Jinzhe Li
admin_assigned
2025-02-24T13:24:50.056Z
67658bd7f7ac7e978ab6f957
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/c8VBgFckkZNUGeqUyotwq.png
Jinzhe Li
user
JinzheFudan
67b9dda03593f69f41cdb5d5
Yue Wang
67b9dda03593f69f41cdb5d6
Yi Chang
67b9dda03593f69f41cdb5d7
Yuan Wu
claimed_verified
2025-02-24T15:47:33.508Z
670e57b3391f1a7021182bff
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
Yuan Wu
user
WhiteCatY
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction
  Following
Multi-turn instruction following capability constitutes a core competency of
large language models (LLMs) in real-world applications. Existing evaluation
benchmarks predominantly focus on fine-grained constraint satisfaction and
domain-specific capability assessment, yet overlook the crucial structural
dependency between dialogue turns that distinguishes multi-turn from
single-turn interactions. This structural dependency not only reflects user
intent but also establishes a second dimension for instruction following
evaluation beyond constraint satisfaction. To address this gap, we propose
StructFlowBench, a multi-turn instruction following benchmark with structural
flow modeling. The benchmark innovatively defines a structural flow framework
comprising six fundamental inter-turn relationships, which not only introduces
novel structural constraints for model evaluation but also serves as generation
parameters for creating customized dialogue flows tailored to specific
scenarios. Adopting established LLM-based automatic evaluation methodologies,
we conduct systematic evaluations of 13 leading open-source and closed-source
LLMs. Experimental results reveal significant deficiencies in current models'
comprehension of multi-turn dialogue structures. The code is available at
https://github.com/MLGroupJLU/StructFlowBench.
StructFlowBench addresses the lack of structural dependency evaluation in multi-turn instruction following by introducing a benchmark with six fundamental inter-turn relationships.
multi-turn instruction following dialogue turns structural dependency user intent constraint satisfaction StructFlowBench structural flow framework LLM-based automatic evaluation multi-turn dialogue structures