## CFBench: A Comprehensive Constraints-Following Benchmark for LLMs

Tao Zhang 1 * ♣ , Yanjun Shen 1 * , Wenjing Luo , Yan Zhang , Hao Liang , Tao Zhang 1 1 2 1 ♢ , Fan Yang , 1 Mingan Lin , Yujing Qiao , Weipeng Chen , Bin Cui , 1 1 1 2 Wentao Zhang 2 † , Zenan Zhou 1†

1 Baichuan Inc. 2 Peking University

{

♣ zhangtao2, shenyanjun, zhouzenan } @baichuan-inc.com, { wentao.zhang } @pku.edu.cn

## Abstract

The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a largescale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multidimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at https://github.com/PKU-BaichuanMLSystemLab/CFBench.

## Introduction

Large Language Models (LLMs) have become the cornerstone of numerous cutting-edge research tasks and are widely utilized in real-world scenarios (Brown et al. 2020; Chowdhery et al. 2023; Achiam et al. 2023; Touvron et al. 2023). In real-world scenarios, human instructions are inherently complex and accompanied by explicit constraints, requiring models to both understand intricate requirements and strictly comply with these constraints (Yang et al. 2023; Zhong et al. 2021; Mishra et al. 2022; Wei et al. 2021; Sanh et al. 2022). The proficiency of LLMs in comprehending requirements and adhering to natural language constraints is essential, as it ensures tasks are executed precisely and resolved perfectly according to user instructions.

* These authors contributed equally.

† Corresponding authors.

Figure 1: Sample data from CFBench. A checklist, constraint type, requirement priority, and satisfaction constitute our evaluation criteria.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-90a593a3-c024-4748-baf5-b4b048c912ad.jpg)

The prevailing method for evaluating a model's instruction-following ability involves using quantitative programs, human evaluators, or advanced LLMs to assess response quality across single constraints, complex problems, and finite constraints (Zhou et al. 2023a; Wang et al. 2023; Li et al. 2023; Zheng et al. 2024; Xu et al. 2023). Laskar et al. (2024) underscores the importance of evaluating data quality, highlighting the necessity for real and extensive data distribution, along with its applicability to real-world scenarios. Sun et al. (2024b) also stresses that realistic evaluation metrics reflect model capabilities and guide iteration. Constraints-following evaluation faces analogous challenges, particularly within complex real-world scenarios, where data sources and contexts are diverse, and where evaluation is both subjective and arduous. Fig. 1, which addresses the aforementioned challenges, presents a sample from CFBench illustrating the Trump assassination event with different colors representing various constraint types. The instruction include multiple constraints, and the evaluation method uses a checklist to break down complex requirements into independent checkpoints, annotating constraint types and priorities. LLMs are then used to assess each checkpoint. Consequently, we introduce two more profound challenges in constraints-following assessment.

## Q1: How to construct high-quality evaluation data?

Many studies focus on evaluating single constraint (Chen et al. 2022; Tang et al. 2023), lacking comprehensive analysis across diverse constraints. He et al. (2024b) examines LLM performance on complex real-world instructions but neglect constraint diversity and scenario coverage. Jiang et al. (2023) incrementally incorporate fine-grained constraints to craft multi-level instructions. However, with only 75 instances of mixed type, which risks variability due to limited data, and equating difficulty with constraint quantity oversimplifies the task. Recent work focuses on evaluating constraints combinability (Wen et al. 2024). To ensure data quality, we systematically categorize constraints by mining real-world online data and using classification, synthesis, and expert design, covering 10 primary categories and over 25 subcategories. We also cross-match these constraints with various domains and scenarios, ensuring balanced representation and expert-validated reasonableness.

## Q2:How to evaluate accurately and meticulously?

Evaluating LLMs' adherence to constraints is challenging and typically involves manual, automated, and programmatic assessments using various metrics. Representative work computes outcomes for verifiable instructions using code (Zhou et al. 2023a; He et al. 2024b). Jiang et al. (2023) uses scripts and constraint-evolution paths to handle diverse challenging instructions, introducing three metrics tailored to the data's characteristics. The DRFR method decomposes complex constraints into binary judgments, with GPT evaluating each criterion (Qin et al. 2024). Indeed, previous work has ensured the feasibility and objectivity of evaluations through various methods, but they have overlooked assessments from the user's multiple perspectives. We deconstruct complex instructions from the user's perspective into multiple sub-needs, categorizing them by priority and constraint type, with LLMs evaluating each checkpoint. Furthermore, a multi-dimensional evaluation criteria is proposed using three metrics from the perspectives of constraints, instructions, and requirements priority.

We introduce CFBench, a comprehensive benchmark designed to thoroughly evaluate the constraint comprehension and following capabilities of LLMs. CFBench aggregates constraints from real-world scenarios and pioneers a hierarchical system for constraint types, encompassing 10 primary categories and over 25 secondary subcategories organized through taxonomic and statistical methodologies. CFBench features 1,000 meticulously curated samples spanning more than 200 real-life scenarios across 20 domains and over 50 NLP tasks, enhancing the breadth and generality of the evaluation data. Additionally, we have seamlessly integrated original instructions and constraint types within each sample, paying particular attention to nuanced combinations, ensuring each constraint is credibly and coherently embedded. Our advanced evaluation methodology incorporates multi-dimensional assessment criteria, which prioritizing requirements to align LLM outputs with user needs, enhance interpretability, and facilitate iterative development. Finally, extensive experiments and exploratory discussions provide strong support for evaluation and optimization.

Overall, our contributions are mainly four-fold:

- · To the best of our knowledge, we are the pioneers in systematically defining an instruction constraint framework utilizing both taxonomic and statistical methodologies.
- · We introduce CFBench, a meticulously annotated, largescale, high-quality benchmark that encompasses a broad spectrum of real-world scenarios and NLP tasks.
- · We propose a multi-dimensional evaluation framework to comprehensively assess model capabilities while prioritizing user-centric needs.
- · We exhaustively evaluated prominent LLMs, uncovering significant deficiencies in constraints following and exploring performance factors and optimization strategies.

## Related Work

Evaluation for LLMs Numerous studies have concurrently evaluated LLMs from various perspectives. By integrating a multitude of existing datasets, many have assessed the overall capabilities of LLMs (Bommasani, Liang, and Lee 2023; Zhong et al. 2024; Dubois et al. 2024; Chia et al. 2024; Hendrycks et al. 2021). Some research has delved into specialized capabilities, such as programming (Chen et al. 2021), reasoning (Cobbe et al. 2021), and knowledge (Huang et al. 2024). Unlike previous studies, we evaluate LLMs' instruction-following abilities with a focus on constraint-based instructions.

Instruction Following Fine-tuning LLMs with annotated instructional data enhances their ability to follow general language instructions (Weller et al. 2020; Sanh et al. 2022; Mishra et al. 2022). Studies show that more complex or constrained instructions further improve this ability. For instance, six methods to create intricate instructions from a small set of handwritten seed data are proposed (Xu et al. 2023; Luo et al. 2023), while Mukherjee et al. (2023) elevate training data complexity by having GPT-4 generate reasoning steps for simple instructions. The latest work (Sun et al. 2024a; He et al. 2024a; Dong et al. 2024) suggests that increasing the number and variety of constraints can enhance the complexity of instructions, thereby further improving the model's ability to follow constraint-based instructions.

Evaluation of Constraints Following Constraints such as word count, position, topics, and content have garnered significant attention in the field of Controlled Text Generation (Yao et al. 2023; Zhou et al. 2023b; Chen et al. 2022). Zhou et al. (2023a) centers on assessing 25 verifiable instructions. Numerous studies have explored the adherence of LLMs to format constraints, such as complex tabular data (Tang et al. 2023) and customized scenario formats (Xia et al. 2024). Qin et al. (2024) decomposing a single instruction into multiple constraints. He et al. (2024b) gathered constraints from

Figure 2: The construction pipeline of CFBench. Initially, it entails the construction of the constraint system, followed by the assembly of the dataset, and culminating in the proposal of a multi-perspective user view evaluation.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-e03082ee-9f78-4287-8606-ea68dd27245c.jpg)

real-world scenarios and developed a sophisticated benchmark using detailed task descriptions and inputs. Jiang et al. (2023) progressively integrates fine-grained constraints to develop multi-level instructions, thereby enhancing complexity across six distinct types. Concurrent work (Wen et al. 2024), constructs a novel benchmark by synthesizing and refining data from the aforementioned benchmarks, with an emphasis on the combinatorial types of constraints. However, previous studies suffered from fragmented constraints, limited scenarios, and misaligned evaluation methods with user perspectives.

## Constraints System

## CFBench

As depicted in Fig. 2, the CFBench construction pipeline includes several key components. First, we collect and systematize constraint expressions from real-world scenarios and various NLP tasks. Using this system, we create high-quality evaluation data by combining instructions from these scenarios with advanced LLMs and manual curation. We then introduce innovative multi-perspective evaluation method. Additionally, we conduct a thorough statistical analysis and validate the quality from various angles to highlight reliability and applicability.

## Constraints System

Constraints Collection We amass a diverse corpus of instructions from real-world scenarios and various NLP tasks (Xia et al. 2024; Li et al. 2024) to ensure a comprehensive system. Initially, we aggregate several million instructions from online logs and NLP tasks, refining these through length filtering and clustering to distill 30,000 high-quality instructions. Utilizing advanced LLM techniques, we extract and expand atomic constraints through evolutionary methods. Using LLMs, we carefully select meaningful atomic constraints, resulting in over 5000 unique constraints. Domain experts first filter out unreasonable or meaningless atomic constraints and then synthesize these into a structured framework with 10 primary categories and 25 subcategories, guided by principles of statistics, taxonomy, and linguistics.

Content constraints control the scope and depth of output content by specifying certain conditions (Zhang et al. 2023), and can be divided into lexical constraints, element constraints, and semantic constraints based on their granularity. Numerical constraints , which ensure that output content meets length and quantity requirements (Yao et al. 2023), can be classified into word-level, sentence-level, paragraphlevel, document-level based on the objects involved in the planning. Stylistic constraints impart a unique flavor and color to the output, revealing the author's traits and chosen social objectives (Tsai et al. 2021), can be subdivided into tonal, formal, audience, and authorial style constraints based on the perspective of application. Format constraints (Tang et al. 2023) standardize expression to guide the generation of complex content and can be categorized into fundamental, bespoke, and specialized scenario constraints based on their usage scenarios. Linguistic constraints (Zhou et al. 2023b) adapt to various scenarios by controlling internal features and logic, grouped into Pragmatic, Syntactic, Morphological, Phonological, and other constraints. Situation constraints (Liu et al. 2023) guide response appropriateness through background or situational parameters, can be classified into role-based, task-specific, and complex contextual constraints. Example constraints regulate new responses by leveraging the intrinsic patterns established by a limited set of samples, with an emphasis on assessing the model's proficiency in contextual constraint learning. Inverse constraints narrow the response space through the mechanism of indirect exclusion. Contradictory constraints denote conditions that are mutually exclusive, rendering it impossible for the response to fulfill all requirements concurrently, which are prevalent in online logs and are often easily overlooked. Rule constraints define logic flows or actions and meticulously crafted to standardize the road of responses. Details are in Appendix Tab. 8.

## Dataset Construction

To guarantee data quality in terms of authority and thorough coverage, we utilize a collaborative iterative methodology

Table 1: CFBench Statistic. The abbreviations of 'Num.', 'Len.', 'Prim.', 'Cons.', 'Type.' denote the sample number, average instruction length, average primary requirements number, average constraint number, average constraint type number, respectively. The designations 'C1'-'C10' denote the Primary Constraint types of content, numerical, style, format, linguistic, situation, example, inverse, contradictory, and rule constraint, respectively.

| Split Set   | Basic Info   | Basic Info   | Basic Info   | Basic Info   | Basic Info   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   | Constraints Count   |
|-------------|--------------|--------------|--------------|--------------|--------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| Split Set   | Num.         | Len.         | Prim.        | Cons.        | Type.        | C1                  | C2                  | C3                  | C4                  | C5                  | C6                  | C7                  | C8                  | C9                  | C10                 |
| Easy Set    | 500          | 413          | 1.69         | 3.59         | 2.83         | 613                 | 214                 | 180                 | 170                 | 134                 | 92                  | 82                  | 95                  | 90                  | 79                  |
| Hard Set    | 500          | 605          | 1.98         | 4.89         | 3.58         | 772                 | 345                 | 233                 | 241                 | 168                 | 122                 | 115                 | 145                 | 137                 | 81                  |
| Full Set    | 1000         | 509          | 1.84         | 4.24         | 3.20         | 1385                | 559                 | 413                 | 411                 | 302                 | 214                 | 197                 | 240                 | 227                 | 160                 |

Figure 3: The distribution of NLP tasks and domains

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-93e8662a-8e05-4316-87c7-39db3a47c5ea.jpg)

Task and Domain Distribution CFBench covers 20 major real-life domains and includes over 200 common scenarios and 50+ NLP tasks. Fig. 3(a) illustrates the classification of NLP tasks, including four major types: classification, generation, sequence labeling, and sentence relation, along with their corresponding specific tasks. Fig. 3(b) shows the real-life scenario-specific domain distribution, where Employment is the most prevalent category, and the other domains are relatively balanced. Our objective is to balance the real distribution with an equitable distribution. Overall, Fig. 3 illustrates that CFBench has evolved into a comprehensive and well-balanced benchmark.

that synergizes expertise with the capabilities of LLMs.

Data Source and Selection Real-world scenarios and NLPtasks form the foundation for CFBench's initial instructions. By harnessing advanced LLMs, we rigorously assess each instruction for constraint types and quantities within a predefined system, filtering out those with unreasonable or ineffective constraints. Subsequently, we balance the scenarios and constraint types, resulting in a refined set of 2,000 instructions covering all scenarios and NLP tasks. Prompts for instruction and checklist generation are in Appendix.

Iterative Refinement Professional annotators carefully review and refine the data, ensuring the rationality of constraints and gold answer. If modifications are needed, instructions are revised, and LLMs generate responses with refined evaluation criteria, repeating this process until satisfactory results are achieved. Ultimately, comprehensive support is formulated for each sample, detailing high-quality instructions, the ideal answer, specific assessment criteria, constraint types, and priority levels, totaling 1000 entries.

## Dataset Statistics

Overall Statistics Tab. 1 provides a statistical overview of CFBench, highlighting the substantial differences between the two sets. The Hard Set features more detailed instructions, a greater variety of constraints, and a higher number of constraints, indicating a higher level of complexity compared to the Easy Set. Furthermore, the table demonstrates the diversity and balanced distribution of primary constraint types within CFBench, an area where we significantly outperform other benchmarks. See the Appendix for division details.

Comparison with Other Benchmarks As shown in Tab. 3, we thoroughly compare our benchmark with various relevant ones. In terms of size, our benchmark contains approximately twice the number of samples as others. Notably, FollowBench (Jiang et al. 2023), despite having 820 samples, originally had only one-fifth of this number, with each sample expanded by adding five constraints. From the constraint perspective, CFBench is the only benchmark with systematized constraints, featuring the highest number of primary constraint types and ensuring a balanced consideration of these types. Regarding evaluation, InFoBench (Qin et al. 2024) also sets response criteria for each sample, while CFBench distinguishes itself by incorporating user demand prioritization and emphasizing objectivity.

## Evaluation Protocol

Evaluation Criteria We breaking down instructions into multiple simple, independent checkpoints to ensure evaluation accuracy, inspiration was drawn from DRFR (Qin et al. 2024). This approach addresses the challenge of evaluating entire responses, especially for complex instructions with multiple constraints. Unlike DRFR, our method emphasizes defining ideal response characteristics and critical evaluation points. The previous sections detailed the checklist generation process, a key part of our evaluation criteria. Furthermore, we employ the state-of-the-art LLM, GPT-4o, as the evaluation model. By repeatedly feeding it the instruction, test model response, and checklist with a carefully tuned prompt, we ensure that the judged response fully meets the judgement format check. This iterative process aims to maximize confidence in our evaluation. The specific evaluation prompt is in the Appendix.

Table 2: The evaluation results of LLMs on CFBench and its splits. Notably, ∗ stands for the model supporting mainstream languages excluding Chinese, and † represents calling through the API. The bold , underlined, and tilde denote the first, second, ✿✿✿✿ and third rankings, respectively.

| Models                 | Easy Set   | Easy Set   | Easy Set   | Hard Set   | Hard Set   | Hard Set   | Full Set   | Full Set   | Full Set   |
|------------------------|------------|------------|------------|------------|------------|------------|------------|------------|------------|
| Models                 | CSR        | ISR        | PSR        | CSR        | ISR        | PSR        | CSR        | ISR        | PSR        |
| GPT-4o †               | 0.956      | 0.868      | 0.888      | 0.816      | 0.438      | 0.582      | 0.886      | 0.653      | 0.735      |
| GPT-4-Turbo-20240409 † | 0.924      | 0.792      | 0.826      | 0.783      | 0.370      | 0.518      | 0.853      | 0.581      | 0.672      |
| GPT-4-0125-Preview †   | 0.923      | 0.790      | 0.826      | 0.763      | 0.310      | 0.468      | 0.843      | 0.550      | 0.647      |
| GPT-3.5-Turbo-1106 †   | 0.797      | 0.520      | 0.602      | 0.631      | 0.176      | 0.326      | 0.714      | 0.348      | 0.464      |
| Claude-3.5-Sonnet †    | 0.943      | 0.844      | 0.882      | 0.799      | 0.408      | 0.564      | 0.871      | 0.626      | 0.723      |
| GLM-4-0520 †           | 0.939      | 0.820      | 0.852      | 0.785      | ✿✿✿✿ 0.372 | ✿✿✿✿ 0.536 | 0.862      | ✿✿✿✿ 0.596 | 0.694      |
| DeepSeek-V2-0628 †     | 0.946      | 0.830      | 0.868      | 0.786      | 0.350      | 0.524      | 0.866      | 0.590      | 0.696      |
| ERNIE-4-Turbo-0628 †   | 0.930      | 0.790      | 0.848      | 0.772      | 0.332      | 0.532      | 0.851      | 0.561      | 0.690      |
| Yi-Large †             | 0.900      | 0.730      | 0.786      | 0.744      | 0.292      | 0.460      | 0.822      | 0.511      | 0.623      |
| abab6.5-chat †         | 0.894      | 0.696      | 0.766      | 0.736      | 0.260      | 0.452      | 0.815      | 0.478      | 0.609      |
| MoonShot-V1-8k †       | 0.919      | 0.764      | 0.812      | 0.758      | 0.308      | 0.464      | 0.838      | 0.536      | 0.638      |
| Llama-3-8B-Instruct ∗  | 0.656      | 0.300      | 0.356      | 0.562      | 0.122      | 0.238      | 0.609      | 0.211      | 0.297      |
| Llama-3-70B-Instruct ∗ | 0.750      | 0.422      | 0.498      | 0.642      | 0.178      | 0.330      | 0.696      | 0.300      | 0.414      |
| DeepSeek-V2-Lite-Chat  | 0.733      | 0.382      | 0.448      | 0.597      | 0.148      | 0.262      | 0.665      | 0.265      | 0.355      |
| YI-1.5-34B-Chat        | 0.881      | 0.672      | 0.740      | 0.745      | 0.302      | 0.474      | 0.813      | 0.487      | 0.607      |
| Qwen1.5-110B-Chat      | 0.905      | 0.724      | 0.792      | 0.730      | 0.276      | 0.438      | 0.818      | 0.500      | 0.615      |
| Qwen2-72B-Instruct     | ✿✿✿✿ 0.944 | ✿✿✿✿ 0.836 | ✿✿✿✿ 0.880 | ✿✿✿✿ 0.791 | 0.342      | 0.530      | ✿✿✿✿ 0.867 | 0.589      | ✿✿✿✿ 0.705 |

Evaluation Metrics Aligned with different perspectives, we define the Constraint Satisfaction Rate (CSR), Instruction Satisfaction Rate (ISR) as follows:

$$\text{CSR} = \frac { 1 } { m } \sum _ { i = 1 } ^ { m } ( \frac { 1 } { n _ { i } } \sum _ { j = 1 } ^ { n _ { i } } s _ { i } ^ { j } ) \text{ \quad \ \ } ( 1 )$$

$$I S R = \frac { 1 } { m } \sum _ { i = 1 } ^ { m } s _ { i } \quad \quad \ ( 2 )$$

| Benchmarks   | Data Quality   | Data Quality   | Data Quality   | Evaluation   | Evaluation   |
|--------------|----------------|----------------|----------------|--------------|--------------|
|              | Num.           | Type.          | Syst.          | Prio.        | Meth.        |
| IFEval       | 541            | 4 ∗            | %              | %            |              |
| CELLO        | 523            | 4              | %              | %            |              |
| FollowBench  | 820            | 5              | %              | %            |              |
| InFoBench    | 500            | 5              | %              | %            |              |
| CfBench      | 1000           | 10             | "              | "            |              |

where s j i = 1 if the j -th constraint of i -th instruction is satisfied and s j i = 0 otherwise. s i = 1 indicates that all constraints in the i -th instruction are satisfied and s i = 0 otherwise. The requirements Priority Satisfaction Rate (PSR) is defined as follows:

$$\text{www.s.} \\ \text{PSR} = \frac { 1 } { m } \sum _ { i = 1 } ^ { m } ( P S R _ { i } ) \quad \quad ( 3 ) \quad \quad \text{(3)}$$

Let the average score for secondary requirements be A . When all primary requirements are met, score = 0 5+0 5 . . × A . If score &gt; 0 8 . , then PSR i = 1 ; otherwise, PSR i = 0 , especially when any primary requirement is not met. The threshold of 0 8 . is based on user feedback, reflecting tolerance for LLMs adhering to constraints. Overall, CSR, ISR, and PSR reflect different levels of user perception from multiple perspectives, including constraints, instructions, and requirement priorities.

## Data Quality

Quality Evolution To enhance the quality of CFBench, we invested considerable effort and financial resources.

∗

Table 3: Detailed Comparison of Relevant Benchmarks. represents our constraint system. 'Num.', 'Type.', 'Syst.', 'Prio.', and 'Meth.' denote the number of samples, primary constraint types, presence of a constraint system, requirement prioritization, and evaluation method.

First, in the instruction generation phase, we utilized multiple advanced LLMs, such as GPT-4 and Claude, to generate diverse instructions and responses for annotator candidates. Second, we implemented a stringent manual annotation process, including annotator training, cross-validation, batch validation, expert team involvement, and iterative refinement of instruction constraints and response quality. We also ensured the objectivity, evaluability, and prioritization of checkpoints. Additionally, we balanced the data for constraint types, scenarios, and NLP task distribution. Detailed information can be found in the Appendix.

Quality Evaluation To thoroughly investigate the quality of CFBench, we randomly selected 100 samples for assess-

Figure 4: Different mainstream models' results under primary and secondary constraint categories.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-16657dc6-91fd-4604-9418-5d63cae2917c.jpg)

Figure 5: Different mainstream models' PSR results in realworld domains and NLP task types.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-b42a03b9-6c0b-42b3-9a39-ece96e0565b4.jpg)

ment. Three professional data inspectors conducted a meticulous evaluation, resulting in high-quality rates of 94 % for instructions, 94 % for responses, and 93 % for checklists, as shown in Appendix Tab. 5. Additionally, three experts provided averaged scores for the outputs of Qwen2-7B-Instruct on a 0-1 scale. The kappa coefficient between GPT-4o PSR and expert evaluations approached 0.77, underscoring the superiority of the PSR evaluation method and metrics, even when applied to smaller models. Further details are provided in the Appendix Tab. 6. In conclusion, CFBench is a highquality benchmark, both in terms of data and evaluation.

## Experiment

## Evaluation Settings

We evaluated a total of 50 models that demonstrated exceptional performance on previous benchmarks (Hendrycks et al. 2020; Cobbe et al. 2021), considering factors such as model size, support for Chinese among mainstream languages, whether they were accessed via API or weights, and whether they were fine-tuned with instruction tuning data. During inference, we set the maximum generation length to 2048, with other parameters using their default values. For evaluation, we used GPT-4o as the judge model, setting the temperature to 0 to ensure deterministic outputs. All model names mentioned correspond to the versions listed in Tab. 7 of the Appendix.

## Overall Result

Tab. 2 presents the detailed evaluation results of CFBench and its splits for the leading models, using three key met-

Table 4: Performance Comparison on Benchmarks

| Model              |   MMLU | GSM8K   |   CFBench |
|--------------------|--------|---------|-----------|
| GPT-4o             |   88.7 | 90.5    |     0.698 |
| Claude-3.5-Sonnet  |   88.7 | 96.4    |     0.691 |
| Qwen2-72B-Instruct |   82.3 | 91.1    |     0.672 |
| GLM-4              |   81.5 | 87.6    |     0.665 |
| DeepSeek-V2        |   78.5 | 79.2    |     0.665 |
| Qwen1.5-110B-Chat  |   80.4 | -       |     0.584 |
| Qwen-1.5-72B-Chat  |   77.5 | 79.5    |     0.577 |

rics. Overall, GPT-4o leads across all splits and metrics, with Claude-3.5-Sonnet close behind. Qwen2-72B-Instruct, GLM4, DeepSeek-V2, and ERNIE4 also performed well, while Yi-large, abab6.5, and MoonShot lagged slightly. Considering the different splits, the Hard split shows a significant drop in metrics, highlighting the clear distinction between Hard and Easy splits. The highest PSR only reached 0.582, indicating room for improvement. Regarding the metrics, CSR is more favorable for weaker models, making it easier to identify issues at a fine-grained level. ISR and PSR highlight differences among stronger models, with ISR being the most challenging and PSR considering user priority and overall satisfaction. Notably, API-accessed models significantly outperformed open-source models accessed via weights, except for Qwen2-72B-Instruct, highlighting a substantial gap in constraint follow for open-source models.

## Constraints-categorized Performance

To observe the performance across different constraint types, we calculated satisfaction scores for each constraint for the top 8 LLMs, as shown in Fig. 4. For primary constraints, many models struggled with contradictory constraints, revealing their limitations in handling conflicts. By considering requirement priorities, we captured nuances among multiple constraints, enabling detailed evaluation. GPT-4o excelled across various constraints, while other models alternated in leading different types. For secondary constraints, all models performed poorly in lexical, word, and sentence count constraints but did better in document count and audience style constraints. No single model consistently led across most constraint types. In summary, even the most advanced LLMs have significant room for improvement in certain constraint types, with each model showing specific weaknesses. This provides valuable insights for future model iterations.

## Domain and Task-categorized Performance

As depicted in Fig. 5, we evaluate performance across 21 domains and 4 major NLP task types, each with 500 examples from the two main sources of CFBench. For domain performance, employment and psychology require significant attention, while technology and recruitment are strengths for most models. For NLP tasks, GPT-4o excels in sentence relationship tasks, while Qwen2-72B-Instruct is strong in sequence labeling, likely due to its optimization for Chinese. In general, models exhibit different rankings across domains

Figure 6: Factors Influencing Constraints-Following Performance

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-956e8f78-69e7-4fa4-9d02-378644fe1d9d.jpg)

and tasks, indicating no clear absolute leader. Comprehensive improvements are needed for better constraint follow across multiple domains and tasks.

## How to improve constraint-following ability?

## Discussion

## Comparisons between Capabilities

Tab. 4 presents a comprehensive comparison of CFBench's PSR with two of the prominent LLM evaluation benchmarks: MMLU (Hendrycks et al. 2020) and GSM8K (Cobbe et al. 2021). MMLU focuses on knowledge proficiency, while GSM8K emphasizes mathematical ability. GPT-4o ranks first on CFBench but significantly lags behind, ranking third on GSM8K. Qwen1.5-110B-Chat performs worse than DeepSeek-V2 on CFBench but outperforms it on MMLU. Notably, the rankings of LLMs on CFBench do not entirely correspond with those on the other two benchmarks, indicating that CFBench provides a novel perspective for LLMs evaluation.

## Factors influencing constraints-following

Previously, we identified a significant gap in LLM constraints following performance, prompting us to further explore the influencing factors. We analyzed the impact of prompt length, number of constraints, constraint types, and primary requirements on evaluation results across five topperforming models and their average values. As shown in Fig. 6, all four factors are positively correlated with the ISR metric, with the number of constraints having the most significant effect. For PSR, the number of constraints and constraint types do not show a completely positive correlation, while the number of primary requirements has a greater influence. Users are more affected by unmet constraints when there are fewer, but become more tolerant of unmet nonprimary constraints when there are many.

In Appendix Tab. 7, we investigated methods to potentially enhance constraint following. Firstly, Supervised FineTuning (SFT) significantly improves performance, with nearly all models that undergo instruction fine-tuning exhibiting substantial improvements in effectiveness, as demonstrated by the Qwen series. Secondly, model size is also an important factor, as evidenced by Qwen272B-Instruct showing a 40% relative PSR improvement over Qwen2-7B-Instruct. Additionally, replicating Conifer's models (Sun et al. 2024a) reveals that fine-tuning with complex constraint instructions further enhances performance, and recent work has also been directed towards this approach (He et al. 2024a). Further exploration is intended to be pursued in future work.

## Conclusion

This study comprehensively examines the constraintsfollowing capabilities of LLMs. CFBench, a comprehensive benchmark, was introduced with 1000 manually annotated samples covering more than 200 real-world scenarios and over 50 NLP tasks, encompassing a wide range of systematically defined constraint types. Each sample in CFBench includes detailed evaluation criteria, providing metrics that accurately reflect model performance and real user needs across multiple dimensions. Extensive experiments on CFBench revealed significant limitations and challenges that advanced LLMs face in following constraint instructions. Key factors and potential strategies to improve constraint following were also analyzed. In conclusion, CFBench offers a novel perspective for evaluating LLM capabilities, providing new directions for performance assessment and improvement.

## References

Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 .

Bommasani, R.; Liang, P.; and Lee, T. 2023. Holistic evaluation of language models. Annals of the New York Academy of Sciences , 1525(1): 140-146.

Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems , 33: 18771901.

Chen, H.; Li, H.; Chen, D.; and Narasimhan, K. 2022. Controllable text generation with language constraints. arXiv preprint arXiv:2212.10466 .

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 .

Chia, Y. K.; Hong, P.; Bing, L.; and Poria, S. 2024. INSTRUCTEVAL:Towards Holistic Evaluation of InstructionTuned Large Language Models. In of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024) , 35.

Chowdhery, A.; Narang, S.; Devlin, J.; Bosma, M.; Mishra, G.; Roberts, A.; Barham, P.; Chung, H. W.; Sutton, C.; Gehrmann, S.; et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240): 1-113.

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 .

Dong, G.; Lu, K.; Li, C.; Xia, T.; Yu, B.; Zhou, C.; and Zhou, J. 2024. Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models. arXiv preprint arXiv:2406.13542 .

Dubois, Y.; Li, C. X.; Taori, R.; Zhang, T.; Gulrajani, I.; Ba, J.; Guestrin, C.; Liang, P. S.; and Hashimoto, T. B. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems , 36.

He, Q.; Zeng, J.; He, Q.; Liang, J.; and Xiao, Y . 2024a. From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models. arXiv preprint arXiv:2404.15846 .

He, Q.; Zeng, J.; Huang, W.; Chen, L.; Xiao, J.; He, Q.; Zhou, X.; Liang, J.; and Xiao, Y. 2024b. Can Large Language Models Understand Real-World Complex Instructions? In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, 18188-18196.

Hendrycks, D.; Burns, C.; Basart, S.; Critch, A.; Li, J.; Song, D.; and Steinhardt, J. 2021. Aligning AI With Shared Human Values. Proceedings of the International Conference on Learning Representations (ICLR) .

Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 .

Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Fu, Y.; et al. 2024. C-eval: A multilevel multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems , 36.

Jiang, Y.; Wang, Y.; Zeng, X.; Zhong, W.; Li, L.; Mi, F.; Shang, L.; Jiang, X.; Liu, Q.; and Wang, W. 2023. Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint arXiv:2310.20410 .

Laskar, M. T. R.; Alqahtani, S.; Bari, M. S.; Rahman, M.; Khan, M. A. M.; Khan, H.; Jahan, I.; Bhuiyan, A.; Tan, C. W.; Parvez, M. R.; et al. 2024. A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations. arXiv preprint arXiv:2407.04069 .

Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Alpacaeval: An automatic evaluator of instruction-following models. Li, Y.; Zhang, G.; Qu, X.; Li, J.; Li, Z.; Wang, Z.; Li, H.; Yuan, R.; Ma, Y.; Zhang, K.; et al. 2024. CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models. arXiv preprint arXiv:2402.13109 .

Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688 .

Luo, Z.; Xu, C.; Zhao, P.; Sun, Q.; Geng, X.; Hu, W.; Tao, C.; Ma, J.; Lin, Q.; and Jiang, D. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 .

Mishra, S.; Khashabi, D.; Baral, C.; and Hajishirzi, H. 2022. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 3470-3487.

Mukherjee, S.; Mitra, A.; Jawahar, G.; Agarwal, S.; Palangi, H.; and Awadallah, A. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 .

Qin, Y.; Song, K.; Hu, Y.; Yao, W.; Cho, S.; Wang, X.; Wu, X.; Liu, F.; Liu, P.; and Yu, D. 2024. InFoBench: Evaluating Instruction Following Ability in Large Language Models. arXiv preprint arXiv:2401.03601 .

Sanh, V.; Webson, A.; Raffel, C.; Bach, S. H.; Sutawika, L.; Alyafeai, Z.; Chaffin, A.; Stiegler, A.; Le Scao, T.; Raja, A.; et al. 2022. Multitask Prompted Training Enables ZeroShot Task Generalization. In ICLR 2022-Tenth International Conference on Learning Representations .

Sun, H.; Liu, L.; Li, J.; Wang, F.; Dong, B.; Lin, R.; and Huang, R. 2024a. Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models. arXiv preprint arXiv:2404.02823 .

Sun, K.; Wang, R.; Liu, H.; and Søgaard, A. 2024b. Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach. arXiv preprint arXiv:2403.15250 .

Tang, X.; Zong, Y.; Zhao, Y.; Cohan, A.; and Gerstein, M. 2023. Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data? arXiv preprint arXiv:2309.08963 .

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Tsai, A.; Oraby, S.; Perera, V.; Kao, J.-Y.; Du, Y.; NarayanChen, A.; Chung, T.; and Hakkani-Tur, D. 2021. Style Control for Schema-Guided Natural Language Generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI , 228-242.

Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 13484-13508.

Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 .

Weller, O.; Lourie, N.; Gardner, M.; and Peters, M. E. 2020. Learning from Task Descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 1361-1375.

Wen, B.; Ke, P.; Gu, X.; Wu, L.; Huang, H.; Zhou, J.; Li, W.; Hu, B.; Gao, W.; Xu, J.; et al. 2024. Benchmarking Complex Instruction-Following with Multiple Constraints Composition. arXiv preprint arXiv:2407.03978 .

Xia, C.; Xing, C.; Du, J.; Yang, X.; Feng, Y.; Xu, R.; Yin, W.; and Xiong, C. 2024. FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability. arXiv preprint arXiv:2402.18667 .

Xu, C.; Sun, Q.; Zheng, K.; Geng, X.; Zhao, P.; Feng, J.; Tao, C.; and Jiang, D. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 .

Yang, S.; Nachum, O.; Du, Y.; Wei, J.; Abbeel, P.; and Schuurmans, D. 2023. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129 .

Yao, S.; Chen, H.; Hanjie, A. W.; Yang, R.; and Narasimhan, K. 2023. Collie: Systematic construction of constrained text generation tasks. arXiv preprint arXiv:2307.08689 .

Zhang, H.; Song, H.; Li, S.; Zhou, M.; and Song, D. 2023. A survey of controllable text generation using transformerbased pre-trained language models. ACM Computing Surveys , 56(3): 1-37.

Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2024.

Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36.

Zhong, R.; Lee, K.; Zhang, Z.; and Klein, D. 2021. Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections. In Findings of the Association for Computational Linguistics: EMNLP 2021 , 28562878.

Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2024. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In Findings of the Association for Computational Linguistics: NAACL 2024 , 2299-2314.

Zhou, J.; Lu, T.; Mishra, S.; Brahma, S.; Basu, S.; Luan, Y.; Zhou, D.; and Hou, L. 2023a. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911 .

Zhou, W.; Jiang, Y. E.; Wilcox, E.; Cotterell, R.; and Sachan, M. 2023b. Controlled text generation with natural language instructions. In International Conference on Machine Learning , 42602-42613. PMLR.

## CFBench Construction

## Constraint System Construction

The construction of the constraint system commenced with the aggregation of data from diverse real-world scenarios and NLP tasks. This encompassed 800,000 query logs from LLM websites over the preceding six months, alongside over 300,000 data points from various NLP tasks. Instructions that were excessively long or short were filtered out, and a vector clustering deduplication algorithm was employed. This meticulous process culminated in a refined dataset comprising approximately 30,000 instructions. Subsequently, GPT was utilized to extract constraint atoms from these instructions, thereby ensuring the comprehensiveness of the constraint system. The prompt employed for GPT4 extraction, as illustrated in Fig. 7, resulted in the identification of approximately 5,000 unique atomic constraint instructions. Three seasoned experts meticulously refined these into 1,000 significant atomic constraints. By integrating statistical analysis, classification, and linguistic principles, a hierarchical constraint system was developed using Top-Down Organization and Bottom-Up Synthesis methodologies. This system comprises 10 primary categories and 25 secondary categories. The system comprehensively categorizes all types of constraints, ensuring that nearly all specific constraints can be systematically classified within its framework. Detailed information regarding the constraint system is presented in Tab. 8.

## Dataset Construction

Weadopted an innovative Human-Machine Collaborative Iterative Construction approach to ensure the highest quality of data. This method involved leveraging advanced LLMs to augment original instructions with additional constraints and generate corresponding responses. These responses were meticulously reviewed for constraint validity, followed by the creation of detailed checklists for each example. Multiple experts participated in this iterative process, continuously refining the outputs by addressing issues encountered by the LLMs and regenerating or manually correcting any substandard samples. The prompts used for GPT to enhance constraints and generate checklists are illustrated in Fig. 8 and Fig. 10. Due to the limited attention given to real-life scenarios, we have meticulously organized and covered 20 domains and over 200 scenarios in our CFBench system, as detailed in Tab. 9. In the end, we gathered 1,000 high-quality data points: 500 from real-world scenarios and 500 from different NLP tasks. Specifically, we implemented the following steps to enhance data quality for manual annotations.

Annotator Training We sourced annotation contractors from the public and selected 21 candidates for training by seasoned data scientists. After a one-week training program, the annotators engaged in multiple rounds of trial annotations, which were then assessed by data experts. From these assessments, the 9 annotators demonstrating the highest accuracy were selected for this dataset.

Cross-Validation To reduce the likelihood of missed and incorrect annotations, we implemented an inter-annotator

Figure 7: Prompt Template Atomic Constraint Extract

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-0a38d692-42cf-42c8-b4a4-907d399f1495.jpg)

Figure 8: Prompt Template for Adding Atomic constraint

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-52a30041-f3e6-48db-91cc-c7025a73162d.jpg)

validation process. Three annotators independently reviewed the labeled instructions, responses, and evaluation criteria, achieving a notable agreement rate of 94%. Any discrepancies that emerged were resolved through expert adjudication, ensuring both consistency and accuracy.

Batch Validation Due to the substantial size of the dataset, it was systematically divided for processing. Following a phased improvement approach, the initial batch sizes were set at 50, 100, and 200, gradually increasing to 400 for later batches. After the annotation process, 50% of the dataset was randomly selected for contractor review, while 20% of the dataset was examined by experts.

Data Split We used a voting mechanism involving experts and ten models, including GPT-4o and Claude-3.5-Sonnet, to partition 10,000 CFBench entries into 'easy' and 'hard' categories. The 'hard' category includes entries where multiple models struggle with PSR performance and are also challenging for humans, as verified by experts.

## Evaluation Method and Metric

The state-of-the-art GPT-4o model was employed as the judge to perform binary scoring (0 or 1) for each checkpoint in the checklist. The specific evaluation prompt is illus-

Figure 9: Prompt Template for Evaluation

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01122v1-figure-2e5aab38-4034-4ec3-9e97-f69ece2687b1.jpg)

Table 5: The High-Quality Rate of 100 selected Samples

| Set      |   Instruction |   Response |   CheckList |
|----------|---------------|------------|-------------|
| Easy Set |          0.96 |       0.94 |        0.93 |
| Hard Set |          0.92 |       0.95 |        0.93 |
| All Set  |          0.94 |       0.94 |        0.93 |

trated in Fig. 9. The Requirement Priority-Satisfaction Ratio (PSR) was proposed as an evaluation metric that simultaneously considers the prioritization of user requirements and satisfaction levels. PSR is calculated by first ensuring that all primary requirements are met. Subsequently, the satisfaction score is determined by averaging the fulfillment of the remaining constraints to obtain A . The final satisfaction score is then calculated using the formula 0 5 + 0 5 . . ∗ A . If the final score exceeds 0 8 . , PSR is set to 1 . The threshold of 0 8 . was established based on the average satisfaction levels derived from multiple users' feedback on the responses to the instructions.

## Quality Assessment

We employed multiple methods to validate the quality of the benchmark on a randomly selected set of 100 samples. First, we engaged three experts to independently evaluate the quality of each sample's instruction, response, and criteria. The average quality rate determined by the three experts was consistently above 90%, as detailed in Tab. 5. To further validate the effectiveness of our proposed evaluation metric, PSR, we had the same three experts score the responses of Qwen2-7B-Instruct on these 100 cases using a 0-1 scale. Simultaneously, we utilized GPT-4o to directly score the responses, referred to as GPT-4o PSR. By calculating the kappa coefficient, we found a strong agreement between our proposed PSR evaluation metric and the human experts' assessments. The detailed results are presented in Tab. 6. Kappa coefficient scores are interpreted as follows: below 0.2 indicates slight agreement, 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 indicates moderate agreement, 0.61 to 0.80 indicates substantial agreement, and 0.81 to 1.00 indicates almost perfect agreement.

| Set                 |   Easy Set |   Hard Set |   Full Set |
|---------------------|------------|------------|------------|
| Avg.Expert          |       1    |       1    |       1    |
| GPT-4o DS           |       0.58 |       0.61 |       0.6  |
| GPT-4o PSR          |       0.76 |       0.77 |       0.77 |
| Qwen2-72B-Inst. PSR |       0.7  |       0.73 |       0.72 |

Table 6: The kappa coefficient between expert evaluations and various assessment methods

## Experimental Setup and Results

## Experiment Setting

We evaluated the most popular Large Language Models (LLMs), with the majority of these models being developed by companies based in China, primarily to accommodate our CFBench's focus on the Chinese language. Among the 50 evaluated models, they can be categorized into two groups based on their access method: API-based and open-source weight-based models. It is worth noting that the Llama series models do not primarily support the Chinese language, which results in noticeably lower performance. Both conifer-base and conifer-test are based on the Mistral-7B foundational model. Llama-38B-Instruct-CN and Llama-3-70B-Instruct-CN respectively represent Llama-3-8B-Instruct-Chinese and Llama-3-70BInstruct-Chinese, both of which have undergone Chinese SFT (Supervised Fine-Tuning). For the base models, we used a 3-shot approach to ensure a fair evaluation. The complete list of evaluated models can be found in Tab. 7.

## Explanation of Results

GPT-4o and Claude3.5-Sonnet have demonstrated nearabsolute leadership, achieving outstanding performance across various metrics and categories. Similarly, models such as GLM-4-0510, ERNIE-4-Bot-0613, ERNIE-4Turbo-0628, DeepSeek-V2-0628, and Qwen2-72B-Instruct have also exhibited strong capabilities. Many models that support less mainstream Chinese languages performed significantly worse, which is unfair to them and only serves to illustrate their relative rankings. This also confirms that performance are highly correlated with language, especially within the scope of language constraints. From the perspective of open-source versus closed-source models, open-source models have generally achieved comprehensive success. However, Qwen2-72B-Instruct, as an open-source model, also demonstrated notable constraint-following capabilities. Regarding model size, within the Qwen series, performance metrics clearly improve with increasing model size. Additionally, models that have undergone Supervised Fine-Tuning (SFT) show significantly enhanced instructionfollowing capabilities. The complete evaluation results and rankings can be found in Tab. 7.

| Models                      | Easy Set    | Easy Set    | Easy Set    | Hard Set    | Hard Set    | Hard Set    | Full Set    | Full Set    | Full Set    |
|-----------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|
| Models                      | CSR         | ISR         | PSR         | CSR         | ISR         | PSR         | CSR         | ISR         | PSR         |
| GPT-4o †                    | 0.956       | 0.868       | 0.888       | 0.816       | 0.438       | 0.582       | 0.886       | 0.653       | 0.735       |
| GPT-4-Turbo-20240409 †      | 0.924       | 0.792       | 0.826       | 0.783       | 0.370       | 0.518       | 0.853       | 0.581       | 0.672       |
| GPT-4-0125-Preview †        | 0.923       | 0.790       | 0.826       | 0.763       | 0.310       | 0.468       | 0.843       | 0.550       | 0.647       |
| GPT-3.5-Turbo-1106 †        | 0.797       | 0.520       | 0.602       | 0.631       | 0.176       | 0.326       | 0.714       | 0.348       | 0.464       |
| Claude-3.5-Sonnet †         | 0.943       | 0.844       | 0.882       | 0.799       | 0.408       | 0.564       | 0.871       | 0.626       | 0.723       |
| GLM-4-0520 †                | 0.939       | 0.820       | 0.852       | 0.785       | ✿✿✿✿ 0.372  | ✿✿✿✿ 0.536  | 0.862       | ✿✿✿✿ 0.596  | 0.694       |
| DeepSeek-V2-0628 †          | 0.946       | 0.830       | 0.868       | 0.786       | 0.350       | 0.524       | 0.866       | 0.590       | 0.696       |
| ERNIE-4-Turbo-0628 †        | 0.930       | 0.790       | 0.848       | 0.772       | 0.332       | 0.532       | 0.851       | 0.561       | 0.690       |
| ERNIE-4-Bot-0613 †          | 0.929       | 0.792       | 0.832       | 0.779       | 0.338       | 0.518       | 0.854       | 0.565       | 0.675       |
| ERNIE-3.5-0613 †            | 0.901       | 0.720       | 0.772       | 0.758       | 0.302       | 0.482       | 0.830       | 0.511       | 0.627       |
| Yi-Large †                  | 0.900       | 0.730       | 0.786       | 0.744       | 0.292       | 0.460       | 0.822       | 0.511       | 0.623       |
| abab6.5-chat †              | 0.894       | 0.696       | 0.766       | 0.736       | 0.260       | 0.452       | 0.815       | 0.478       | 0.609       |
| MoonShot-V1-8k †            | 0.919       | 0.764       | 0.812       | 0.758       | 0.308       | 0.464       | 0.838       | 0.536       | 0.638       |
| Vicuna-7B-V13 ∗             | 0.563       | 0.206       | 0.262       | 0.468       | 0.100       | 0.168       | 0.516       | 0.153       | 0.215       |
| Vicuna-33B-V13 ∗            | 0.621       | 0.270       | 0.352       | 0.527       | 0.110       | 0.196       | 0.574       | 0.190       | 0.274       |
| Vicuna-13B-V13 ∗            | 0.605       | 0.248       | 0.302       | 0.503       | 0.100       | 0.178       | 0.554       | 0.174       | 0.240       |
| Llama-2-7B-Chat ∗           | 0.5268      | 0.198       | 0.250       | 0.448       | 0.096       | 0.152       | 0.487       | 0.147       | 0.201       |
| Llama-2-13B-Chat ∗          | 0.574       | 0.242       | 0.280       | 0.488       | 0.094       | 0.178       | 0.531       | 0.168       | 0.229       |
| Llama-3-8B-Instruct ∗       | 0.656       | 0.300       | 0.356       | 0.562       | 0.122       | 0.238       | 0.609       | 0.211       | 0.297       |
| Llama-3-70B-Instruct ∗      | 0.750       | 0.422       | 0.498       | 0.642       | 0.178       | 0.330       | 0.696       | 0.300       | 0.414       |
| Mistral-7B-Instruct-V03 ∗   | 0.227       | 0.072       | 0.086       | 0.148       | 0.008       | 0.022       | 0.188       | 0.040       | 0.054       |
| Conifer-Base ∗              | 0.510       | 0.184       | 0.232       | 0.300       | 0.018       | 0.048       | 0.405       | 0.101       | 0.140       |
| Conifer-Test ∗              | 0.559       | 0.215       | 0.255       | 0.328       | 0.102       | 0.156       | 0.443       | 0.159       | 0.206       |
| BaiChuan-13B-Chat           | 0.630       | 0.306       | 0.366       | 0.521       | 0.114       | 0.196       | 0.575       | 0.210       | 0.281       |
| BaiChuan2-13B-Chat          | 0.669       | 0.348       | 0.418       | 0.547       | 0.134       | 0.226       | 0.608       | 0.241       | 0.322       |
| Llama-3-8B-Instruct-CN      | 0.743       | 0.458       | 0.510       | 0.627       | 0.162       | 0.314       | 0.685       | 0.310       | 0.412       |
| Llama-3-70B-Instruct-CN     | 0.756       | 0.482       | 0.536       | 0.636       | 0.190       | 0.322       | 0.696       | 0.336       | 0.429       |
| DeepSeek-7B-Chat            | 0.695       | 0.378       | 0.442       | 0.580       | 0.150       | 0.270       | 0.638       | 0.264       | 0.356       |
| DeepSeek-V2-Lite-Chat       | 0.733       | 0.382       | 0.448       | 0.597       | 0.148       | 0.262       | 0.665       | 0.265       | 0.355       |
| DeepSeek-67B-Chat           | 0.802       | 0.516       | 0.578       | 0.662       | 0.180       | 0.350       | 0.732       | 0.348       | 0.464       |
| InternLM2-Chat-7B           | 0.767       | 0.452       | 0.538       | 0.625       | 0.172       | 0.320       | 0.696       | 0.312       | 0.429       |
| GLM-4-9B-Chat               | 0.885       | 0.678       | 0.742       | 0.742       | 0.288       | 0.450       | 0.813       | 0.483       | 0.596       |
| YI-1.5-34B-Chat             | 0.881       | 0.672       | 0.740       | 0.745       | 0.302       | 0.474       | 0.813       | 0.487       | 0.607       |
| Qwen1.5-4B                  | 0.454       | 0.170       | 0.198       | 0.376       | 0.074       | 0.116       | 0.415       | 0.122       | 0.157       |
| Qwen1.5-4B-Chat             | 0.652       | 0.310       | 0.362       | 0.536       | 0.104       | 0.198       | 0.594       | 0.207       | 0.280       |
| Qwen1.5-7B                  | 0.473       | 0.176       | 0.212       | 0.400       | 0.090       | 0.142       | 0.437       | 0.133       | 0.177       |
| Qwen1.5-7B-Chat             | 0.799       | 0.534       | 0.592       | 0.654       | 0.194       | 0.338       | 0.726       | 0.364       | 0.465       |
| Qwen1.5-14B                 | 0.498       | 0.228       | 0.280       | 0.430       | 0.110       | 0.176       | 0.464       | 0.169       | 0.228       |
| Qwen1.5-14B-Chat            | 0.822       | 0.558       | 0.626       | 0.671       | 0.202       | 0.370       | 0.746       | 0.380       | 0.498       |
| Qwen1.5-32B-Chat            | 0.647       | 0.336       | 0.408       | 0.528       | 0.132       | 0.224       | 0.587       | 0.234       | 0.316       |
| Qwen1.5-32B                 | 0.883       | 0.678       | 0.744       | 0.704       | 0.228       | 0.412       | 0.793       | 0.453       | 0.578       |
| Qwen1.5-72B                 | 0.627       | 0.324       | 0.380       | 0.556       | 0.148       | 0.248       | 0.591       | 0.236       | 0.314       |
| Qwen1.5-72B-Chat            | 0.896       | 0.710       | 0.776       | 0.730       | 0.254       | 0.436       | 0.813       | 0.482       | 0.606       |
| Qwen1.5-110B-Chat           | 0.905       | 0.724       | 0.792       | 0.730       | 0.276       | 0.438       | 0.818       | 0.500       | 0.615       |
| Qwen2-0.5B-Instruct         | 0.446       | 0.150       | 0.172       | 0.393       | 0.070       | 0.110       | 0.419       | 0.110       | 0.141       |
| Qwen2-1.5B-Instruct         | 0.607       | 0.250       | 0.316       | 0.496       | 0.104       | 0.168       | 0.551       | 0.177       | 0.242       |
| Qwen2-7B                    | 0.576       | 0.260       | 0.316       | 0.478       | 0.120       | 0.192       | 0.527       | 0.190       | 0.254       |
| Qwen2-7B-Instruct Qwen2-72B | 0.835 0.711 | 0.584 0.424 | 0.642 0.484 | 0.682 0.568 | 0.198 0.170 | 0.362 0.274 | 0.758 0.640 | 0.391 0.297 | 0.502 0.379 |
|                             |             |             | 0.880       | 0.791       | 0.342       | 0.530       | 0.867       | 0.589       | 0.705       |
| Qwen2-72B-Instruct          | ✿✿✿✿ 0.944  | ✿✿✿✿ 0.836  | ✿✿✿✿        | ✿✿✿✿        |             |             | ✿✿✿✿        |             | ✿✿✿✿        |

Table 7: The complete evaluation results and rankings of CFBench and its respective subsets. Notably, ∗ stands for the model supporting mainstream languages excluding Chinese, and † represents calling through the API. The bold , underlined, and tilde ✿✿✿ denote the first, second, and third rankings, respectively. Llama-3-8B-Instruct-CN and Llama-3-70B-Instruct-CN respectively represent Llama-3-8B-Instruct-Chinese and Llama-3-70B-Instruct-Chinese, both of which have undergone Chinese SFT (Supervised Fine-Tuning). Both conifer base and conifer test are based on the Mistral-7B foundational model. For the base model, we used a 3-shot approach for generation.

Table 8: Constraint System of CFBench

| Primary                  | Secondary                                                                     | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                            | Example                                                                                                                                                              |
|--------------------------|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Content Constraint       | Lexical Element Semantic                                                      | Mandatory use of specific terms or symbols, in- cluding their inclusion and precise placement. Mandates for including specific elements or con- cepts in responses, reflecting a scenario or object. Directives on thematic content, perspective, or tone, emphasizing response significance.                                                                                                                                                         | ...must include the word 'beautiful.' ...highlights the Great Wall. Write a poem about London.                                                                       |
| Numerical Constraint     | Word Count Sentence Count Paragraph Count Document Count                      | Limit the number of words or tokens. Limit the number of sentences. Limit the number of paragraphs. Limit the number of documents.                                                                                                                                                                                                                                                                                                                    | A 50-word poem. ... three sentences. divided into 3 sections. ... list 3 articles.                                                                                   |
| Stylistic Constraint     | Tone and emotion Form and style Audience-specific Authorial style Fundamental | The emotional tone must adhere to standards such as seriousness, anger, joy, humor, and politeness. Text expression standards ensure alignment with specific stylistic criteria in both presentation and perception. Text should be tailored to specific audiences, en- suring clarity and relevance for children, students, or specialized groups. Texts should emulate the styles of authors like Shakespeare to achieve artistic effects or depth. | Write a letter in an an- gry and sarcastic tone. Write a passage in an encyclopedic style. Write a pome for a 6- year-old. Write a passage the style of Shakespeare. |
| Format Constraint        | Bespoke                                                                       | Widely accepted and utilized standard formats, in- cluding JSON, XML, LaTeX, HTML, Table, and Markdown. Protocols for information expression tailored to specific needs,including paragraphing, headings, text emphasis, examples, and bullet points. Formatting standards tailored for specialized ap-                                                                                                                                               | Extract keywords and output in JSON for- mat. Summarize the main idea and output in un- ordered list format. Conform to electronic                                   |
| Linguistic Constraint    | Pragmatic Syntactic Morphological                                             | Contextual language study, encompassing speech acts, implicature, discourse, dialects, sociolects, and language policy. Sentence structure, including phrases, con- stituents, subordinate clauses, ba-constructions, and imperatives. The internal structure and formation rules of words, including roots, affixes, and morphologi- cal changes.                                                                                                    | Output in English, in classical Chinese style. Use imperatives with nouns and verb phrases. Output all content in lowercase English.                                 |
| Situation Constraint     | Role-based Task-specific Complex context -                                    | their traits, language, and behaviors. Offer tailored solutions based on a nuanced under- standing of situational demands. Reasoning and problem-solving within intricate and multifaceted contexts. Regulate new responses by leveraging intrinsic                                                                                                                                                                                                   | how do you decide? Must work from home, how to report? 4 on the left, 10 total, which from right? Example:input:xxx, output: { ... } ; input:xx,                     |
| Example Constraint       | -                                                                             | patterns from a limited set of samples.                                                                                                                                                                                                                                                                                                                                                                                                               | output?                                                                                                                                                              |
| Inverse Constraint       |                                                                               | Narrow the response space through inverse con- straints and indirect exclusion.                                                                                                                                                                                                                                                                                                                                                                       | Prohibited from an- swer political topics.                                                                                                                           |
|                          |                                                                               |                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Write a five-character                                                                                                                                               |
| Contradictory Constraint | -                                                                             | Mutually exclusive constraints prevent fulfilling all requirements concurrently.                                                                                                                                                                                                                                                                                                                                                                      | quatrain, 1000 words.                                                                                                                                                |
| Rule Constraint          | -                                                                             | the road of responses through metic- crafted logic flows or actions.                                                                                                                                                                                                                                                                                                                                                                                  | Each answer adds 1+1=3, then 2+3=?                                                                                                                                   |
|                          |                                                                               | Standardize ulously                                                                                                                                                                                                                                                                                                                                                                                                                                   | 1,                                                                                                                                                                   |

## Domain

## Scenarios List

Table 9: Domain and Scenarios List

|                | Symptom Consultation                                                     | Diagnostic Explanation                                  | Medication Guidance                                            | Procedures                                                  |
|----------------|--------------------------------------------------------------------------|---------------------------------------------------------|----------------------------------------------------------------|-------------------------------------------------------------|
| Healthcare     | Wellness Medical Education                                               | Medical Info                                            | Guidelines Inquiry                                             | Public Health                                               |
| Education      | Teaching Methods Academic Counseling Reports                             | Resource Access Mental Health Support Interests         | Curriculum Design Tutoring                                     | Communication Subject Q&A                                   |
| Finance        | Market Research Corporate Tax Financial Education Regulatory Analysis    | Stock Analysis Insurance Management Product Development | Investment Analysis Corporate Financing Customer Service       | Personal Finance Compliance &Risk Financial Reports         |
|                | Legal Education Statute Explanation                                      | Legal Consultation Regulation Analysis                  | Document Review IP Management                                  | Case Analysis Legal Training                                |
| Legal Media    | Case Management Content Creation                                         | Compliance &Risk Information Analysis                   | Marketing &Promotion                                           | News Reporting                                              |
| Recruitment    | Interview Evaluation Performance Review Policy Research Document Writing | Career Planning Public Education Content Review         | Offer Comparison Service Guide Business Procedures             | Communication Skills Public Services Civil Servant Training |
| Gov Affairs    | Emergency                                                                |                                                         |                                                                |                                                             |
| Real           | Management Purchase Planning Leasing Property Description                | Market Trends                                           | Property Policies                                              | Development                                                 |
| Estate         | Renovation Marketing &Sales                                              | Property Valuation Content Creation                     | Amenities Sales &Marketing                                     | Financial Services Qualifications                           |
| Automotive     | Model Comparison Car Reviews                                             | Driving &Safety Loan Calculation Maintenance &Repair    | Customer Experience Insurance Evaluation                       | Model Consultation Claims Assessment                        |
|                | Romance                                                                  | Family                                                  | Friendship Sexuality &Gender                                   | Workplace                                                   |
| Psychology     | Self &Health Organizational                                              | Social Client Relations                                 | Crisis Intervention User Research                              | Life Stages Public Psychology Coding                        |
| Internet       | Business Analysis Product Testing                                        | Product Design Data Management                          | Cybersecurity                                                  | &Debugging Computer Q&A UI/UX Design                        |
| Spirituality   | Internet News Beliefs &Rituals                                           | Marketing Divination                                    | Operations Feng Shui                                           | Astrology                                                   |
| Sports         | Metaphysics Training                                                     | Spirituality Goal Setting Performance                   | Healing Nutrition                                              | Content Review Workout Plans Specialized                    |
|                | Equipment &Tech Mental Motivation Life Tips                              | Data Tracking Shopping Decisions                        | Injury Care Instant Queries                                    | Training Skincare                                           |
| Lifestyle      | Fashion &Styling Socializing Podcasts &Radio                             | Naming Life Creations TV &Film                          | Recommendations Q&A Music                                      | Planning Literature                                         |
| Culturtainment | Theater &Dance                                                           | Art Content Creation                                    | Cultural Events                                                | Short Videos &Live                                          |
| Employment     | Gossip Project Management Administration                                 | Translation Customer Service Reviews &Feedback          | Office Efficiency Team Collaboration                           | Marketing Collaboration                                     |
| Cuisine        | Food &Restaurant Recs Culinary Culture Nutrition &Health                 | Recipes &Menus Food Safety Reviews                      | Marketing &Promo Cooking Techniques Culinary Training Hardware | Food Content Ingredient Prep                                |
|                | Guides                                                                   |                                                         |                                                                |                                                             |
|                |                                                                          |                                                         | &Peripherals                                                   | News                                                        |
| Gaming         | Software &Services                                                       | Development &Design                                     | Operations                                                     | Mini Games                                                  |
|                | Search                                                                   | Marketing &Promotion Launches                           | Esports &Tournaments                                           | Culture                                                     |
| Technology     | Reviews                                                                  |                                                         | Buying Guides                                                  | &Education Tips &Tricks                                     |
|                | Content Creation                                                         | Product Design                                          | Marketing Copy                                                 | After-sales &Repairs                                        |

## # Task Description

## # Given Input

[Instruction]:{instruction}

[Respone]:{response}

# Your Answer about Checklist

## # Given Example

## Prompt Template for CheckList Generation

You are a professional instruction and response review expert. You need to extract all possible checklist items that could affect the evaluation of the result based on the [Instruction] and [Response] within the [Constraint System]. Your output should be as detailed as possible. Use numerical numbering and "\n" to list all points in the checklist.

## # Constraint System

1. Lexical content constraint: {Definition} {Example}

2. Word Count ：

{Definition} {Example}

...

25.Rule Constraint:  {Definition} {Example}

## CheckList

1.Content on Trump Shooting Incident   2.Includes cause, process, impact ...     3.Use numbered points and bold text     4.Cause in JSON

5.Process in three sentences                    6.Impact: international and domestic    7.Total content under 500 words            8.Seven-word quatrain

Response:    Overview of the Trump Shooting Incident. 1.Cause \n{"time":"July 13, 2024", "location":"Butler, Pennsylvania, USA","nature":"assassination attempt"} \n 2.Process \n-Trump was giving a speech at the rally when the shooting occurred... 3.Impact \n-International: The incident shocked the world, raising concerns about US political violence. \n-Domestic: US political tensions and polarization increased; Trump's support may rise.\n 4.Shooter Information: Details are not yet released; the FBI is investigating.\n 5.Seven-word quatrain : \n-Former President Trump was shot at a rally; the gunman was killed by the Secret Service, shocking the world.

Overview of the Trump Shooting Incident, include the cause, process, impact, aftermath, shooter information and Trump's feedback. Organize the content using numbered points and bold text. The cause should be in JSON format with event keywords: time,  location, and  nature. Summarize the process in three sentences. Discuss the impact from both international and domestic perspectives, keeping the total content under 500 words. Finally,   summarize this event in a seven-word quatrain. Instruction:

Figure 10: Prompt Template for CheckList Generation
Tao Zhang Yanjun Shen Wenjing Luo Yan Zhang Hao Liang Tao Zhang Fan Yang Mingan Lin Yujing Qiao Weipeng Chen Bin Cui Wentao Zhang Zenan Zhou
CFBench: A Comprehensive Constraints-Following Benchmark for LLMs
The adeptness of Large Language Models (LLMs) in comprehending and following
natural language instructions is critical for their deployment in sophisticated
real-world applications. Existing evaluations mainly focus on fragmented
constraints or narrow scenarios, but they overlook the comprehensiveness and
authenticity of constraints from the user's perspective. To bridge this gap, we
propose CFBench, a large-scale Comprehensive Constraints Following Benchmark
for LLMs, featuring 1,000 curated samples that cover more than 200 real-life
scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from
real-world instructions and constructs an innovative systematic framework for
constraint types, which includes 10 primary categories and over 25
subcategories, and ensures each constraint is seamlessly integrated within the
instructions. To make certain that the evaluation of LLM outputs aligns with
user perceptions, we propose an advanced methodology that integrates
multi-dimensional assessment criteria with requirement prioritization, covering
various perspectives of constraints, instructions, and requirement fulfillment.
Evaluating current leading LLMs on CFBench reveals substantial room for
improvement in constraints following, and we further investigate influencing
factors and enhancement strategies. The data and code are publicly available at
https://github.com/PKU-Baichuan-MLSystemLab/CFBench