title: WildIFEval: Instruction Following in the Wild
summary: Recent LLMs have shown remarkable success in following user instructions, yet
  handling instructions with multiple constraints remains a significant
  challenge. In this work, we introduce WildIFEval - a large-scale dataset of 12K
  real user instructions with diverse, multi-constraint conditions. Unlike prior
  datasets, our collection spans a broad lexical and topical spectrum of
  constraints, in natural user prompts. We categorize these constraints into
  eight high-level classes to capture their distribution and dynamics in
  real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments
  to benchmark the instruction-following capabilities of leading LLMs. Our
  findings reveal that all evaluated models experience performance degradation
  with an increasing number of constraints. Thus, we show that all models have a
  large room for improvement on such tasks. Moreover, we observe that the
  specific type of constraint plays a critical role in model performance. We
  release our dataset to promote further research on instruction-following under
  complex, realistic conditions.
paper_authors._id: 67d02a147d82f613a31ed396
paper_authors.name: Gili Lior
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:22:05.211Z
paper_authors._id: 62fe0c7d7a4e53610af813a8
paper_authors.avatarUrl: /avatars/4188f4275249d10dc6747f63490d0cb6.svg
paper_authors.fullname: Gili Lior
paper_authors.type: user
paper_authors.user: gililior
paper_authors._id: 67d02a147d82f613a31ed397
paper_authors.name: Asaf Yehudai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:22:10.848Z
paper_authors._id: 638324f862badff43269e588
paper_authors.avatarUrl: /avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
paper_authors.fullname: Asaf Yehudai
paper_authors.type: user
paper_authors.user: Asaf-Yehudai
paper_authors._id: 67d02a147d82f613a31ed398
paper_authors.name: Ariel Gera
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T11:22:17.235Z
paper_authors._id: 6415ebd2107962562e9a0712
paper_authors.avatarUrl: /avatars/dfd2e6d42c6110bb64b6a73c96eeb3f3.svg
paper_authors.fullname: Ariel Gera
paper_authors.type: user
paper_authors.user: arielgera
paper_authors._id: 67d02a147d82f613a31ed399
paper_authors.name: Liat Ein-Dor
paper_title: WildIFEval: Instruction Following in the Wild
paper_summary: Recent LLMs have shown remarkable success in following user instructions, yet
  handling instructions with multiple constraints remains a significant
  challenge. In this work, we introduce WildIFEval - a large-scale dataset of 12K
  real user instructions with diverse, multi-constraint conditions. Unlike prior
  datasets, our collection spans a broad lexical and topical spectrum of
  constraints, in natural user prompts. We categorize these constraints into
  eight high-level classes to capture their distribution and dynamics in
  real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments
  to benchmark the instruction-following capabilities of leading LLMs. Our
  findings reveal that all evaluated models experience performance degradation
  with an increasing number of constraints. Thus, we show that all models have a
  large room for improvement on such tasks. Moreover, we observe that the
  specific type of constraint plays a critical role in model performance. We
  release our dataset to promote further research on instruction-following under
  complex, realistic conditions.