title: CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs
    with Controllable Puzzle Generation
summary: Existing reasoning evaluation frameworks for Large Language Models (LLMs) and
  Large Vision-Language Models (LVLMs) predominantly either assess text-based
  reasoning or vision-language understanding capabilities, with limited dynamic
  interplay between textual and visual constraints. To address this limitation,
  we introduce CrossWordBench, a benchmark designed to evaluate the reasoning
  capabilities of both LLMs and LVLMs through the medium of crossword puzzles-a
  task requiring multimodal adherence to semantic constraints from text-based
  clues and intersectional constraints from visual grid structures.
  CrossWordBench leverages a controllable puzzle generation framework that
  produces puzzles in multiple formats (text and image) and offers different
  evaluation strategies ranging from direct puzzle solving to interactive modes.
  Our extensive evaluation of over 20 models reveals that reasoning LLMs
  outperform non-reasoning models substantially by effectively leveraging
  crossing-letter constraints. We further demonstrate that LVLMs struggle with
  the task, showing a strong correlation between their puzzle-solving performance
  and grid-parsing accuracy. Our findings offer insights into the limitations of
  the reasoning capabilities of current LLMs and LVLMs, and provide an effective
  approach for creating multimodal constrained tasks for future evaluations.
paper_authors._id: 67ec9d4ad327ed17ec707488
paper_authors.name: Jixuan Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:47:14.691Z
paper_authors._id: 64811214cacb1c4a06988bd9
paper_authors.avatarUrl: /avatars/538a3029878c2200df049b54c97c9327.svg
paper_authors.fullname: Jixuan Leng
paper_authors.type: user
paper_authors.user: JixuanLeng
paper_authors._id: 67ec9d4ad327ed17ec707489
paper_authors.name: Chengsong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:47:27.410Z
paper_authors._id: 62ea79dd01ed9b0e8f61ccd3
paper_authors.avatarUrl: /avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
paper_authors.fullname: Chengsong Huang
paper_authors.type: user
paper_authors.user: ChengsongHuang
paper_authors._id: 67ec9d4ad327ed17ec70748a
paper_authors.name: Langlin Huang
paper_authors._id: 67ec9d4ad327ed17ec70748b
paper_authors.name: Bill Yuchen Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:47:44.967Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 67ec9d4ad327ed17ec70748c
paper_authors.name: William W. Cohen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:48:09.463Z
paper_authors._id: 66a28706d3449709d6943ef2
paper_authors.avatarUrl: /avatars/804f45be500d40e1c0e973c691dd2c73.svg
paper_authors.fullname: William Cohen
paper_authors.type: user
paper_authors.user: wwcohen
paper_authors._id: 67ec9d4ad327ed17ec70748d
paper_authors.name: Haohan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:48:23.506Z
paper_authors._id: 670d920206218f52e8ea376d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/j_-zv1ftLCJbjjw9-wnN8.png
paper_authors.fullname: Haohan Wang
paper_authors.type: user
paper_authors.user: haohanw
paper_authors._id: 67ec9d4ad327ed17ec70748e
paper_authors.name: Jiaxin Huang
paper_title: CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs
    with Controllable Puzzle Generation
paper_summary: Existing reasoning evaluation frameworks for Large Language Models (LLMs) and
  Large Vision-Language Models (LVLMs) predominantly either assess text-based
  reasoning or vision-language understanding capabilities, with limited dynamic
  interplay between textual and visual constraints. To address this limitation,
  we introduce CrossWordBench, a benchmark designed to evaluate the reasoning
  capabilities of both LLMs and LVLMs through the medium of crossword puzzles-a
  task requiring multimodal adherence to semantic constraints from text-based
  clues and intersectional constraints from visual grid structures.
  CrossWordBench leverages a controllable puzzle generation framework that
  produces puzzles in multiple formats (text and image) and offers different
  evaluation strategies ranging from direct puzzle solving to interactive modes.
  Our extensive evaluation of over 20 models reveals that reasoning LLMs
  outperform non-reasoning models substantially by effectively leveraging
  crossing-letter constraints. We further demonstrate that LVLMs struggle with
  the task, showing a strong correlation between their puzzle-solving performance
  and grid-parsing accuracy. Our findings offer insights into the limitations of
  the reasoning capabilities of current LLMs and LVLMs, and provide an effective
  approach for creating multimodal constrained tasks for future evaluations.
paper_ai_summary: CrossWordBench evaluates reasoning capabilities of LLMs and LVLMs using crossword puzzles, highlighting differences in textual and visual constraint handling.
paper_ai_keywords: CrossWordBench Large Language Models Large Vision-Language Models multimodal adherence semantic constraints visual grid structures controllable puzzle generation crossing-letter constraints grid-parsing accuracy