title: HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex
    Diagrams in Coding Tasks
summary: Understanding and reasoning over diagrams is a fundamental aspect of human
  intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive
  capabilities across various tasks, existing benchmarks lack comprehensive
  evaluation of their diagram interpretation and reasoning abilities,
  particularly in coding contexts. We present HumanEval-V, a rigorous benchmark
  of human-annotated coding tasks that spans six task types and evaluates diverse
  visual reasoning capabilities. Each task features carefully crafted diagrams
  paired with function signatures and test cases, employing novel code generation
  tasks to thoroughly assess models' diagram comprehension. Through extensive
  experiments with 22 LMMs, we find that even top-performing models achieve
  modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1,
  highlighting substantial room for improvement. Our analysis reveals that
  current LMMs struggle with spatial transformations, topological relationships,
  and dynamic patterns that humans find intuitive. These findings provide
  valuable insights for advancing LMMs' visual reasoning abilities. We have
  open-sourced our code and benchmark at
  https://github.com/HumanEval-V/HumanEval-V-Benchmark.
paper_authors._id: 67107f8dde5ba28c43c8c3f9
paper_authors.name: Fengji Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:37.248Z
paper_authors._id: 64c8d4ac120a85440bfad55d
paper_authors.avatarUrl: /avatars/d65334288d69700250de51d0df344d9b.svg
paper_authors.fullname: Fengji Zhang
paper_authors.type: user
paper_authors.user: zfj1998
paper_authors._id: 67107f8dde5ba28c43c8c3fa
paper_authors.name: Linquan Wu
paper_authors._id: 67107f8dde5ba28c43c8c3fb
paper_authors.name: Huiyu Bai
paper_authors._id: 67107f8dde5ba28c43c8c3fc
paper_authors.name: Guancheng Lin
paper_authors._id: 67107f8dde5ba28c43c8c3fd
paper_authors.name: Xiao Li
paper_authors._id: 67107f8dde5ba28c43c8c3fe
paper_authors.name: Xiao Yu
paper_authors._id: 67107f8dde5ba28c43c8c3ff
paper_authors.name: Yue Wang
paper_authors._id: 67107f8dde5ba28c43c8c400
paper_authors.name: Bei Chen
paper_authors._id: 67107f8dde5ba28c43c8c401
paper_authors.name: Jacky Keung
paper_title: HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex
    Diagrams in Coding Tasks
paper_summary: Understanding and reasoning over diagrams is a fundamental aspect of human
  intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive
  capabilities across various tasks, existing benchmarks lack comprehensive
  evaluation of their diagram interpretation and reasoning abilities,
  particularly in coding contexts. We present HumanEval-V, a rigorous benchmark
  of human-annotated coding tasks that spans six task types and evaluates diverse
  visual reasoning capabilities. Each task features carefully crafted diagrams
  paired with function signatures and test cases, employing novel code generation
  tasks to thoroughly assess models' diagram comprehension. Through extensive
  experiments with 22 LMMs, we find that even top-performing models achieve
  modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1,
  highlighting substantial room for improvement. Our analysis reveals that
  current LMMs struggle with spatial transformations, topological relationships,
  and dynamic patterns that humans find intuitive. These findings provide
  valuable insights for advancing LMMs' visual reasoning abilities. We have
  open-sourced our code and benchmark at
  https://github.com/HumanEval-V/HumanEval-V-Benchmark.
paper_ai_summary: HumanEval-V provides a comprehensive benchmark for evaluating LMMs' diagram interpretation and reasoning abilities, particularly in coding contexts, revealing significant areas for improvement.
paper_ai_keywords: Large Multimodal Models HumanEval-V diagram interpretation visual reasoning coding contexts function signatures test cases code generation spatial transformations topological relationships dynamic patterns