HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex
  Diagrams in Coding Tasks
Understanding and reasoning over diagrams is a fundamental aspect of human
intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive
capabilities across various tasks, existing benchmarks lack comprehensive
evaluation of their diagram interpretation and reasoning abilities,
particularly in coding contexts. We present HumanEval-V, a rigorous benchmark
of human-annotated coding tasks that spans six task types and evaluates diverse
visual reasoning capabilities. Each task features carefully crafted diagrams
paired with function signatures and test cases, employing novel code generation
tasks to thoroughly assess models' diagram comprehension. Through extensive
experiments with 22 LMMs, we find that even top-performing models achieve
modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1,
highlighting substantial room for improvement. Our analysis reveals that
current LMMs struggle with spatial transformations, topological relationships,
and dynamic patterns that humans find intuitive. These findings provide
valuable insights for advancing LMMs' visual reasoning abilities. We have
open-sourced our code and benchmark at
https://github.com/HumanEval-V/HumanEval-V-Benchmark.
67107f8dde5ba28c43c8c3f9
Fengji Zhang
claimed_verified
2024-10-17T11:24:37.248Z
64c8d4ac120a85440bfad55d
/avatars/d65334288d69700250de51d0df344d9b.svg
Fengji Zhang
user
zfj1998
67107f8dde5ba28c43c8c3fa
Linquan Wu
67107f8dde5ba28c43c8c3fb
Huiyu Bai
67107f8dde5ba28c43c8c3fc
Guancheng Lin
67107f8dde5ba28c43c8c3fd
Xiao Li
67107f8dde5ba28c43c8c3fe
Xiao Yu
67107f8dde5ba28c43c8c3ff
Yue Wang
67107f8dde5ba28c43c8c400
Bei Chen
67107f8dde5ba28c43c8c401
Jacky Keung
HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex
  Diagrams in Coding Tasks
Understanding and reasoning over diagrams is a fundamental aspect of human
intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive
capabilities across various tasks, existing benchmarks lack comprehensive
evaluation of their diagram interpretation and reasoning abilities,
particularly in coding contexts. We present HumanEval-V, a rigorous benchmark
of human-annotated coding tasks that spans six task types and evaluates diverse
visual reasoning capabilities. Each task features carefully crafted diagrams
paired with function signatures and test cases, employing novel code generation
tasks to thoroughly assess models' diagram comprehension. Through extensive
experiments with 22 LMMs, we find that even top-performing models achieve
modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1,
highlighting substantial room for improvement. Our analysis reveals that
current LMMs struggle with spatial transformations, topological relationships,
and dynamic patterns that humans find intuitive. These findings provide
valuable insights for advancing LMMs' visual reasoning abilities. We have
open-sourced our code and benchmark at
https://github.com/HumanEval-V/HumanEval-V-Benchmark.
HumanEval-V provides a comprehensive benchmark for evaluating LMMs' diagram interpretation and reasoning abilities, particularly in coding contexts, revealing significant areas for improvement.
Large Multimodal Models HumanEval-V diagram interpretation visual reasoning coding contexts function signatures test cases code generation spatial transformations topological relationships dynamic patterns