HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks.
Abstract
Understanding and reasoning over diagrams is a fundamental aspect of human intelligence. While Large Multimodal Models (LMMs) have demonstrated impressive capabilities across various tasks, existing benchmarks lack comprehensive evaluation of their diagram interpretation and reasoning abilities, particularly in coding contexts. We present HumanEval-V, a rigorous benchmark of human-annotated coding tasks that spans six task types and evaluates diverse visual reasoning capabilities. Each task features carefully crafted diagrams paired with function signatures and test cases, employing novel code generation tasks to thoroughly assess models' diagram comprehension. Through extensive experiments with 22 LMMs, we find that even top-performing models achieve modest success rates, with Claude 3.5 Sonnet reaching only 36.8% pass@1, highlighting substantial room for improvement. Our analysis reveals that current LMMs struggle with spatial transformations, topological relationships, and dynamic patterns that humans find intuitive. These findings provide valuable insights for advancing LMMs' visual reasoning abilities. We have open-sourced our code and benchmark at https://github.com/ HumanEval-V/HumanEval-V-Benchmark. 1 Figure 1: A task example from HumanEval-V. LMMs are required to figure out the facts and patterns in the diagram and complete the function body. simple pattern recognition; they necessitate sophisticated cognitive capabilities, including interpreting transformation patterns, recognizing hierarchical structures, and integrating multiple visual cues such as arrows, symbols, and their relative positions to perform spatial or logical reasoning. The rapid development of Large Multimodal Models (LMMs) has led to the creation of various benchmarks designed to assess the alignment between LMMs' capabilities and human intelligence. However, there remains a significant gap in benchmarks that specifically evaluate the ability to understand and reason over complex diagrams. Popular multimodal benchmarks, such as MMMU (Yue et al., 2024), MathVista (Lu et al., 2023), and ChartQA (Masry et al., 2022), focus primarily on scientific, mathematical, and chartbased analytic