HumanEval-V: Systematic Evaluation of Visual Reasoning in Large Multimodal Models for Code Generation
Abstract
reasoning, they overlook the coding domain, where visual reasoning is deeply intertwined with algorithmic thinking and code generation. Within the multimodal programming domain, tasks such as Plot2Code [79] and Design2Code [62] have emerged, but these primarily focus on translating visual layouts into code, with minimal emphasis on algorithmic reasoning. A more relevant effort is the MMCode benchmark [45], which compiles coding problems accompanied by visual demonstrations from competitive programming platforms to evaluate LMMs on code generation tasks. However, MMCode exhibits notable limitations in assessing visual reasoning capabilities. First, it does not clearly isolate the contribution of visual information to problem-solving. In many cases, the visual content is replicated in the accompanying textual description, rendering the visual modality redundant. Second, the benchmark fails to disentangle whether the primary challenge of a problem stems from its algorithmic complexity or the visual reasoning required. Consequently, when LMMs fail to produce correct code, it remains unclear whether the cause is inadequate visual reasoning or the intrinsic difficulty of the coding task. Collectively, these limitations underscore persistent gap in effectively evaluating the visual reasoning capabilities of LMMs in programming contexts. ACM Trans. Softw. Eng. Methodol.