VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
<|reference_start|>VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?: Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce \bench{}, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. \bench{} consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 14 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on \bench{}, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe \bench{} will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.<|reference_end|>
@article{liu2024visualwebbench:,
    title={VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page
  Understanding and Grounding?},
    author={Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig,
  Yuanzhi Li, and Xiang Yue},
    journal={arXiv preprint arXiv:2404.05955},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.05955},
    primaryClass={cs.CL cs.AI}
}