Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and
  Benchmarking Multimodal LLM Agents
CAPTCHAs have been a critical bottleneck for deploying web agents in
real-world applications, often blocking them from completing end-to-end
automation tasks. While modern multimodal LLM agents have demonstrated
impressive performance in static perception tasks, their ability to handle
interactive, multi-step reasoning challenges like CAPTCHAs is largely untested.
To address this gap, we introduce Open CaptchaWorld, the first web-based
benchmark and platform specifically designed to evaluate the visual reasoning
and interaction capabilities of MLLM-powered agents through diverse and dynamic
CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225
CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth,
which quantifies the number of cognitive and motor steps required to solve each
puzzle. Experimental results show that humans consistently achieve near-perfect
scores, state-of-the-art MLLM agents struggle significantly, with success rates
at most 40.0% by Browser-Use Openai-o3, far below human-level performance,
93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing
the limits of current multimodal agents and guiding the development of more
robust multimodal reasoning systems. Code and Data are available at this https
URL.
683d160e51706d12b2c6f79f
Yaxin Luo
683d160e51706d12b2c6f7a0
Zhaoyi Li
683d160e51706d12b2c6f7a1
Jiacheng Liu
683d160e51706d12b2c6f7a2
Jiacheng Cui
claimed_verified
2025-06-02T07:40:52.498Z
683d2ac900c71614bab8ea02
/avatars/7cb1a5c2c778774262a7d7cb6d309abe.svg
Jiacheng Cui
user
jiachengcui888
683d160e51706d12b2c6f7a3
Xiaohan Zhao
683d160e51706d12b2c6f7a4
Zhiqiang Shen
Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and
  Benchmarking Multimodal LLM Agents
CAPTCHAs have been a critical bottleneck for deploying web agents in
real-world applications, often blocking them from completing end-to-end
automation tasks. While modern multimodal LLM agents have demonstrated
impressive performance in static perception tasks, their ability to handle
interactive, multi-step reasoning challenges like CAPTCHAs is largely untested.
To address this gap, we introduce Open CaptchaWorld, the first web-based
benchmark and platform specifically designed to evaluate the visual reasoning
and interaction capabilities of MLLM-powered agents through diverse and dynamic
CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225
CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth,
which quantifies the number of cognitive and motor steps required to solve each
puzzle. Experimental results show that humans consistently achieve near-perfect
scores, state-of-the-art MLLM agents struggle significantly, with success rates
at most 40.0% by Browser-Use Openai-o3, far below human-level performance,
93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing
the limits of current multimodal agents and guiding the development of more
robust multimodal reasoning systems. Code and Data are available at this https
URL.
Open CaptchaWorld benchmark evaluates MLLM-powered agents on diverse CAPTCHA puzzles, revealing significant performance gaps compared to humans.
multimodal LLM CAPTCHA visual reasoning interaction capabilities CAPTCHA Reasoning Depth Browser-Use Openai-o3