title: Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and
    Benchmarking Multimodal LLM Agents
summary: CAPTCHAs have been a critical bottleneck for deploying web agents in
  real-world applications, often blocking them from completing end-to-end
  automation tasks. While modern multimodal LLM agents have demonstrated
  impressive performance in static perception tasks, their ability to handle
  interactive, multi-step reasoning challenges like CAPTCHAs is largely untested.
  To address this gap, we introduce Open CaptchaWorld, the first web-based
  benchmark and platform specifically designed to evaluate the visual reasoning
  and interaction capabilities of MLLM-powered agents through diverse and dynamic
  CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225
  CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth,
  which quantifies the number of cognitive and motor steps required to solve each
  puzzle. Experimental results show that humans consistently achieve near-perfect
  scores, state-of-the-art MLLM agents struggle significantly, with success rates
  at most 40.0% by Browser-Use Openai-o3, far below human-level performance,
  93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing
  the limits of current multimodal agents and guiding the development of more
  robust multimodal reasoning systems. Code and Data are available at this https
  URL.
paper_authors._id: 683d160e51706d12b2c6f79f
paper_authors.name: Yaxin Luo
paper_authors._id: 683d160e51706d12b2c6f7a0
paper_authors.name: Zhaoyi Li
paper_authors._id: 683d160e51706d12b2c6f7a1
paper_authors.name: Jiacheng Liu
paper_authors._id: 683d160e51706d12b2c6f7a2
paper_authors.name: Jiacheng Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:40:52.498Z
paper_authors._id: 683d2ac900c71614bab8ea02
paper_authors.avatarUrl: /avatars/7cb1a5c2c778774262a7d7cb6d309abe.svg
paper_authors.fullname: Jiacheng Cui
paper_authors.type: user
paper_authors.user: jiachengcui888
paper_authors._id: 683d160e51706d12b2c6f7a3
paper_authors.name: Xiaohan Zhao
paper_authors._id: 683d160e51706d12b2c6f7a4
paper_authors.name: Zhiqiang Shen
paper_title: Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and
    Benchmarking Multimodal LLM Agents
paper_summary: CAPTCHAs have been a critical bottleneck for deploying web agents in
  real-world applications, often blocking them from completing end-to-end
  automation tasks. While modern multimodal LLM agents have demonstrated
  impressive performance in static perception tasks, their ability to handle
  interactive, multi-step reasoning challenges like CAPTCHAs is largely untested.
  To address this gap, we introduce Open CaptchaWorld, the first web-based
  benchmark and platform specifically designed to evaluate the visual reasoning
  and interaction capabilities of MLLM-powered agents through diverse and dynamic
  CAPTCHA puzzles. Our benchmark spans 20 modern CAPTCHA types, totaling 225
  CAPTCHAs, annotated with a new metric we propose: CAPTCHA Reasoning Depth,
  which quantifies the number of cognitive and motor steps required to solve each
  puzzle. Experimental results show that humans consistently achieve near-perfect
  scores, state-of-the-art MLLM agents struggle significantly, with success rates
  at most 40.0% by Browser-Use Openai-o3, far below human-level performance,
  93.3%. This highlights Open CaptchaWorld as a vital benchmark for diagnosing
  the limits of current multimodal agents and guiding the development of more
  robust multimodal reasoning systems. Code and Data are available at this https
  URL.
paper_ai_summary: Open CaptchaWorld benchmark evaluates MLLM-powered agents on diverse CAPTCHA puzzles, revealing significant performance gaps compared to humans.
paper_ai_keywords: multimodal LLM CAPTCHA visual reasoning interaction capabilities CAPTCHA Reasoning Depth Browser-Use Openai-o3