MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness
  Against VLM-based Attacks
As automated attack techniques rapidly advance, CAPTCHAs remain a critical
defense mechanism against malicious bots. However, existing CAPTCHA schemes
encompass a diverse range of modalities -- from static distorted text and
obfuscated images to interactive clicks, sliding puzzles, and logic-based
questions -- yet the community still lacks a unified, large-scale, multimodal
benchmark to rigorously evaluate their security robustness. To address this
gap, we introduce MCA-Bench, a comprehensive and reproducible benchmarking
suite that integrates heterogeneous CAPTCHA types into a single evaluation
protocol. Leveraging a shared vision-language model backbone, we fine-tune
specialized cracking agents for each CAPTCHA category, enabling consistent,
cross-modal assessments. Extensive experiments reveal that MCA-Bench
effectively maps the vulnerability spectrum of modern CAPTCHA designs under
varied attack settings, and crucially offers the first quantitative analysis of
how challenge complexity, interaction depth, and model solvability interrelate.
Based on these findings, we propose three actionable design principles and
identify key open challenges, laying the groundwork for systematic CAPTCHA
hardening, fair benchmarking, and broader community collaboration. Datasets and
code are available online.
684b86913b733ba333686fb8
Zonglin Wu
684b86913b733ba333686fb9
Yule Xue
684b86913b733ba333686fba
Xin Wei
684b86913b733ba333686fbb
Yiren Song
MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness
  Against VLM-based Attacks
As automated attack techniques rapidly advance, CAPTCHAs remain a critical
defense mechanism against malicious bots. However, existing CAPTCHA schemes
encompass a diverse range of modalities -- from static distorted text and
obfuscated images to interactive clicks, sliding puzzles, and logic-based
questions -- yet the community still lacks a unified, large-scale, multimodal
benchmark to rigorously evaluate their security robustness. To address this
gap, we introduce MCA-Bench, a comprehensive and reproducible benchmarking
suite that integrates heterogeneous CAPTCHA types into a single evaluation
protocol. Leveraging a shared vision-language model backbone, we fine-tune
specialized cracking agents for each CAPTCHA category, enabling consistent,
cross-modal assessments. Extensive experiments reveal that MCA-Bench
effectively maps the vulnerability spectrum of modern CAPTCHA designs under
varied attack settings, and crucially offers the first quantitative analysis of
how challenge complexity, interaction depth, and model solvability interrelate.
Based on these findings, we propose three actionable design principles and
identify key open challenges, laying the groundwork for systematic CAPTCHA
hardening, fair benchmarking, and broader community collaboration. Datasets and
code are available online.
MCA-Bench provides a unified benchmark for evaluating CAPTCHA security using a shared vision-language model and attackers specialized for each type of CAPTCHA.
vision-language model CAPTCHA benchmark evaluation protocol cracking agents vulnerability spectrum challenge complexity interaction depth model solvability