Multimodal Auto Validation For Self-Refinement in Web Agents
<|reference_start|>Multimodal Auto Validation For Self-Refinement in Web Agents: As our world digitizes, web agents that can automate complex and monotonous tasks are becoming essential in streamlining workflows. This paper introduces an approach to improving web agent performance through multi-modal validation and self-refinement. We present a comprehensive study of different modalities (text, vision) and the effect of hierarchy for the automatic validation of web agents, building upon the state-of-the-art Agent-E web automation framework. We also introduce a self-refinement mechanism for web automation, using the developed auto-validator, that enables web agents to detect and self-correct workflow failures. Our results show significant gains on Agent-E's (a SOTA web agent) prior state-of-art performance, boosting task-completion rates from 76.2\% to 81.24\% on the subset of the WebVoyager benchmark. The approach presented in this paper paves the way for more reliable digital assistants in complex, real-world scenarios.<|reference_end|>
@article{azam2024multimodal,
    title={Multimodal Auto Validation For Self-Refinement in Web Agents},
    author={Ruhana Azam and Tamer Abuelsaad and Aditya Vempaty and Ashish Jagmohan},
    journal={arXiv preprint arXiv:2410.00689},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00689},
    primaryClass={cs.AI cs.SE}
}