GP-VLS: A general-purpose vision language model for surgery
<|reference_start|>GP-VLS: A general-purpose vision language model for surgery: Surgery requires comprehensive medical knowledge, visual assessment skills, and procedural expertise. While recent surgical AI models have focused on solving task-specific problems, there is a need for general-purpose systems that can understand surgical scenes and interact through natural language. This paper introduces GP-VLS, a general-purpose vision language model for surgery that integrates medical and surgical knowledge with visual scene understanding. For comprehensively evaluating general-purpose surgical models, we propose SurgiQual, which evaluates across medical and surgical knowledge benchmarks as well as surgical vision-language questions. To train GP-VLS, we develop six new datasets spanning medical knowledge, surgical textbooks, and vision-language pairs for tasks like phase recognition and tool identification. We show that GP-VLS significantly outperforms existing open- and closed-source models on surgical vision-language tasks, with 8-21% improvements in accuracy across SurgiQual benchmarks. GP-VLS also demonstrates strong performance on medical and surgical knowledge tests compared to open-source alternatives. Overall, GP-VLS provides an open-source foundation for developing AI assistants to support surgeons across a wide range of tasks and scenarios. The code and data for this work is publicly available at gpvls-surgery-vlm.github.io.<|reference_end|>
@article{schmidgall2024gp-vls:,
    title={GP-VLS: A general-purpose vision language model for surgery},
    author={Samuel Schmidgall, Joseph Cho, Cyril Zakka, William Hiesinger},
    journal={arXiv preprint arXiv:2407.19305},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.19305},
    primaryClass={cs.CV cs.LG q-bio.TO}
}