In-context learning enables multimodal large language models to classify cancer pathology images
<|reference_start|>In-context learning enables multimodal large language models to classify cancer pathology images: Medical image classification requires labeled, task-specific datasets which are used to train deep learning networks de novo, or to fine-tune foundation models. However, this process is computationally and technically demanding. In language processing, in-context learning provides an alternative, where models learn from within prompts, bypassing the need for parameter updates. Yet, in-context learning remains underexplored in medical image analysis. Here, we systematically evaluate the model Generative Pretrained Transformer 4 with Vision capabilities (GPT-4V) on cancer image processing with in-context learning on three cancer histopathology tasks of high importance: Classification of tissue subtypes in colorectal cancer, colon polyp subtyping and breast tumor detection in lymph node sections. Our results show that in-context learning is sufficient to match or even outperform specialized neural networks trained for particular tasks, while only requiring a minimal number of samples. In summary, this study demonstrates that large vision language models trained on non-domain specific data can be applied out-of-the box to solve medical image-processing tasks in histopathology. This democratizes access of generalist AI models to medical experts without technical background especially for areas where annotated data is scarce.<|reference_end|>
@article{ferber2024in-context,
    title={In-context learning enables multimodal large language models to classify
  cancer pathology images},
    author={Dyke Ferber, Georg W"olflein, Isabella C. Wiest, Marta Ligero,
  Srividhya Sainath, Narmin Ghaffari Laleh, Omar S.M. El Nahhas, Gustav
  M"uller-Franzes, Dirk J"ager, Daniel Truhn, Jakob Nikolas Kather},
    journal={arXiv preprint arXiv:2403.07407},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.07407},
    primaryClass={cs.CV}
}