title: NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
    Brain MRI
summary: In many real-world applications, deployed models encounter inputs that differ
  from the data seen during training. Out-of-distribution detection identifies
  whether an input stems from an unseen distribution, while open-world
  recognition flags such inputs to ensure the system remains robust as
  ever-emerging, previously unknown categories appear and must be addressed
  without retraining. Foundation and vision-language models are pre-trained on
  large and diverse datasets with the expectation of broad generalization across
  domains, including medical imaging. However, benchmarking these models on test
  sets with only a few common outlier types silently collapses the evaluation
  back to a closed-set problem, masking failures on rare or truly novel
  conditions encountered in clinical use.
    We therefore present NOVA, a challenging, real-life evaluation-only
  benchmark of sim900 brain MRI scans that span 281 rare pathologies and
  heterogeneous acquisition protocols. Each case includes rich clinical
  narratives and double-blinded expert bounding-box annotations. Together, these
  enable joint assessment of anomaly localisation, visual captioning, and
  diagnostic reasoning. Because NOVA is never used for training, it serves as an
  extreme stress-test of out-of-distribution generalisation: models must bridge
  a distribution gap both in sample appearance and in semantic space. Baseline
  results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
  Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
  establishing NOVA as a rigorous testbed for advancing models that can detect,
  localize, and reason about truly unknown anomalies.
paper_authors._id: 682d92a2e1fd895b29ed2b46
paper_authors.name: Cosmin I. Bercea
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-21T08:45:26.580Z
paper_authors._id: 6717cef5dca8fbc18c5c4216
paper_authors.avatarUrl: /avatars/8c6e420bb4ece42b07e107ba3199fed2.svg
paper_authors.fullname: Cosmin
paper_authors.type: user
paper_authors.user: ci-ber
paper_authors._id: 682d92a2e1fd895b29ed2b47
paper_authors.name: Jun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T12:11:19.995Z
paper_authors._id: 6526523fa754153bd4da45f3
paper_authors.avatarUrl: /avatars/848b234fcd59d9885ddb673ffea20a63.svg
paper_authors.fullname: RioJune
paper_authors.type: user
paper_authors.user: RioJune
paper_authors._id: 682d92a2e1fd895b29ed2b48
paper_authors.name: Philipp Raffler
paper_authors._id: 682d92a2e1fd895b29ed2b49
paper_authors.name: Evamaria O. Riedel
paper_authors._id: 682d92a2e1fd895b29ed2b4a
paper_authors.name: Lena Schmitzer
paper_authors._id: 682d92a2e1fd895b29ed2b4b
paper_authors.name: Angela Kurz
paper_authors._id: 682d92a2e1fd895b29ed2b4c
paper_authors.name: Felix Bitzer
paper_authors._id: 682d92a2e1fd895b29ed2b4d
paper_authors.name: Paula Roßmüller
paper_authors._id: 682d92a2e1fd895b29ed2b4e
paper_authors.name: Julian Canisius
paper_authors._id: 682d92a2e1fd895b29ed2b4f
paper_authors.name: Mirjam L. Beyrle
paper_authors._id: 682d92a2e1fd895b29ed2b50
paper_authors.name: Che Liu
paper_authors._id: 682d92a2e1fd895b29ed2b51
paper_authors.name: Wenjia Bai
paper_authors._id: 682d92a2e1fd895b29ed2b52
paper_authors.name: Bernhard Kainz
paper_authors._id: 682d92a2e1fd895b29ed2b53
paper_authors.name: Julia A. Schnabel
paper_authors._id: 682d92a2e1fd895b29ed2b54
paper_authors.name: Benedikt Wiestler
paper_title: NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
    Brain MRI
paper_summary: In many real-world applications, deployed models encounter inputs that differ
  from the data seen during training. Out-of-distribution detection identifies
  whether an input stems from an unseen distribution, while open-world
  recognition flags such inputs to ensure the system remains robust as
  ever-emerging, previously unknown categories appear and must be addressed
  without retraining. Foundation and vision-language models are pre-trained on
  large and diverse datasets with the expectation of broad generalization across
  domains, including medical imaging. However, benchmarking these models on test
  sets with only a few common outlier types silently collapses the evaluation
  back to a closed-set problem, masking failures on rare or truly novel
  conditions encountered in clinical use.
    We therefore present NOVA, a challenging, real-life evaluation-only
  benchmark of sim900 brain MRI scans that span 281 rare pathologies and
  heterogeneous acquisition protocols. Each case includes rich clinical
  narratives and double-blinded expert bounding-box annotations. Together, these
  enable joint assessment of anomaly localisation, visual captioning, and
  diagnostic reasoning. Because NOVA is never used for training, it serves as an
  extreme stress-test of out-of-distribution generalisation: models must bridge
  a distribution gap both in sample appearance and in semantic space. Baseline
  results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
  Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
  establishing NOVA as a rigorous testbed for advancing models that can detect,
  localize, and reason about truly unknown anomalies.
paper_ai_summary: NOVA is a benchmark for evaluating vision-language models on rare, clinically relevant MRI pathologies, challenging their out-of-distribution and open-world recognition capabilities.
paper_ai_keywords: out-of-distribution detection open-world recognition foundation models vision-language models brain MRI scans rare pathologies heterogeneous acquisition protocols clinical narratives bounding-box annotations joint assessment anomaly localization visual captioning diagnostic reasoning extreme stress-test out-of-distribution generalisation distribution gap sample appearance semantic space