NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
  Brain MRI
In many real-world applications, deployed models encounter inputs that differ
from the data seen during training. Out-of-distribution detection identifies
whether an input stems from an unseen distribution, while open-world
recognition flags such inputs to ensure the system remains robust as
ever-emerging, previously unknown categories appear and must be addressed
without retraining. Foundation and vision-language models are pre-trained on
large and diverse datasets with the expectation of broad generalization across
domains, including medical imaging. However, benchmarking these models on test
sets with only a few common outlier types silently collapses the evaluation
back to a closed-set problem, masking failures on rare or truly novel
conditions encountered in clinical use.
  We therefore present NOVA, a challenging, real-life evaluation-only
benchmark of sim900 brain MRI scans that span 281 rare pathologies and
heterogeneous acquisition protocols. Each case includes rich clinical
narratives and double-blinded expert bounding-box annotations. Together, these
enable joint assessment of anomaly localisation, visual captioning, and
diagnostic reasoning. Because NOVA is never used for training, it serves as an
extreme stress-test of out-of-distribution generalisation: models must bridge
a distribution gap both in sample appearance and in semantic space. Baseline
results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
establishing NOVA as a rigorous testbed for advancing models that can detect,
localize, and reason about truly unknown anomalies.
682d92a2e1fd895b29ed2b46
Cosmin I. Bercea
extracted_pending
2025-05-21T08:45:26.580Z
6717cef5dca8fbc18c5c4216
/avatars/8c6e420bb4ece42b07e107ba3199fed2.svg
Cosmin
user
ci-ber
682d92a2e1fd895b29ed2b47
Jun Li
claimed_verified
2025-05-28T12:11:19.995Z
6526523fa754153bd4da45f3
/avatars/848b234fcd59d9885ddb673ffea20a63.svg
RioJune
user
RioJune
682d92a2e1fd895b29ed2b48
Philipp Raffler
682d92a2e1fd895b29ed2b49
Evamaria O. Riedel
682d92a2e1fd895b29ed2b4a
Lena Schmitzer
682d92a2e1fd895b29ed2b4b
Angela Kurz
682d92a2e1fd895b29ed2b4c
Felix Bitzer
682d92a2e1fd895b29ed2b4d
Paula Roßmüller
682d92a2e1fd895b29ed2b4e
Julian Canisius
682d92a2e1fd895b29ed2b4f
Mirjam L. Beyrle
682d92a2e1fd895b29ed2b50
Che Liu
682d92a2e1fd895b29ed2b51
Wenjia Bai
682d92a2e1fd895b29ed2b52
Bernhard Kainz
682d92a2e1fd895b29ed2b53
Julia A. Schnabel
682d92a2e1fd895b29ed2b54
Benedikt Wiestler
NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in
  Brain MRI
In many real-world applications, deployed models encounter inputs that differ
from the data seen during training. Out-of-distribution detection identifies
whether an input stems from an unseen distribution, while open-world
recognition flags such inputs to ensure the system remains robust as
ever-emerging, previously unknown categories appear and must be addressed
without retraining. Foundation and vision-language models are pre-trained on
large and diverse datasets with the expectation of broad generalization across
domains, including medical imaging. However, benchmarking these models on test
sets with only a few common outlier types silently collapses the evaluation
back to a closed-set problem, masking failures on rare or truly novel
conditions encountered in clinical use.
  We therefore present NOVA, a challenging, real-life evaluation-only
benchmark of sim900 brain MRI scans that span 281 rare pathologies and
heterogeneous acquisition protocols. Each case includes rich clinical
narratives and double-blinded expert bounding-box annotations. Together, these
enable joint assessment of anomaly localisation, visual captioning, and
diagnostic reasoning. Because NOVA is never used for training, it serves as an
extreme stress-test of out-of-distribution generalisation: models must bridge
a distribution gap both in sample appearance and in semantic space. Baseline
results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and
Qwen2.5-VL-72B) reveal substantial performance drops across all tasks,
establishing NOVA as a rigorous testbed for advancing models that can detect,
localize, and reason about truly unknown anomalies.
NOVA is a benchmark for evaluating vision-language models on rare, clinically relevant MRI pathologies, challenging their out-of-distribution and open-world recognition capabilities.
out-of-distribution detection open-world recognition foundation models vision-language models brain MRI scans rare pathologies heterogeneous acquisition protocols clinical narratives bounding-box annotations joint assessment anomaly localization visual captioning diagnostic reasoning extreme stress-test out-of-distribution generalisation distribution gap sample appearance semantic space