  Major advances have been made regarding the utilization of artificial
intelligence in health care. In particular, deep learning approaches have been
successfully applied for automated and assisted disease diagnosis and prognosis
based on complex and high-dimensional data. However, despite all justified
enthusiasm, overoptimistic assessments of predictive performance are still
common. Automated medical testing devices based on machine-learned prediction
models should thus undergo a throughout evaluation before being implemented
into clinical practice. In this work, we propose a multiple testing framework
for (comparative) phase III diagnostic accuracy studies with sensitivity and
specificity as co-primary endpoints. Our approach challenges the frequent
recommendation to strictly separate model selection and evaluation, i.e. to
only assess a single diagnostic model in the evaluation study. We show that our
parametric simultaneous test procedure asymptotically allows strong control of
the family-wise error rate. Moreover, we demonstrate in extensive simulation
studies that our multiple testing strategy on average leads to a better final
diagnostic model and increased statistical power. To plan such studies, we
propose a Bayesian approach to determine the optimal number of models to
evaluate. For this purpose, our algorithm optimizes the expected final model
performance given previous (hold-out) data from the model development phase. We
conclude that an assessment of multiple promising diagnostic models in the same
evaluation study has several advantages when suitable adjustments for multiple
comparisons are implemented.

Methodology;Machine Learning