Testing for statistical significance of the true positive detection rate between different machine learning models Background
Currently I am working on true positive detection for an image analysis problem. I have 4 methods and would like to test which methods differ from each other.
Description of Data
For each patient I have the number of false positives and true positives per method.
What have I tried?
I have looked through Andy Field's Discovering statistics using SPSS. According to this book I would have to run a one way repeated measures ANOVA. However, this would tell me that the group means differ significantly, but not what post hoc test to run.
Questions

*

*What test would be best suited for this problem?

*Given the right test, what assumptions would I have to test for?


If all you care about is the proportion of binary "true positives" estimated by 4 different methods, then you have a pretty simply logistic regression model.* The binary outcome for the regression would be success/failure, with a "true positive" being a success and all other results being failures.
The 4 methods would be treated as 4 levels of a single categorical predictor variable. As the different methods evidently evaluate all the same images/patients, you need to take the repeated measures on the same images/patients into account. Treating the images/patients as random effects within a mixed-effect model would be a way to handle that. That's analogous to the "one way repeated measures ANOVA" that you mention, but uses an analysis that's suited to binary yes/no outcomes. The links about "mixed effect models" on this UCLA web page can provide guidance.
If the model indicates a significant association of your categorical method predictor with outcome, then you have evidence that the methods do differ. A simple post-hoc comparison would be to do all 6 pairwise comparisons between the 4 methods with Wald tests, then do a correction for the multiple comparisons.
That type of analysis should be possible with standard software packages. Otherwise, the 3 comparisons of other methods against the reference level of method would typically be reported directly in the summary output of the model. Comparisons of the non-reference levels against each other would use the modeled coefficient covariance matrix together with the formula for the variance of a weighted sum.

*As indicated in my comment, I'm not so sure that a sole focus on true positives is wise. This is how to proceed if you are nevertheless convinced that it is.
