Precision-Recall-Gain Curves: PR Analysis Done Right
Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracybased performance assessment, many researchers have taken to report PrecisionRecall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of incoherent scale assumptions -e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the F β score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves conveys an expected F 1 score on a harmonic scale, and the convex hull of a PrecisionRecall-Gain curve allows us to calibrate the classifier's scores so as to determine, for each operating point on the convex hull, the interval of β values for which the point optimises F β . We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected F 1 score than others, and so the use of Precision-Recall-Gain curves will result in better model selection.
These authors demonstrate that traditional Precision-Recall (PR) curves and associated area under PR curve are fraught with difficulties.They show how to fix this by plotting PR curves in a different coordinate system, and propose Precision-Recall-Gain curves which inherit all key advantages of ROC curves. This study is of interest to the readership and suitable for the NIPS audience. This innovative study will advance the field of performance assessment for predictive models and diagnostic tests. However, several problems are noted in the methods and clarity of presentation. a. At line 390, what are AUROCs for IBK and Logistic Regression? b. It is difficult to interpret the information embedded in Figure 5.I would recommend using a table to report the outcomes for five Weka models trained on 19 UCI datasets, and using Figure 5 to display performance for several examples including the mushroom dataset.c. For Figure 5, it is better to avoid abbreviations and to provide detailed explanation so that it can stand alone. It is also better to provide the definition for IBK since it isn't a widely used model.