How to calculate mAP for detection task for the PASCAL VOC Challenge?
How to calculate the mAP (mean Average Precision) for the detection task for the Pascal VOC leaderboards? http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?challengeid=11&compid=4
There said - at page 11: http://homepages.inf.ed.ac.uk/ckiw/postscript/ijcv_voc09.pdf
Average Precision (AP). For the VOC2007 challenge, the interpolated average precision (<PERSON> and <PERSON> 1986) was used to evaluate both classification and detection. For a given task and class, the precision/recall curve is computed from a method’s ranked output. Recall is defined as the proportion of all positive examples ranked above a given rank. Precision is the proportion of all examples above that rank which are from the positive class. The AP summarises the shape of the precision/recall curve, and is defined as the mean precision at a set of eleven equally spaced recall levels [0,0.1,...,1]: AP = 1/11 ∑ r∈{0,0.1,...,1} pinterp(r)
The precision at each recall level r is interpolated by taking the maximum precision measured for a method for which the corresponding recall exceeds r: pinterp(r) = max p(r˜), where p(r˜) is the measured precision at recall ˜r
About mAP: http://0agr.ru/wiki/index.php/Precision_and_Recall#Average_Precision
So does it mean that:
1. We calculate Precision and Recall:
2. 506
A) For many different IoU > {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1} we calculate True/False Positive/Negative values
Where True positive = Number_of_detection with IoU > {0, 0.1,..., 1}, as said here: https://datascience.stackexchange.com/a/<PHONE_NUMBER> and then we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)
* B) Or for many different thresholds of detection algorithms we calculate:
Precision = True positive / (True positive + False positive)
Recall = True positive / (True positive + False negative)
Where True positive = Number_of_detection with IoU > 0.5 as said here: https://stackoverflow.com/a/43168882/1558037
* C) Or for many different thresholds of detection algorithms we calculate:
Precision = Intersect / Detected_box
Recall = Intersect / Object
As shown here? https://en.wikipedia.org/wiki/Precision_and_recall
1. Then we build Precision-Recall curve, as shown here:
1. Then we calculate AP (average precision) as average of 11 values of Precision at the points where Recall = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, i.e. AP = 1/11 ∑ recall∈{0,0.1,...,1} Precision(Recall)
(In general for each point, for example 0.3, we get MAX of Precision for Recall <= 0.3, instead of value of Precision at this point Recall=0.3)
1. And when we calculate AP only for 1 something object class on all images - then we get AP (average precision) for this class, for example, only for air.
So AP is a integral (area under the curve): https://stats.stackexchange.com/a/157019/111998
But when we calculate AP for all object classes on all images - then we get mAP (mean average precision) for all images dataset, for example, 88.6 for R4D_faster_rcnn: http://host.robots.ox.ac.uk:8080/leaderboard/displaylb.php?challengeid=11&compid=4
Questions:
1. Is it right, and if it isn't, then how to calculate mAP for Pascal VOC Challenge?
2. And which of the 3 formulas (A, B or C) is correct for calculating Precision and Recall, in paragraph 1? 506