events.action: opened
events.author: Mxbonn
events.masked_author: username_0
events.text: By default COCO doesn't use a single IoU to calculate the mAP but instead averages over a range of IoUs (IoU=0.50:0.95).
  The evaluate function also outputs the mAP for IoU 0.5 and 0.75.
  However calculating the mAP for IoU 0.5 and 0.75 using the `evaluate` method from the `super` class `CustomDataset`  gives a different result.
  
  **This means that implementing a new dataset will give different evaluation results depending on whether it is implemented as a new subclass of `CustomDataset` or with COCO compatible labels as a `CocoDataset`.**
  
  Minimal example:
  ```python
  import os
  
  import mmdet.datasets
  import mmdet.apis
  import mmcv
  ```
  
  
  ```python
  checkpoint_path = "checkpoints/ssd300_coco_20200307-a92d2092.pth"
  config_path = "configs/ssd/ssd300_coco.py"
  
  cfg = mmcv.Config.fromfile(config_path)
  cfg.model.pretrained = None
  cfg.data.test.test_mode = True
  cfg.test_cfg.max_per_img = 100 # Set to 100 instead of 200 as CustomDataset.evaluate has no maxDets
  ```
  
  
  ```python
  test_dataset = mmdet.datasets.build_dataset(cfg.data.test)
  test_dataloader = mmdet.datasets.build_dataloader(
      test_dataset,
      samples_per_gpu=1,
      workers_per_gpu=cfg.data.workers_per_gpu,
      dist=False,
      shuffle=False
  )
  ```
  
      loading annotations into memory...
      Done (t=0.52s)
      creating index...
      index created!
  
  
  
  ```python
  raw_model = mmdet.models.build_detector(cfg.model, train_cfg=None, test_cfg=cfg.test_cfg)
  checkpoint = mmcv.runner.load_checkpoint(raw_model, checkpoint_path, map_location='cpu')
  raw_model.CLASSES = test_dataset.CLASSES
  model = mmcv.parallel.MMDataParallel(raw_model, device_ids=[0])
  ```
  
  
  ```python
  results = mmdet.apis.single_gpu_test(model, test_dataloader)
  ```
  
      [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 39.0 task/s, elapsed: 128s, ETA:     0s
  
  
  ```python
  test_dataset.evaluate(results)
  ```
  
  
      Evaluating bbox...
      Loading and preparing results...
      DONE (t=3.77s)
      creating index...
      index created!
      Running per image evaluation...
      Evaluate annotation type *bbox*
      DONE (t=53.15s).
      Accumulating evaluation results...
      DONE (t=7.20s).
       Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.255
       Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.436
       Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.263
       Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.065
       Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.277
       Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.422
       Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.237
       Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.347
       Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.364
       Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.108
       Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.401
       Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.582
  
  
  
  
  
      {'bbox_mAP': 0.255,
       'bbox_mAP_50': 0.436,
       'bbox_mAP_75': 0.263,
       'bbox_mAP_s': 0.065,
       'bbox_mAP_m': 0.277,
       'bbox_mAP_l': 0.422,
       'bbox_mAP_copypaste': '0.255 0.436 0.263 0.065 0.277 0.422'}
  
  
  
  
  ```python
  super(type(test_dataset), test_dataset).evaluate(results, iou_thr=0.5)
  ```
  
  
      +----------------+-------+--------+--------+-------+
      | class          | gts   | dets   | recall | ap    |
      +----------------+-------+--------+--------+-------+
      | person         | 10777 | 151136 | 0.781  | 0.619 |
      | bicycle        | 314   | 1935   | 0.503  | 0.376 |
      | car            | 1918  | 17858  | 0.626  | 0.440 |
      | motorcycle     | 367   | 2282   | 0.703  | 0.571 |
      | airplane       | 143   | 3942   | 0.895  | 0.737 |
      | bus            | 283   | 875    | 0.742  | 0.693 |
      | train          | 190   | 4048   | 0.863  | 0.795 |
      | truck          | 414   | 2610   | 0.693  | 0.431 |
      | boat           | 424   | 8890   | 0.547  | 0.308 |
      | traffic light  | 634   | 12564  | 0.465  | 0.236 |
      | fire hydrant   | 101   | 274    | 0.703  | 0.662 |
      | stop sign      | 75    | 479    | 0.640  | 0.581 |
      | parking meter  | 60    | 147    | 0.583  | 0.498 |
      | bench          | 411   | 12074  | 0.457  | 0.267 |
      | bird           | 427   | 13891  | 0.433  | 0.296 |
      | cat            | 202   | 1221   | 0.931  | 0.851 |
      | dog            | 218   | 1027   | 0.821  | 0.713 |
      | horse          | 272   | 2077   | 0.794  | 0.671 |
      | sheep          | 354   | 5733   | 0.782  | 0.559 |
      | cow            | 372   | 4083   | 0.766  | 0.534 |
      | elephant       | 252   | 3908   | 0.897  | 0.738 |
      | bear           | 71    | 308    | 0.831  | 0.777 |
      | zebra          | 266   | 5540   | 0.880  | 0.797 |
      | giraffe        | 232   | 5620   | 0.901  | 0.812 |
      | backpack       | 371   | 2462   | 0.305  | 0.111 |
      | umbrella       | 407   | 4079   | 0.634  | 0.462 |
      | handbag        | 540   | 4092   | 0.274  | 0.079 |
      | tie            | 252   | 4043   | 0.480  | 0.314 |
      | suitcase       | 299   | 1934   | 0.515  | 0.334 |
      | frisbee        | 115   | 566    | 0.687  | 0.499 |
      | skis           | 240   | 5994   | 0.592  | 0.301 |
      | snowboard      | 69    | 343    | 0.478  | 0.311 |
      | sports ball    | 260   | 4571   | 0.504  | 0.382 |
      | kite           | 327   | 7137   | 0.648  | 0.385 |
      | baseball bat   | 145   | 1113   | 0.441  | 0.283 |
      | baseball glove | 148   | 1161   | 0.480  | 0.296 |
      | skateboard     | 179   | 1872   | 0.721  | 0.559 |
      | surfboard      | 267   | 4453   | 0.592  | 0.413 |
      | tennis racket  | 225   | 2015   | 0.631  | 0.524 |
      | bottle         | 1013  | 7547   | 0.523  | 0.277 |
      | wine glass     | 341   | 1171   | 0.408  | 0.290 |
      | cup            | 895   | 4976   | 0.534  | 0.330 |
      | fork           | 215   | 1029   | 0.414  | 0.228 |
      | knife          | 325   | 1714   | 0.237  | 0.065 |
      | spoon          | 253   | 1237   | 0.249  | 0.088 |
      | bowl           | 623   | 3130   | 0.652  | 0.431 |
      | banana         | 370   | 7206   | 0.635  | 0.300 |
      | apple          | 236   | 2028   | 0.424  | 0.154 |
      | sandwich       | 177   | 1156   | 0.678  | 0.462 |
      | orange         | 285   | 2710   | 0.561  | 0.285 |
      | broccoli       | 312   | 13007  | 0.769  | 0.325 |
      | carrot         | 365   | 9667   | 0.573  | 0.193 |
      | hot dog        | 125   | 470    | 0.480  | 0.363 |
      | pizza          | 284   | 2390   | 0.736  | 0.600 |
      | donut          | 328   | 2525   | 0.649  | 0.389 |
      | cake           | 310   | 1742   | 0.626  | 0.412 |
      | chair          | 1771  | 33923  | 0.537  | 0.279 |
      | couch          | 261   | 1659   | 0.785  | 0.590 |
      | potted plant   | 342   | 12017  | 0.661  | 0.308 |
      | bed            | 163   | 1625   | 0.785  | 0.632 |
      | dining table   | 695   | 8174   | 0.662  | 0.381 |
      | toilet         | 179   | 3298   | 0.866  | 0.730 |
      | tv             | 288   | 2276   | 0.795  | 0.671 |
      | laptop         | 231   | 913    | 0.740  | 0.647 |
      | mouse          | 106   | 627    | 0.708  | 0.557 |
      | remote         | 283   | 2146   | 0.378  | 0.138 |
      | keyboard       | 153   | 1376   | 0.784  | 0.624 |
      | cell phone     | 262   | 2546   | 0.462  | 0.287 |
      | microwave      | 55    | 366    | 0.727  | 0.562 |
      | oven           | 143   | 3618   | 0.783  | 0.523 |
      | toaster        | 9     | 16     | 0.111  | 0.007 |
      | sink           | 225   | 8280   | 0.702  | 0.447 |
      | refrigerator   | 126   | 2632   | 0.794  | 0.599 |
      | book           | 1129  | 21431  | 0.417  | 0.100 |
      | clock          | 267   | 4601   | 0.674  | 0.562 |
      | vase           | 274   | 1612   | 0.504  | 0.315 |
      | scissors       | 36    | 256    | 0.472  | 0.379 |
      | teddy bear     | 190   | 2154   | 0.674  | 0.535 |
      | hair drier     | 11    | 1      | 0.000  | 0.000 |
      | toothbrush     | 57    | 326    | 0.263  | 0.146 |
      +----------------+-------+--------+--------+-------+
      | mAP            |       |        |        | 0.430 |
      +----------------+-------+--------+--------+-------+
  
  
  
  
  
      {'mAP': 0.43032437562942505}
  
  
  
  
  ```python
  super(type(test_dataset), test_dataset).evaluate(results, iou_thr=0.75)
  ```
  
  
      +----------------+-------+--------+--------+-------+
      | class          | gts   | dets   | recall | ap    |
      +----------------+-------+--------+--------+-------+
      | person         | 10777 | 151136 | 0.424  | 0.322 |
      | bicycle        | 314   | 1935   | 0.220  | 0.141 |
      | car            | 1918  | 17858  | 0.301  | 0.209 |
      | motorcycle     | 367   | 2282   | 0.351  | 0.275 |
      | airplane       | 143   | 3942   | 0.664  | 0.538 |
      | bus            | 283   | 875    | 0.625  | 0.589 |
      | train          | 190   | 4048   | 0.679  | 0.623 |
      | truck          | 414   | 2610   | 0.425  | 0.236 |
      | boat           | 424   | 8890   | 0.200  | 0.094 |
      | traffic light  | 634   | 12564  | 0.129  | 0.056 |
      | fire hydrant   | 101   | 274    | 0.594  | 0.532 |
      | stop sign      | 75    | 479    | 0.600  | 0.567 |
      | parking meter  | 60    | 147    | 0.433  | 0.373 |
      | bench          | 411   | 12074  | 0.221  | 0.138 |
      | bird           | 427   | 13891  | 0.267  | 0.194 |
      | cat            | 202   | 1221   | 0.757  | 0.617 |
      | dog            | 218   | 1027   | 0.651  | 0.529 |
      | horse          | 272   | 2077   | 0.540  | 0.418 |
      | sheep          | 354   | 5733   | 0.432  | 0.290 |
      | cow            | 372   | 4083   | 0.462  | 0.333 |
      | elephant       | 252   | 3908   | 0.643  | 0.511 |
      | bear           | 71    | 308    | 0.718  | 0.664 |
      | zebra          | 266   | 5540   | 0.624  | 0.552 |
      | giraffe        | 232   | 5620   | 0.668  | 0.582 |
      | backpack       | 371   | 2462   | 0.127  | 0.029 |
      | umbrella       | 407   | 4079   | 0.327  | 0.218 |
      | handbag        | 540   | 4092   | 0.093  | 0.019 |
      | tie            | 252   | 4043   | 0.183  | 0.118 |
      | suitcase       | 299   | 1934   | 0.261  | 0.143 |
      | frisbee        | 115   | 566    | 0.539  | 0.393 |
      | skis           | 240   | 5994   | 0.133  | 0.046 |
      | snowboard      | 69    | 343    | 0.217  | 0.134 |
      | sports ball    | 260   | 4571   | 0.342  | 0.255 |
      | kite           | 327   | 7137   | 0.321  | 0.175 |
      | baseball bat   | 145   | 1113   | 0.172  | 0.067 |
      | baseball glove | 148   | 1161   | 0.257  | 0.117 |
      | skateboard     | 179   | 1872   | 0.358  | 0.237 |
      | surfboard      | 267   | 4453   | 0.270  | 0.156 |
      | tennis racket  | 225   | 2015   | 0.293  | 0.183 |
      | bottle         | 1013  | 7547   | 0.247  | 0.116 |
      | wine glass     | 341   | 1171   | 0.194  | 0.115 |
      | cup            | 895   | 4976   | 0.345  | 0.214 |
      | fork           | 215   | 1029   | 0.172  | 0.080 |
      | knife          | 325   | 1714   | 0.098  | 0.029 |
      | spoon          | 253   | 1237   | 0.119  | 0.046 |
      | bowl           | 623   | 3130   | 0.456  | 0.293 |
      | banana         | 370   | 7206   | 0.265  | 0.117 |
      | apple          | 236   | 2028   | 0.250  | 0.094 |
      | sandwich       | 177   | 1156   | 0.486  | 0.318 |
      | orange         | 285   | 2710   | 0.411  | 0.210 |
      | broccoli       | 312   | 13007  | 0.308  | 0.108 |
      | carrot         | 365   | 9667   | 0.216  | 0.064 |
      | hot dog        | 125   | 470    | 0.304  | 0.233 |
      | pizza          | 284   | 2390   | 0.521  | 0.434 |
      | donut          | 328   | 2525   | 0.463  | 0.293 |
      | cake           | 310   | 1742   | 0.374  | 0.199 |
      | chair          | 1771  | 33923  | 0.244  | 0.126 |
      | couch          | 261   | 1659   | 0.521  | 0.321 |
      | potted plant   | 342   | 12017  | 0.266  | 0.102 |
      | bed            | 163   | 1625   | 0.534  | 0.405 |
      | dining table   | 695   | 8174   | 0.397  | 0.230 |
      | toilet         | 179   | 3298   | 0.659  | 0.549 |
      | tv             | 288   | 2276   | 0.635  | 0.507 |
      | laptop         | 231   | 913    | 0.576  | 0.499 |
      | mouse          | 106   | 627    | 0.557  | 0.459 |
      | remote         | 283   | 2146   | 0.166  | 0.073 |
      | keyboard       | 153   | 1376   | 0.484  | 0.333 |
      | cell phone     | 262   | 2546   | 0.282  | 0.177 |
      | microwave      | 55    | 366    | 0.545  | 0.438 |
      | oven           | 143   | 3618   | 0.413  | 0.277 |
      | toaster        | 9     | 16     | 0.111  | 0.007 |
      | sink           | 225   | 8280   | 0.351  | 0.179 |
      | refrigerator   | 126   | 2632   | 0.556  | 0.424 |
      | book           | 1129  | 21431  | 0.117  | 0.024 |
      | clock          | 267   | 4601   | 0.468  | 0.339 |
      | vase           | 274   | 1612   | 0.270  | 0.167 |
      | scissors       | 36    | 256    | 0.250  | 0.189 |
      | teddy bear     | 190   | 2154   | 0.374  | 0.240 |
      | hair drier     | 11    | 1      | 0.000  | 0.000 |
      | toothbrush     | 57    | 326    | 0.070  | 0.040 |
      +----------------+-------+--------+--------+-------+
      | mAP            |       |        |        | 0.259 |
      +----------------+-------+--------+--------+-------+
  
  
      {'mAP': 0.2592655122280121}
events.title: COCO mAP and CustomDataset mAP don't match on single IoU value.
events.type: issue
events.action: created
events.author: ZwwWayne
events.masked_author: username_1
events.text: The `customdataset` uses VOC-style AP calculation, its implementation when calculating TP and FPs are a little bit different from COCO evaluation.
events.type: comment
events.action: created
events.author: hellock
events.masked_author: username_2
events.text: @username_0 If you use the official evaluation tool provided by COCO to evaluate the results of Pascal VOC dataset or ImageNet DET dataset, you will also get slightly different mAPs. The implementation of evaluation metrics of different datasets are not exactly the same.
events.type: comment
events.action: closed
events.author: hellock
events.masked_author: username_2
events.type: issue