Answers.Body: <p>If you want to evaluate on real thermal image dataset, you can use this one.</p>
  
  <p><a href="https://www.flir.com/oem/adas/dataset/" rel="nofollow noreferrer">Thermal Image dataset</a></p>
  
  <p>is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")</p>
  
  <blockquote>
    <p>Mean Average Precision is the relevant metric but it's more technical. You can start explaining with False Positives and False 
    Negatives in the predictions. In turn, it leads to Precision and Recall. It mostly depends upon your use case because there will always be a trade-off between them.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T13:33:56.637Z
Answers.Text: Ty for your answer on the mAP but you did not answer my main question on "how to build a good evaluation/test set ?".
Answers.UserDisplayName: antoine Mathu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T15:04:53.607Z
Answers.Text: At the beginning itself, we have to split the data as train/val/test. It depends upon how much data you have for your task to train the model. The test set should be taken from almost the same data distribution of the training set. If we train a model with only daylight images and test with night conditions, it is a bad idea to judge the model.
Answers.UserDisplayName: Sudhakar17
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T15:40:15.223Z
Answers.LastActivityDate: 2020-02-28T15:40:15.223Z
Answers.UserDisplayName: Sudhakar17
Body: <p>I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.</p>
  
  <p>Even I have more than 2k different poses, but <strong>the background does not change much</strong> (I have only 5 different points of view).</p>
  
  <p>I trained my model on these images (70% for the training and 30% for validation).</p>
  
  <p><strong>Now, I want to evaluate</strong> the model on an unbiased dataset.</p>
  
  <p><strong>Should I keep images of my dataset for this purpose or should I use a real life dataset ?</strong></p>
  
  <p>(A good solution would be to have a real life training set, but I don't have)</p>
  
  <p>I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.</p>
  
  <p><strong>Bonus question</strong>: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")</p>
  
  <p>Precision-Recall ? (but I have to choose a pose classification threshold...)</p>
  
Tags: <deep-learning><datasets><object-detection>
Title: On which data evaluate an object detection model ? (similar or real life data ?)
UserDisplayName: antoine Mathu
ThreadText: # On which data evaluate an object detection model ? (similar or real life data ?)
  
  (Asked by: username_0 on 2020-02-05)
  
  I'm training an object detection model (SSD300) to detect and classify body poses in thermal images.
  
  Even I have more than 2k different poses, but **the background does not change much** (I have only 5 different points of view).
  
  I trained my model on these images (70% for the training and 30% for validation).
  
  **Now, I want to evaluate** the model on an unbiased dataset.
  
  **Should I keep images of my dataset for this purpose or should I use a real life dataset ?**
  
  (A good solution would be to have a real life training set, but I don't have)
  
  I tried both, but as expected, I have an mAP=0.9 when evaluated on similar pictures and mAP=0.5 when evaluated on completely different images.
  
  **Bonus question**: is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")
  
  Precision-Recall ? (but I have to choose a pose classification threshold...)
  
  **Tags:** deep-learning, datasets, object-detection
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-28. Score: 1
  
  If you want to evaluate on real thermal image dataset, you can use this one.
  
  [Thermal Image dataset](https://www.flir.com/oem/adas/dataset/)
  
  is mAP a relevant metric when I want to show result to a client ? (e.g a client doesn't understand if I tell him "my model has a mAP=0.7")
  
  > Mean Average Precision is the relevant metric but it's more technical. You can start explaining with False Positives and False
  > Negatives in the predictions. In turn, it leads to Precision and Recall. It mostly depends upon your use case because there will always be a trade-off between them.
  
  > **Comments:**
  > **username_0**: Ty for your answer on the mAP but you did not answer my main question on "how to build a good evaluation/test set ?".  
  > **username_1**: At the beginning itself, we have to split the data as train/val/test. It depends upon how much data you have for your task to train the model. The test set should be taken from almost the same data distribution of the training set. If we train a model with only daylight images and test with night conditions, it is a bad idea to judge the model.  
  
  ---
  
  