Answers.Body: <p>Unfortunately, if you want to evaluate recall, you absolutely need to know ALL the true events. In theory, you can use any method to find them but the underlying hypothesis is that the method gives 100% correct results. In practice, the &quot;true&quot; events are found by manually inspecting all the files because it is what is considered to be the closest to ground truth (only subject to human error).</p>
  <p>You can annotate a subset of your database but it has to be representative of the rest of the database. Then, you can make the assumption that the recall on the annotated database is equal to the recall on the rest of the database.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T14:16:40.090Z
Answers.Text: @dtsavage this question is related and you might find it helpful https://bioacoustics.stackexchange.com/questions/16/when-assessing-automated-detector-performance-how-much-manual-checking-of-the-d
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T15:44:20.740Z
Answers.Text: Has anyone written a paper on determining the size of a subsample that you might need to annotate to get a reliable estimate of that recall? I'm not familiar with one but it feels like something that should be out there.
Answers.UserDisplayName: dtsavage
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T15:27:30.163Z
Answers.LastActivityDate: 2022-07-06T15:27:30.163Z
Answers.UserDisplayName: Maxime BRU
Body: <p>Hi fellow bioacousticians,</p>
  <p>I am working on a needle-in-a-haystack detection problem and am trying to figure out the best way to assess the rate of false negatives. My goal is to detect gunshots in terabytes of PAM data from Central Africa using a template-based detector with monitoR in R, which seems to be working decently well. Calculating the precision (true positives / positives) is a breeze, but recall (true positives / false negatives) is more challenging, since gunshots occur very rarely (there is a gunshot every 1-2 days and the data is continuous). Manually scanning files to see how many gunshots were missed by my detector would be extremely time consuming, not to mention that I would want to see how the recall varies with habitat type, but I can't think of another option to see how many gunshots were missed by my model.</p>
  <p>Have any of you ever been in a similar situation with a similar problem? If so, what was the most time-efficient way to get a robust understanding of recall in this context? Or, did you not calculate recall and just spoke about precision?</p>
  <p>Thanks!</p>
  
Tags: <automated-detection><pam><detector><gunshot>
Title: Method for assessing false negatives for rare sounds in long-term PAM dataset
ThreadText: # Method for assessing false negatives for rare sounds in long-term PAM dataset
  
  (Asked by: username_0 on 2022-07-05)
  
  Hi fellow bioacousticians,
  
  I am working on a needle-in-a-haystack detection problem and am trying to figure out the best way to assess the rate of false negatives. My goal is to detect gunshots in terabytes of PAM data from Central Africa using a template-based detector with monitoR in R, which seems to be working decently well. Calculating the precision (true positives / positives) is a breeze, but recall (true positives / false negatives) is more challenging, since gunshots occur very rarely (there is a gunshot every 1-2 days and the data is continuous). Manually scanning files to see how many gunshots were missed by my detector would be extremely time consuming, not to mention that I would want to see how the recall varies with habitat type, but I can't think of another option to see how many gunshots were missed by my model.
  
  Have any of you ever been in a similar situation with a similar problem? If so, what was the most time-efficient way to get a robust understanding of recall in this context? Or, did you not calculate recall and just spoke about precision?
  
  Thanks!
  
  **Tags:** automated-detection, pam, detector, gunshot
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-06. Score: 3
  
  Unfortunately, if you want to evaluate recall, you absolutely need to know ALL the true events. In theory, you can use any method to find them but the underlying hypothesis is that the method gives 100% correct results. In practice, the "true" events are found by manually inspecting all the files because it is what is considered to be the closest to ground truth (only subject to human error).
  
  You can annotate a subset of your database but it has to be representative of the rest of the database. Then, you can make the assumption that the recall on the annotated database is equal to the recall on the rest of the database.
  
  > **Comments:**
  > **username_3**: Has anyone written a paper on determining the size of a subsample that you might need to annotate to get a reliable estimate of that recall? I'm not familiar with one but it feels like something that should be out there.  
  > **username_2**: @username_3 this question is related and you might find it helpful https://bioacoustics.stackexchange.com/questions/16/when-assessing-automated-detector-performance-how-much-manual-checking-of-the-d  
  
  ---
  
  