Answers.Body: <p>First, are you planning to train a detector yourself, to use a pre-trained detector or an off-the-shelf model (maybe there is an already existing detector for sperm whale echolocation clicks)? This will influence the amount of labeling you will need to do.</p>
  <ul>
  <li><p>In the first case, an empirical rule of thumb I found to yield good results is to have <strong>40 hours per sound category</strong> (in your case that would be 40 hours of whale echolocation clicks and 40 hours of background soundscape).</p>
  </li>
  <li><p>In the second case, using a model that has been pre-trained on Audioset (or even a model that has been trained on marine mammal already) will reduce enormously the amount of manual labeling you will need to do. I got great success training models using only a few hours of sound per category (in my case it was anthropogenic noise, which may be easier to discriminate for a model).</p>
  </li>
  <li><p>In the last case, if you already have a model that you are able to use off-the-shelf the amount of labeling you need would just be a few hours (it should be enough to be confident about model performance). In my opinion, the most important factor is to randomly select audio samples (just don't take samples you know contains whale clicks) so you get a <strong>balanced dataset</strong> (relative to the amount of whale echolocation clicks).</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T17:36:57.637Z
Answers.Text: Planning to use a sperm whale detector I've used and modified in the past (in program Ishmael) but anticipate needing to do some initial tuning for my particular recordings
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T15:06:22.863Z
Answers.LastActivityDate: 2022-07-26T23:54:41.100Z
Answers.LastEditDate: 2022-07-26T23:54:41.100Z
Answers.UserDisplayName: Benjamin Cretois
Answers.Body: <p>Sampling is a perfectly reasonable strategy.  However, you need to make sure that you sample across the different regimes that you might have which could affect performance.  As an example, if you are recording in a region that has changes in noise patterns (e.g. pleasure boats on weekends, tidal flow noise, etc.), ideally it would be nice if you had an idea of how your performance varied in these situations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T07:41:28.660Z
Answers.LastActivityDate: 2022-07-26T23:54:22.803Z
Answers.LastEditDate: 2022-07-26T23:54:22.803Z
Answers.UserDisplayName: Marie R
Answers.Body: <p>Selene, hi.  I don't work in marine acoustics, but the amount of manual checking required to assess precision/recall and enumerate your false positives and negatives is probably less than you think. Chambert et al (2018) <a href="https://doi.org/10.1111/2041-210X.12910" rel="nofollow noreferrer">https://doi.org/10.1111/2041-210X.12910</a> stated this:</p>
  <p><em>&quot;When false positives occur, estimator accuracy can be improved when a subset of detections produced by the classification algorithm is post-validated by a human observer. We use simulations to investigate the relationship between accuracy and effort spent on post-validation, and found that very accurate occupancy estimates can be obtained with as little as 1% of data being validated&quot;.</em></p>
  <p>For my paper on capercaillie <a href="https://doi.org/10.1007/s10336-019-01649-8" rel="nofollow noreferrer">https://doi.org/10.1007/s10336-019-01649-8</a>, I checked 4% of the dataset and got useful outputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T19:16:33.240Z
Answers.Text: Thanks @carlos that paper is really helpful. I modified the tags for my question to remove "marine-species" because this question can apply across all taxa!
Answers.UserDisplayName: selene
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-22T17:16:17.323Z
Answers.LastActivityDate: 2022-07-26T23:27:16.140Z
Answers.LastEditDate: 2022-07-26T23:27:16.140Z
Answers.UserDisplayName: Carlos Abrahams
Body: <p>I have 16 weeks of underwater acoustic recordings (2 recorders, 8 weeks each), and I plan to use an automated detector to detect sperm whale echolocation clicks. I would like to assess the detector performance (precision, recall, false positive rate) but am unsure how much manual checking of the recordings that I need to do to know I have accurately captured the performance.</p>
  <p>At the lowest analysis level for this project, I just want to assess presence/absence. But, at a higher level (and thinking back on past work) I'd like to know the answer for this for acoustic density estimation outcomes.</p>
  <p>When assessing automated detector performance, how much manual checking of the recordings do I need to do?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-22T22:16:05.083Z
Comments.Text: I ask myself this question all the time! Could you edit to clarify the ultimate motivating question that led you to use a detector in the first place? This might help folks provide better answers. If it is a question of sperm whale occupancy (presence/absence), then the answer may indeed be "less than you think" (as Carlos points out below). If it's raw detector performance on ALL echolocation clicks (perhaps because the motivating question is related to click behavior, or something similar that requires finer grain understanding) then much more manual checking might be necessary.
Comments.UserDisplayName: Cathleen B
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-23T10:20:13.693Z
Comments.Text: One small suggestion: with your manually-checked data, you can determine **confidence intervals** on your evaluation scores, using bootstrap sampling. Then look at those confidence intervals. Are they too wide? Check more data!
Comments.UserDisplayName: Dan Stowell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-23T19:04:23.400Z
Comments.Text: @DanStowell that is a great approach, and would get around how variable the answer to this is for different questions (and with Marie R's answer below)
Comments.UserDisplayName: selene
Tags: <automated-detection><detector>
Title: When assessing automated detector performance, how much manual checking of the detections is "enough"?
ThreadText: # When assessing automated detector performance, how much manual checking of the detections is "enough"?
  
  (Asked by: username_0 on 2022-06-22)
  
  I have 16 weeks of underwater acoustic recordings (2 recorders, 8 weeks each), and I plan to use an automated detector to detect sperm whale echolocation clicks. I would like to assess the detector performance (precision, recall, false positive rate) but am unsure how much manual checking of the recordings that I need to do to know I have accurately captured the performance.
  
  At the lowest analysis level for this project, I just want to assess presence/absence. But, at a higher level (and thinking back on past work) I'd like to know the answer for this for acoustic density estimation outcomes.
  
  When assessing automated detector performance, how much manual checking of the recordings do I need to do?
  
  **Tags:** automated-detection, detector
  
  > **Comments:**
  > **username_1**: I ask myself this question all the time! Could you edit to clarify the ultimate motivating question that led you to use a detector in the first place? This might help folks provide better answers. If it is a question of sperm whale occupancy (presence/absence), then the answer may indeed be "less than you think" (as Carlos points out below). If it's raw detector performance on ALL echolocation clicks (perhaps because the motivating question is related to click behavior, or something similar that requires finer grain understanding) then much more manual checking might be necessary.  
  > **username_2**: One small suggestion: with your manually-checked data, you can determine **confidence intervals** on your evaluation scores, using bootstrap sampling. Then look at those confidence intervals. Are they too wide? Check more data!  
  > **username_0**: @DanStowell that is a great approach, and would get around how variable the answer to this is for different questions (and with username_4's answer below)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2022-06-22. Score: 14 | Accepted answer
  
  Selene, hi. I don't work in marine acoustics, but the amount of manual checking required to assess precision/recall and enumerate your false positives and negatives is probably less than you think. Chambert et al (2018) <https://doi.org/10.1111/2041-210X.12910> stated this:
  
  *"When false positives occur, estimator accuracy can be improved when a subset of detections produced by the classification algorithm is post-validated by a human observer. We use simulations to investigate the relationship between accuracy and effort spent on post-validation, and found that very accurate occupancy estimates can be obtained with as little as 1% of data being validated".*
  
  For my paper on capercaillie <https://doi.org/10.1007/s10336-019-01649-8>, I checked 4% of the dataset and got useful outputs.
  
  > **Comments:**
  > **username_0**: Thanks @carlos that paper is really helpful. I modified the tags for my question to remove "marine-species" because this question can apply across all taxa!  
  
  ---
  
  ### Answer by username_3 on 2022-06-22. Score: 6
  
  First, are you planning to train a detector yourself, to use a pre-trained detector or an off-the-shelf model (maybe there is an already existing detector for sperm whale echolocation clicks)? This will influence the amount of labeling you will need to do.
  
  * In the first case, an empirical rule of thumb I found to yield good results is to have **40 hours per sound category** (in your case that would be 40 hours of whale echolocation clicks and 40 hours of background soundscape).
  * In the second case, using a model that has been pre-trained on Audioset (or even a model that has been trained on marine mammal already) will reduce enormously the amount of manual labeling you will need to do. I got great success training models using only a few hours of sound per category (in my case it was anthropogenic noise, which may be easier to discriminate for a model).
  * In the last case, if you already have a model that you are able to use off-the-shelf the amount of labeling you need would just be a few hours (it should be enough to be confident about model performance). In my opinion, the most important factor is to randomly select audio samples (just don't take samples you know contains whale clicks) so you get a **balanced dataset** (relative to the amount of whale echolocation clicks).
  
  > **Comments:**
  > **username_0**: Planning to use a sperm whale detector I've used and modified in the past (in program Ishmael) but anticipate needing to do some initial tuning for my particular recordings  
  
  ---
  
  ### Answer by username_4 on 2022-06-23. Score: 6
  
  Sampling is a perfectly reasonable strategy. However, you need to make sure that you sample across the different regimes that you might have which could affect performance. As an example, if you are recording in a region that has changes in noise patterns (e.g. pleasure boats on weekends, tidal flow noise, etc.), ideally it would be nice if you had an idea of how your performance varied in these situations.
  
  
  ---
  
  