Answers.Body: <p>I guess that depends on the context and the audience of the presentation.
  I know both metrics can be translated.</p>
  <p>I consider detections as hypothesis testing and then a first-kind error is a false alarm (false positive) and a true positive is a correct detection.</p>
  <p>I understand that considering everything as classification (detection is binary classification) then different metrics could be used.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-05T19:32:16.947Z
Answers.LastActivityDate: 2022-07-05T19:32:16.947Z
Answers.UserDisplayName: WMXZ
Body: <p>When using automated detection of focal species sounds, when and why should we focus on characterizing probability of detection and probability of false alarm, vs. focusing on precision and recall? What are the pros and cons of either set of evaluation metrics, and are there heuristics for when to use one over the other?</p>
  <p>I would like to consider this question under two different motivating examples:</p>
  <ul>
  <li>Using an automated detection system to detect a species of interest for use in an occupancy modeling framework (i.e., detection/non-detection data are needed within selected survey periods)</li>
  <li>Using an automated detection system to characterize phenological trends for a species of interest (i.e., number of detections per day over the course of a breeding season)</li>
  </ul>
  <p>Inspired by discussion here: <a href="https://bioacoustics.stackexchange.com/questions/271/why-report-snr-and-what-snr-is-acceptable/274#274">Why report SNR, and what SNR is acceptable?</a> (specifically, <a href="https://bioacoustics.stackexchange.com/a/274/119">this answer</a>) and here:
  <a href="https://bioacoustics.stackexchange.com/questions/16/when-assessing-automated-detector-performance-how-much-manual-checking-of-the-d">When assessing automated detector performance, how much manual checking of the detections is &quot;enough&quot;?</a></p>
  <p>ETA: My experience is in terrestrial bioacoustics (largely an avian lens) which affects my system assumptions. If there is nuance in how to consider this question for terrestrial vs. underwater problems, it will be worth including that in any answers.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T20:31:15.097Z
Comments.Text: How would you define probability of detection in this case? I struggle with that phrase because it is so integral to density estimation, but I am thinking in this case it is just some normalized number of detections per unit recording effort? Or do you define it a different way?
Comments.UserDisplayName: selene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T21:07:55.833Z
Comments.Text: Great point of clarification. My terrestrial occupancy lens leads me to define detection probability as the probability of detecting a species at a site, given that the species is present. In an occupancy question, I could estimate detection probability 'p' directly from the occupancy modeling framework, and I suppose not worry about other evaluation metrics (?). Observing some other relevant discussion across the site has made me second-guess myself in considering the varied assumptions we might have about applying that terminology, though.
Comments.UserDisplayName: Cathleen B
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T21:17:46.583Z
Comments.Text: So, in your case (sorry...I'm not familiar with occupancy modeling!) 'p' would be based on the detection process as well as some other environmental variables that maybe would impact detectability, because you are getting an overall 'p' from the model? Or would it be solely based on the detection process?
Comments.UserDisplayName: selene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T21:45:28.730Z
Comments.Text: In an occupancy model, detection probability 'p' can be based solely on the detection process (i.e., the parameter is assumed constant), OR can be modeled as a function of other environmental variables (e.g., p ~ time of detection + wind). I suspect there might be some implicit assumptions from terrestrial researchers about this terminology which is why the question and potential terminology differences piqued my interest in the context of bioacoustics.
Comments.UserDisplayName: Cathleen B
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T21:49:36.043Z
Comments.Text: Ok thanks - that makes sense, and I am interested to follow this discussion as well :)
Comments.UserDisplayName: selene
Tags: <machine-learning><automated-detection><classifier>
Title: Probability of detection / probability of false alarm vs. precision / recall
ThreadText: # Probability of detection / probability of false alarm vs. precision / recall
  
  (Asked by: username_0 on 2022-07-05)
  
  When using automated detection of focal species sounds, when and why should we focus on characterizing probability of detection and probability of false alarm, vs. focusing on precision and recall? What are the pros and cons of either set of evaluation metrics, and are there heuristics for when to use one over the other?
  
  I would like to consider this question under two different motivating examples:
  
  * Using an automated detection system to detect a species of interest for use in an occupancy modeling framework (i.e., detection/non-detection data are needed within selected survey periods)
  * Using an automated detection system to characterize phenological trends for a species of interest (i.e., number of detections per day over the course of a breeding season)
  
  Inspired by discussion here: [Why report SNR, and what SNR is acceptable?](https://bioacoustics.stackexchange.com/questions/271/why-report-snr-and-what-snr-is-acceptable/274#274) (specifically, [this answer](https://bioacoustics.stackexchange.com/a/274/119)) and here:
  [When assessing automated detector performance, how much manual checking of the detections is "enough"?](https://bioacoustics.stackexchange.com/questions/16/when-assessing-automated-detector-performance-how-much-manual-checking-of-the-d)
  
  ETA: My experience is in terrestrial bioacoustics (largely an avian lens) which affects my system assumptions. If there is nuance in how to consider this question for terrestrial vs. underwater problems, it will be worth including that in any answers.
  
  **Tags:** machine-learning, automated-detection, classifier
  
  > **Comments:**
  > **username_1**: How would you define probability of detection in this case? I struggle with that phrase because it is so integral to density estimation, but I am thinking in this case it is just some normalized number of detections per unit recording effort? Or do you define it a different way?  
  > **username_0**: Great point of clarification. My terrestrial occupancy lens leads me to define detection probability as the probability of detecting a species at a site, given that the species is present. In an occupancy question, I could estimate detection probability 'p' directly from the occupancy modeling framework, and I suppose not worry about other evaluation metrics (?). Observing some other relevant discussion across the site has made me second-guess myself in considering the varied assumptions we might have about applying that terminology, though.  
  > **username_1**: So, in your case (sorry...I'm not familiar with occupancy modeling!) 'p' would be based on the detection process as well as some other environmental variables that maybe would impact detectability, because you are getting an overall 'p' from the model? Or would it be solely based on the detection process?  
  > **username_0**: In an occupancy model, detection probability 'p' can be based solely on the detection process (i.e., the parameter is assumed constant), OR can be modeled as a function of other environmental variables (e.g., p ~ time of detection + wind). I suspect there might be some implicit assumptions from terrestrial researchers about this terminology which is why the question and potential terminology differences piqued my interest in the context of bioacoustics.  
  > **username_1**: Ok thanks - that makes sense, and I am interested to follow this discussion as well :)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-07-05. Score: 1
  
  I guess that depends on the context and the audience of the presentation.
  I know both metrics can be translated.
  
  I consider detections as hypothesis testing and then a first-kind error is a false alarm (false positive) and a true positive is a correct detection.
  
  I understand that considering everything as classification (detection is binary classification) then different metrics could be used.
  
  
  ---
  
  