Answers.Body: <p>I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-10T00:49:27.523Z
Answers.Text: That would only apply for stereotyped signals. Not all detection methods are trying to find sounds that closely match a template though.
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-10T04:22:46.780Z
Answers.Text: yes, but OP asked for similarity (overlap) with reference signal.
Answers.UserDisplayName: WMXZ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-10T23:30:41.763Z
Answers.Text: Cathleen mentions that "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table". So it is overlap in time. not similarity.
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T14:28:32.727Z
Answers.Text: Marcelo is correct, I meant an overlap in time. An automated detection might overlap one partial note out of a three-note song by a target species. If I'm trying to diagnose performance of the detector, do I count that detection or not? The answer is usually "it depends on the question" so I'm curious about the heuristics others use. (Apologies for any imprecise terminology; I'm coming at this from a biology background.)
Answers.UserDisplayName: Cathleen B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T15:01:23.530Z
Answers.Text: I'm confused but try to understand.  Assume you have a three-note song and your detector detected only one note? is that the question?  Was the detector trained for this three-note song (is the three-note song in the reference table)?
Answers.UserDisplayName: WMXZ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T16:18:40.040Z
Answers.Text: Some detector tells you not only if something is detected but also the start and end times of the detection. That start and end might include only a portion of the signal of interest (compared to the annotated start and end times), so a partial overlap, or might be a perfect match. Overlap can be seen as a continuous parameter, and it seems that Cathleen is asking about cutoff values for deciding whether a detection is good enough to be counted as a true positive.
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T22:16:07.370Z
Answers.Text: @MarceloAraya-Salas that is correct. I'm reading through ohun's documentation and it seem like you have thought deeply about this and had to make some relevant decisions in your performance evaluation functions. I'm interested to know how your software handles it. Let me know if there's an edit I can make to my question to make it more clear (or feel free to make that edit).
Answers.UserDisplayName: Cathleen B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-09T04:52:15.703Z
Answers.LastActivityDate: 2022-07-09T04:52:15.703Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>In the R package <a href="https://marce10.github.io/ohun/" rel="noreferrer">ohun</a> any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function <a href="https://marce10.github.io/ohun/reference/diagnose_detection.html" rel="noreferrer">diagnose_detection</a>) also includes a summary about the overall overlap of the true positives (&quot;overlap.to.true.positives&quot;, a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function <a href="https://marce10.github.io/ohun/reference/label_detection.html" rel="noreferrer">label_detection</a>. That would add an extra column that you can use to remove those with little overlap.</p>
  <p>Here is an example on how to do that:</p>
  <pre><code>#remotes::install_github(&quot;maRce10/ohun&quot;)
  library(ohun)
  
  # load example data
  data(&quot;lbh_reference&quot;)
  
  # create a simulated detection ouput ##
  # copy reference
  sim_detection &lt;- lbh_reference
  
  # add some time to the first 3 rows so they barely overlap with the original data
  sim_detection$start[1:3] &lt;- sim_detection$start[1:3] + 0.1
  sim_detection$end[1:3] &lt;- sim_detection$end[1:3] + 0.1
  
  # check peformance (look at the overlap.to.true positives) ##
  # for a perfect detection
  diagnose_detection(reference = lbh_reference, detection = lbh_reference)
  
  # for the simulated detection
  diagnose_detection(reference = lbh_reference, detection = sim_detection)
  
  # remove low overlap ones #
  # get amount of overlap
  lab_detection &lt;- label_detection(reference = lbh_reference, detection = sim_detection)
  
  print(lab_detection$overlap)
  
  # exclude those below 0.5
  filtered_detection &lt;- lab_detection[lab_detection$overlap &gt; 0.5, ]
  
  # check performance again (of course recall will go down!)
  diagnose_detection(reference = lbh_reference, detection = filtered_detection)
  </code></pre>
  <p>links:
  ohun: <a href="https://marce10.github.io/ohun/" rel="noreferrer">https://marce10.github.io/ohun/</a></p>
  <p>label_detection: <a href="https://marce10.github.io/ohun/reference/label_detection.html" rel="noreferrer">https://marce10.github.io/ohun/reference/label_detection.html</a></p>
  <p>diagnose_detection: <a href="https://marce10.github.io/ohun/reference/diagnose_detection.html" rel="noreferrer">https://marce10.github.io/ohun/reference/diagnose_detection.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-12T21:37:59.583Z
Answers.LastActivityDate: 2022-07-13T12:49:08.057Z
Answers.LastEditDate: 2022-07-13T12:49:08.057Z
Answers.UserDisplayName: Marcelo Araya-Salas
Body: <p>For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?</p>
  <p>Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a &quot;truth&quot; data set. Depending on the motivating question, I've gone back and forth between coding it as, &quot;a true positive means the centroid of a detection needs to be contained within an annotation in the reference table&quot;, vs. &quot;ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive&quot;. Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?</p>
  <p>Inspired by this relevant question: <a href="https://bioacoustics.stackexchange.com/questions/295/software-options-for-diagnosing-the-performance-of-a-detection-routine">Software options for diagnosing the performance of a detection routine</a></p>
  
Tags: <automated-detection><classifier><spectrogram-annotation><machine-learning><diagnose-performance>
Title: What criteria should be used to decide whether an automated detection overlaps with a reference annotation?
ThreadText: # What criteria should be used to decide whether an automated detection overlaps with a reference annotation?
  
  (Asked by: username_0 on 2022-07-08)
  
  For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?
  
  Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a "truth" data set. Depending on the motivating question, I've gone back and forth between coding it as, "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?
  
  Inspired by this relevant question: [Software options for diagnosing the performance of a detection routine](https://bioacoustics.stackexchange.com/questions/295/software-options-for-diagnosing-the-performance-of-a-detection-routine)
  
  **Tags:** automated-detection, classifier, spectrogram-annotation, machine-learning, diagnose-performance
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-12. Score: 8 | Accepted answer
  
  In the R package ohun any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function diagnose_detection) also includes a summary about the overall overlap of the true positives ("overlap.to.true.positives", a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function label_detection. That would add an extra column that you can use to remove those with little overlap.
  Here is an example on how to do that:
  #remotes::install_github("maRce10/ohun")
  library(ohun)
  
  # load example data
  data("lbh_reference")
  
  # create a simulated detection ouput ##
  # copy reference
  sim_detection <- lbh_reference
  
  # add some time to the first 3 rows so they barely overlap with the original data
  sim_detection$start[1:3] <- sim_detection$start[1:3] + 0.1
  sim_detection$end[1:3] <- sim_detection$end[1:3] + 0.1
  
  # check peformance (look at the overlap.to.true positives) ##
  # for a perfect detection
  diagnose_detection(reference = lbh_reference, detection = lbh_reference)
  
  # for the simulated detection
  diagnose_detection(reference = lbh_reference, detection = sim_detection)
  
  # remove low overlap ones #
  # get amount of overlap
  lab_detection <- label_detection(reference = lbh_reference, detection = sim_detection)
  
  print(lab_detection$overlap)
  
  # exclude those below 0.5
  filtered_detection <- lab_detection[lab_detection$overlap > 0.5, ]
  
  # check performance again (of course recall will go down!)
  diagnose_detection(reference = lbh_reference, detection = filtered_detection)
  
  links:
  ohun: https://marce10.github.io/ohun/
  label_detection: https://marce10.github.io/ohun/reference/label_detection.html
  diagnose_detection: https://marce10.github.io/ohun/reference/diagnose_detection.html
  
  
  ---
  
  ### Answer by username_1 on 2022-07-09. Score: 1
  
  I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.
  
  > **Comments:**
  > **username_2**: That would only apply for stereotyped signals. Not all detection methods are trying to find sounds that closely match a template though.  
  > **username_1**: yes, but OP asked for similarity (overlap) with reference signal.  
  > **username_2**: Cathleen mentions that "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table". So it is overlap in time. not similarity.  
  > **username_0**: Marcelo is correct, I meant an overlap in time. An automated detection might overlap one partial note out of a three-note song by a target species. If I'm trying to diagnose performance of the detector, do I count that detection or not? The answer is usually "it depends on the question" so I'm curious about the heuristics others use. (Apologies for any imprecise terminology; I'm coming at this from a biology background.)  
  > **username_1**: I'm confused but try to understand.  Assume you have a three-note song and your detector detected only one note? is that the question?  Was the detector trained for this three-note song (is the three-note song in the reference table)?  
  > **username_2**: Some detector tells you not only if something is detected but also the start and end times of the detection. That start and end might include only a portion of the signal of interest (compared to the annotated start and end times), so a partial overlap, or might be a perfect match. Overlap can be seen as a continuous parameter, and it seems that Cathleen is asking about cutoff values for deciding whether a detection is good enough to be counted as a true positive.  
  > **username_0**: @MarceloAraya-Salas that is correct. I'm reading through ohun's documentation and it seem like you have thought deeply about this and had to make some relevant decisions in your performance evaluation functions. I'm interested to know how your software handles it. Let me know if there's an edit I can make to my question to make it more clear (or feel free to make that edit).  
  
  ---
  
  