Answers.Body: <p>The new R package ohun allows to do exactly that (diagnose the performance of detection routines).</p>
  <p><a href="https://marce10.github.io/ohun/index.html" rel="nofollow noreferrer">https://marce10.github.io/ohun/index.html</a></p>
  <p>It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.</p>
  <p>I am still working on it so would be happy to hear suggestions for new features.</p>
  <p>Edit: to echo @lostanlen, the <a href="http://tut-arg.github.io/sed_eval/" rel="nofollow noreferrer">python package sed_eval</a> for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T11:43:57.920Z
Answers.Text: Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison.
Answers.UserDisplayName: Pavel Linhart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T17:10:01.070Z
Answers.Text: Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms?
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T09:36:31.437Z
Answers.Text: Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: https://bioacoustics.stackexchange.com/questions/1043/tools-to-visualize-and-compare-species-detections-in-long-term-recordings
Answers.UserDisplayName: Pavel Linhart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-16T22:38:15.793Z
Answers.Text: Hello @MarceloAraya-Salas, i'm afraid there is a problem in your algorithm for counting true positives. https://github.com/maRce10/ohun/blob/master/R/diagnose_detection.R#L170-L183
  
  As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp)
  
  See mir_eval package: https://craffel.github.io/mir_eval/
  implementation by Brian McFee https://github.com/craffel/mir_eval/pull/258
Answers.UserDisplayName: lostanlen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-17T19:46:43.313Z
Answers.Text: thanks I will look at it
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T17:12:23.230Z
Answers.LastActivityDate: 2022-08-26T18:29:00.647Z
Answers.LastEditDate: 2022-08-26T18:29:00.647Z
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.Body: <p>I am also looking for tools doing that and seems like people usually do not share them.</p>
  <p>Some time ago I tested sed_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: <a href="https://webpages.tuni.fi/arg/index.php/code" rel="nofollow noreferrer">https://webpages.tuni.fi/arg/index.php/code</a> .
  I used it on my Raven selection tables but needed to tweak the selection tables before.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T16:03:59.143Z
Answers.Text: Hi Pavel and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50).
Answers.UserDisplayName: Noil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T11:46:15.090Z
Answers.Text: @Noil Thanks, I did so.
Answers.UserDisplayName: Pavel Linhart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-09T11:25:15.273Z
Answers.LastActivityDate: 2022-08-10T11:45:38.317Z
Answers.LastEditDate: 2022-08-10T11:45:38.317Z
Answers.UserDisplayName: Pavel Linhart
Body: <p>Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T20:01:36.520Z
Comments.Text: @CathleenBalantic that would be a great additional question. I've asked myself that!
Comments.UserDisplayName: selene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T21:43:22.350Z
Comments.Text: Thanks @selene, I added this question here: https://bioacoustics.stackexchange.com/questions/510/what-criteria-should-be-used-to-decide-whether-an-automated-detection-overlaps-w
Comments.UserDisplayName: Cathleen B
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T20:00:12.507Z
Comments.Text: I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in,
Comments.UserDisplayName: selene
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T20:13:15.273Z
Comments.Text: Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this.
Comments.UserDisplayName: Cathleen B
Tags: <automated-detection><diagnose-performance><signal-detection>
Title: Software options for diagnosing the performance of a detection routine
ThreadText: # Software options for diagnosing the performance of a detection routine
  
  (Asked by: username_0 on 2022-07-05)
  
  Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.
  
  **Tags:** automated-detection, diagnose-performance, signal-detection
  
  > **Comments:**
  > **username_1**: I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in,  
  > **username_2**: Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this.  
  > **username_1**: @CathleenBalantic that would be a great additional question. I've asked myself that!  
  > **username_2**: Thanks @username_1, I added this question here: https://bioacoustics.stackexchange.com/questions/510/what-criteria-should-be-used-to-decide-whether-an-automated-detection-overlaps-w  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-07-11. Score: 8
  
  The new R package ohun allows to do exactly that (diagnose the performance of detection routines).
  
  <https://marce10.github.io/ohun/index.html>
  
  It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.
  
  I am still working on it so would be happy to hear suggestions for new features.
  
  Edit: to echo @username_4, the [python package sed\_eval](http://tut-arg.github.io/sed_eval/) for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection
  
  > **Comments:**
  > **username_3**: Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison.  
  > **username_0**: Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms?  
  > **username_3**: Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: https://bioacoustics.stackexchange.com/questions/1043/tools-to-visualize-and-compare-species-detections-in-long-term-recordings  
  > **username_4**: Hello @MarceloAraya-Salas, i'm afraid there is a problem in your algorithm for counting true positives. https://github.com/maRce10/ohun/blob/master/R/diagnose_detection.R#L170-L183  As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp)  See mir_eval package: https://craffel.github.io/mir_eval/ implementation by Brian McFee https://github.com/craffel/mir_eval/pull/258  
  > **username_0**: thanks I will look at it  
  
  ---
  
  ### Answer by username_3 on 2022-08-09. Score: 2
  
  I am also looking for tools doing that and seems like people usually do not share them.
  
  Some time ago I tested sed\_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: <https://webpages.tuni.fi/arg/index.php/code> .
  I used it on my Raven selection tables but needed to tweak the selection tables before.
  
  > **Comments:**
  > **username_5**: Hi Pavel and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50).  
  > **username_3**: @username_5 Thanks, I did so.  
  
  ---
  
  