Answers.Body: <p>I'd agree that these subjective measures are useful. For example, you may test an automatic detectors ability to detect the 'certain' sounds, but when measuring false alarm rate you probably shouldn't penalise yourself if the detector picks up a 'possible'.</p>
  <p>However, you might also consider trying to introduce some more objective measured, the obvious one being Signal to Noise Ratio. Humans are very good at picking out weak signals, especially when prompted by the presence of nearby louder ones and it may not be reasonable to expect an automatic detector to find the low SNR calls. So again, judge your detector on it's ability to detect sounds with SNR &gt; ?? dB, but don't penalise your algorithm if it gets a quiet one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-27T18:14:36.580Z
Answers.LastActivityDate: 2022-09-27T18:14:36.580Z
Answers.UserDisplayName: Doug Gillespie
Answers.Body: <p>As detectability is really a statistical variable (probability of detection), I would suggest annotating the data with at least levels (certain, probable, possible).</p>
  <p>For this I use as criteria (being a non-native speaker):</p>
  <ul>
  <li>certain: there is no doubt on the detection</li>
  <li>probable: not certain, but high probability</li>
  <li>possible: may or may not be</li>
  </ul>
  <p>As by Doug's answer, yes, a quantitative measure (SNR for example) if provided by the software is an important parameter for annotations.  Even if this is not available, because you use spectrograms and have no time to measure values and the software does not do it for you, qualitative measures (strong, average, weak) are also important.</p>
  <p>This could yield to such interesting observation like (strong possible click, or is it really an electronic transient) or (weak but certain dolphin whistle)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-24T13:17:04.587Z
Answers.LastActivityDate: 2022-09-30T14:41:30.127Z
Answers.LastEditDate: 2022-09-30T14:41:30.127Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>I have found such annotations to be very useful when working on detection tasks. The practice is not quite standardized, but there are several examples out there. For example:</p>
  <p>The <a href="https://urbansounddataset.weebly.com/urbansound8k.html" rel="nofollow noreferrer">Urbansound8k</a> dataset annotates &quot;salience&quot;:</p>
  <p>A (subjective) salience rating of the sound. 1 = foreground, 2 = background.</p>
  <p>The <a href="https://zenodo.org/record/3966543" rel="nofollow noreferrer">SONYC-UST</a> dataset annotates &quot;proximity&quot;:</p>
  <p>Indicates the proximity of the sound event to the sensor, in 3 possible values: (near, far, notsure) plus Not Available (encoded as -1).</p>
  <p>It can be useful to compute some objective metrics based on these annotations.
  For example, to estimate the Signal to Noise ratio (SNR) of the annotated events. Can be done by computing a sound level indicator such as A-weighted Fast/125ms, and then comparing the level of the event to the surrounding/preceeding background level.
  Then one can plot performance metrics such as False Positive / False Negative versus the SNR levels, and also look at the subjective ratings vs detection metrics or vs SNR.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-05T00:31:27.323Z
Answers.LastActivityDate: 2023-06-05T00:31:27.323Z
Answers.UserDisplayName: Jon Nordby
Body: <p>When creating an algorithm to perform a specific task on audio recordings, it is sometimes needed to first perform the task manually in order to create a reference which can then be used to train and evaluate the algorithm.
  When doing such manual annotations, sometimes the sound events you are annotating have a high variability in detectability (some are more easily detectable than others due to the distance of sound emitters for instance; if using a spectrogram display, some events are sometimes visible but not audible or the other way around).</p>
  <p>I would imagine that &quot;human&quot; detectability is related to &quot;algorithm&quot; detectability. That's why it's sometimes tempting to only annotate the events that are salient/loud/detectable enough (and deliberately ignore those which are barely detectable) in order to &quot;help&quot; the algorithm to train.</p>
  <p>Is it a good idea to do that? More generally, what good practices would you recommand to deal with this particular problem?</p>
  
Tags: <annotations><spectrogram><algorithm>
Title: Manual annotation of events with varying detectability
ThreadText: # Manual annotation of events with varying detectability
  
  (Asked by: username_0 on 2022-09-23)
  
  When creating an algorithm to perform a specific task on audio recordings, it is sometimes needed to first perform the task manually in order to create a reference which can then be used to train and evaluate the algorithm.
  When doing such manual annotations, sometimes the sound events you are annotating have a high variability in detectability (some are more easily detectable than others due to the distance of sound emitters for instance; if using a spectrogram display, some events are sometimes visible but not audible or the other way around).
  
  I would imagine that "human" detectability is related to "algorithm" detectability. That's why it's sometimes tempting to only annotate the events that are salient/loud/detectable enough (and deliberately ignore those which are barely detectable) in order to "help" the algorithm to train.
  
  Is it a good idea to do that? More generally, what good practices would you recommand to deal with this particular problem?
  
  **Tags:** annotations, spectrogram, algorithm
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2022-09-27. Score: 8
  
  I'd agree that these subjective measures are useful. For example, you may test an automatic detectors ability to detect the 'certain' sounds, but when measuring false alarm rate you probably shouldn't penalise yourself if the detector picks up a 'possible'.
  
  However, you might also consider trying to introduce some more objective measured, the obvious one being Signal to Noise Ratio. Humans are very good at picking out weak signals, especially when prompted by the presence of nearby louder ones and it may not be reasonable to expect an automatic detector to find the low SNR calls. So again, judge your detector on it's ability to detect sounds with SNR > ?? dB, but don't penalise your algorithm if it gets a quiet one.
  
  
  ---
  
  ### Answer by username_2 on 2022-09-24. Score: 6
  
  As detectability is really a statistical variable (probability of detection), I would suggest annotating the data with at least levels (certain, probable, possible).
  
  For this I use as criteria (being a non-native speaker):
  
  * certain: there is no doubt on the detection
  * probable: not certain, but high probability
  * possible: may or may not be
  
  As by Doug's answer, yes, a quantitative measure (SNR for example) if provided by the software is an important parameter for annotations. Even if this is not available, because you use spectrograms and have no time to measure values and the software does not do it for you, qualitative measures (strong, average, weak) are also important.
  
  This could yield to such interesting observation like (strong possible click, or is it really an electronic transient) or (weak but certain dolphin whistle)
  
  
  ---
  
  ### Answer by username_3 on 2023-06-05. Score: 1
  
  I have found such annotations to be very useful when working on detection tasks. The practice is not quite standardized, but there are several examples out there. For example:
  
  The [Urbansound8k](https://urbansounddataset.weebly.com/urbansound8k.html) dataset annotates "salience":
  
  A (subjective) salience rating of the sound. 1 = foreground, 2 = background.
  
  The [SONYC-UST](https://zenodo.org/record/3966543) dataset annotates "proximity":
  
  Indicates the proximity of the sound event to the sensor, in 3 possible values: (near, far, notsure) plus Not Available (encoded as -1).
  
  It can be useful to compute some objective metrics based on these annotations.
  For example, to estimate the Signal to Noise ratio (SNR) of the annotated events. Can be done by computing a sound level indicator such as A-weighted Fast/125ms, and then comparing the level of the event to the surrounding/preceeding background level.
  Then one can plot performance metrics such as False Positive / False Negative versus the SNR levels, and also look at the subjective ratings vs detection metrics or vs SNR.
  
  
  ---
  
  