Answers.Body: <p>I don't think there is an easy answer yet; it feels as if the heuristics are still very much emerging as the field matures. If you're using BirdNET data downstream in an occupancy model, I have found inspiration in this recent publication from <a href="https://academic.oup.com/condor/article-abstract/124/2/duac003/6572065" rel="noreferrer">Cole et al. 2022</a>. It illustrates the reality of tradeoffs in true positives and false positives under varying thresholds and provides some precedent for how to deal with these. Recent work by <a href="https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13905" rel="noreferrer">Rhinehart et al. 2022</a> may also be relevant with respect to incorporating uncertain automated detection data into an occupancy framework.</p>
  <p>One suggestion I've heard is to require a minimum number of detections within a timestep of interest, to minimize the chance that non-target noise is triggering false alarms. This still doesn't answer the stickiest part of the question -- how to choose what number of detections above what confidence level over what timestep?</p>
  <p>Since it may be inadvisable to trust results without some degree of expert review, one method for conducting verifications quickly is to conduct &quot;top-down listening&quot; (where clips are sorted from highest to lowest confidence to quickly assess species presence). I'm dreaming about an eventual community science approach to solicit expedient BirdNET validations from local site experts, which would require an accessible platform and clear training heuristics guiding these field experts in how to verify (for example, how to verify songs vs. calls? And does it count as a correct detection if only a very small portion of the signal of interest occurs within the 3-second window?).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T06:56:40.983Z
Answers.Text: Thanks, those are useful pointers.
Answers.UserDisplayName: Carlos Abrahams
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-05T22:24:51.207Z
Answers.LastActivityDate: 2022-07-05T22:24:51.207Z
Answers.UserDisplayName: Cathleen B
Answers.Body: <p>I have found a published article that addresses this question.</p>
  <p>Toenies, M &amp; Rich, L.N (2021) Advancing bird survey efforts through novel recorder technology and automated species identification California Fish and Wildlife 107(2):56-70 state this:</p>
  <blockquote>
  <p>&quot;First, we removed species that only had a single detection across all five recordings for the location since these were more likely to represent misidentifications or species flying over but not occupying the location. Second, we subsetted BirdNET output based on two parameters that it assigns for every identification: 1) confidence, indicating the degree of confidence BirdNET has in each species identification (on a scale where 0 represents lowest confidence and 1 represents highest confidence); and 2) rank, which indicates the species with the highest confidence value when BirdNET identifies multiple possible species. We chose to only include detections if BirdNET assigned them a Rank of 1 and a Confidence value of 0.95 or higher so that we would retain only the highest confidence detections. Finally, we excluded purported detections of diurnal species if they were detected during the nighttime (2100 to 0430 PDT). We did this to correct for BirdNET’s tendency to produce high-confidence false positive
  detections at higher rates during this period (often due to apparent misidentifications of rustling vegetation or vocalizations from nocturnal animals). We believe that excluding these purported detections reduced false positive identifications without compromising our ability to detect these species because any diurnal species acoustically active at a location
  should be more active outside nighttime hours&quot;.</p>
  </blockquote>
  <p>They seem like good ideas to me..</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-18T23:30:34.877Z
Answers.Text: One thing to note is that Toenies, M & Rich, L.N (2021) used BirdNET 1.0 not the more recent BirdNET 2.1 and thus the threshold suggestions etc they made are outdated.
Answers.UserDisplayName: Abram Fleishman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T22:09:38.513Z
Answers.Text: @Abram Fleishman, this is a great point and something I'm thinking about in my BirdNET work. If we come up with useful verification thresholds in our research, will they essentially be obsolete as soon as a new BirdNET model version comes out? In which case, in a long-term monitoring program, do we stick with the old model for which we've been already evaluating BirdNET's accuracy, or cut losses and start using the latest model?
Answers.UserDisplayName: Cathleen B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T19:51:34.737Z
Answers.Text: @CathleenB That is a challenge we have been experiencing for a long time and do not have a great answer for how to address it yet.  Essentially, if you analysis relies on manual vetting of ML predictions, every time to apply a new model to a dataset, you have to go back and re-review the predictions for new detections.
Answers.UserDisplayName: Abram Fleishman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-17T11:27:46.123Z
Answers.LastActivityDate: 2022-07-17T11:27:46.123Z
Answers.UserDisplayName: Carlos Abrahams
Answers.Body: <p>I happen to have two HW implementations of BirdNet called <a href="https://haikubox.com/" rel="nofollow noreferrer">Haikubox</a> that classify the birds in situ. As I have no real knowledge on birds, my heuristic is something like this:</p>
  <p>I would call it a false positive</p>
  <ul>
  <li>If I see a bird classified out of geographic context, say Cormorant far away from sea and rivers.</li>
  <li>if there are only few occurrences of the bird with low SNR in the spectrogram</li>
  <li>if I hear in the audio typical anthropogenic noise masking the bird call/song (seems humans are still good classifiers)</li>
  </ul>
  <p>I assume experts on bird sound and distributions may have more criteria that would indicate false positives</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T05:49:10.850Z
Answers.LastActivityDate: 2022-07-06T05:49:10.850Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>I haven't worked directly with BirdNet but the last time I worked with large-scale ML classification, we'd set a threshold to manually review for false positives down to some confidence level, with that threshold varying significantly depending on the research question involved. Some detection tasks require an approach where you need to get every last true positive, even those at low confidence, even if it means wading through a number of false positives. For others, you can simply assume that every event above some match threshold is a positive and even if that does involve a few false positives you'll still end up with useful information to answer your ecological question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-06T14:19:22.653Z
Answers.LastActivityDate: 2022-07-06T14:19:22.653Z
Answers.UserDisplayName: dtsavage
Body: <p>I'm starting to use BirdNET,  both downloaded from Github at <a href="https://github.com/kahst/BirdNET-Analyzer" rel="noreferrer">https://github.com/kahst/BirdNET-Analyzer</a> onto my latop, and on a BirdNET-Pi installation.</p>
  <p>I'm getting some fairly obvious false positives, plus a range of other suspect classifications. No criticism intended here - this is absolutely to be expected.  I am interested, though, if anyone is developing their own 'rules' for rapidly processing data to deal with incorrect classifications in terms of numbers of calls detected or % match, i.e. if you only detect one call with 60% match, it's probably safe to discount it, but where is a useful upper limit above this?  Any ideas/discussion would be appreciated.</p>
  
Tags: <classifier><birdnet>
Title: BirdNET false-positives - any heuristics?
ThreadText: # BirdNET false-positives - any heuristics?
  
  (Asked by: username_0 on 2022-07-05)
  
  I'm starting to use BirdNET, both downloaded from Github at <https://github.com/kahst/BirdNET-Analyzer> onto my latop, and on a BirdNET-Pi installation.
  
  I'm getting some fairly obvious false positives, plus a range of other suspect classifications. No criticism intended here - this is absolutely to be expected. I am interested, though, if anyone is developing their own 'rules' for rapidly processing data to deal with incorrect classifications in terms of numbers of calls detected or % match, i.e. if you only detect one call with 60% match, it's probably safe to discount it, but where is a useful upper limit above this? Any ideas/discussion would be appreciated.
  
  **Tags:** classifier, birdnet
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2022-07-05. Score: 6
  
  I don't think there is an easy answer yet; it feels as if the heuristics are still very much emerging as the field matures. If you're using BirdNET data downstream in an occupancy model, I have found inspiration in this recent publication from [Cole et al. 2022](https://academic.oup.com/condor/article-abstract/124/2/duac003/6572065). It illustrates the reality of tradeoffs in true positives and false positives under varying thresholds and provides some precedent for how to deal with these. Recent work by [Rhinehart et al. 2022](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13905) may also be relevant with respect to incorporating uncertain automated detection data into an occupancy framework.
  
  One suggestion I've heard is to require a minimum number of detections within a timestep of interest, to minimize the chance that non-target noise is triggering false alarms. This still doesn't answer the stickiest part of the question -- how to choose what number of detections above what confidence level over what timestep?
  
  Since it may be inadvisable to trust results without some degree of expert review, one method for conducting verifications quickly is to conduct "top-down listening" (where clips are sorted from highest to lowest confidence to quickly assess species presence). I'm dreaming about an eventual community science approach to solicit expedient BirdNET validations from local site experts, which would require an accessible platform and clear training heuristics guiding these field experts in how to verify (for example, how to verify songs vs. calls? And does it count as a correct detection if only a very small portion of the signal of interest occurs within the 3-second window?).
  
  > **Comments:**
  > **username_0**: Thanks, those are useful pointers.  
  
  ---
  
  ### Answer by username_0 on 2022-07-17. Score: 6
  
  I have found a published article that addresses this question.
  
  Toenies, M & Rich, L.N (2021) Advancing bird survey efforts through novel recorder technology and automated species identification California Fish and Wildlife 107(2):56-70 state this:
  
  > "First, we removed species that only had a single detection across all five recordings for the location since these were more likely to represent misidentifications or species flying over but not occupying the location. Second, we subsetted BirdNET output based on two parameters that it assigns for every identification: 1) confidence, indicating the degree of confidence BirdNET has in each species identification (on a scale where 0 represents lowest confidence and 1 represents highest confidence); and 2) rank, which indicates the species with the highest confidence value when BirdNET identifies multiple possible species. We chose to only include detections if BirdNET assigned them a Rank of 1 and a Confidence value of 0.95 or higher so that we would retain only the highest confidence detections. Finally, we excluded purported detections of diurnal species if they were detected during the nighttime (2100 to 0430 PDT). We did this to correct for BirdNET’s tendency to produce high-confidence false positive
  > detections at higher rates during this period (often due to apparent misidentifications of rustling vegetation or vocalizations from nocturnal animals). We believe that excluding these purported detections reduced false positive identifications without compromising our ability to detect these species because any diurnal species acoustically active at a location
  > should be more active outside nighttime hours".
  
  They seem like good ideas to me..
  
  > **Comments:**
  > **username_2**: One thing to note is that Toenies, M & Rich, L.N (2021) used BirdNET 1.0 not the more recent BirdNET 2.1 and thus the threshold suggestions etc they made are outdated.  
  > **username_1**: @username_2, this is a great point and something I'm thinking about in my BirdNET work. If we come up with useful verification thresholds in our research, will they essentially be obsolete as soon as a new BirdNET model version comes out? In which case, in a long-term monitoring program, do we stick with the old model for which we've been already evaluating BirdNET's accuracy, or cut losses and start using the latest model?  
  > **username_2**: @CathleenB That is a challenge we have been experiencing for a long time and do not have a great answer for how to address it yet.  Essentially, if you analysis relies on manual vetting of ML predictions, every time to apply a new model to a dataset, you have to go back and re-review the predictions for new detections.  
  
  ---
  
  ### Answer by username_3 on 2022-07-06. Score: 3
  
  I happen to have two HW implementations of BirdNet called [Haikubox](https://haikubox.com/) that classify the birds in situ. As I have no real knowledge on birds, my heuristic is something like this:
  
  I would call it a false positive
  
  * If I see a bird classified out of geographic context, say Cormorant far away from sea and rivers.
  * if there are only few occurrences of the bird with low SNR in the spectrogram
  * if I hear in the audio typical anthropogenic noise masking the bird call/song (seems humans are still good classifiers)
  
  I assume experts on bird sound and distributions may have more criteria that would indicate false positives
  
  
  ---
  
  ### Answer by username_4 on 2022-07-06. Score: 2
  
  I haven't worked directly with BirdNet but the last time I worked with large-scale ML classification, we'd set a threshold to manually review for false positives down to some confidence level, with that threshold varying significantly depending on the research question involved. Some detection tasks require an approach where you need to get every last true positive, even those at low confidence, even if it means wading through a number of false positives. For others, you can simply assume that every event above some match threshold is a positive and even if that does involve a few false positives you'll still end up with useful information to answer your ecological question.
  
  
  ---
  
  