Answers.Body: <p>I use OpenSoundscape (Python package developed by Tessa Rhinehart &amp; the Kitzes lab at Univ of Pittsburgh), which has a lot of augmentation/pre-processing tools. Their documentation &amp; tutorials are also amazing for the package &amp; associated functions. If you haven't tried it out yet it's pretty nice.</p>
  <p>In particular, I've used the overlay function to &quot;combine&quot; 2 files for, in my specific use case, adding rain to non-rain files. I have recordings from the Malagasy rainforest that I'm using ML models for to detect lemur calls but the model is frequently confused by rain and it's challenging to find enough clips of calls in the rain for the training dataset (both because lemurs just call less frequently when it's raining and rain decreases the detection distance of calls because of the high noise levels it creates). So, I'm trying out overlaying rain onto existing lemur call clips, similar to your &quot;adding noise&quot; bullet point, except that the noise I'm picking is not random, it's targeted. It would be interesting to see how those different strategies worked out though.</p>
  <p>In terms of the random flip, it sounds like the spectrogram image would be inverted at random stages within a recording - am I understanding that correctly? I think it would depend on the bandwidth and frequency structure of the sound as well. If you have calls that have high power at low frequencies and low power at high frequencies I'm not sure inverting that would help the model. Perhaps more so with calls that are more variable rather than stereotyped, as you mention.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-30T15:49:26.963Z
Answers.LastActivityDate: 2022-06-30T15:49:26.963Z
Answers.UserDisplayName: Carly Batist
Answers.Body: <p>I <strong>do not recommend random flip or any kind of time-axis flipping</strong>. In the context of animal sound, this could easily change the &quot;meaning&quot; of the signal, in a way which causes problems for your data. Temporal flipping reverses the sequence of calls, but also the temporal pattern <em>within</em> each call. In a deep learning context, the outcome might be &quot;OK&quot; but it might also lead to your target class being less distinctive for the classifier.</p>
  <p><a href="http://ceur-ws.org/Vol-2125/paper_140.pdf" rel="noreferrer">Mario Lasseck's 2018 BirdClef paper</a> gives a very clear list of augmentations, and an evaluation of which ones give the most benefit for bird species classification.</p>
  <p>In my lab we use data augmentation a lot, most often for bird sounds but for many other terrestrial sounds. We use <a href="https://github.com/iver56/audiomentations" rel="noreferrer">the audiomentations Python library</a> which makes it easy to apply various modifications. The README gives a good list of the effects and what they do.</p>
  <p>For bioacoustic data augmentation I would always recommend:</p>
  <ul>
  <li><strong>Time-shifting</strong></li>
  <li>Mixing in some <strong>background</strong> &quot;silence&quot; recorded in the wild (i.e. ambient soundscape that doesn't contain your target classes).</li>
  <li>You can even go further and <strong>mix together <em>any</em> two sound files</strong>, if you're using multi-label classification/detection and you take care that the resulting recording also has the right annotation (e.g. two different species presence). Some people refer to this as <a href="https://arxiv.org/abs/1710.09412" rel="noreferrer">&quot;<strong>mixup</strong>&quot; in image recognition</a> (see also this <a href="https://twitter.com/mclduk/status/1268495995626807297" rel="noreferrer">twitter comment about &quot;maxup&quot;</a>). Combined with temporal shifting this yields a lot of data diversity despite very little signal processing.</li>
  <li><strong>Gentle modifications to the frequency equalisation</strong> (this is similar to slightly changing the response characteristics of your mic), achieved e.g. by convolving with an impulse response, or some gentle use of band-stop filtering or <code>SevenBandParametricEQ</code>.</li>
  <li>Closely related to frequency EQ is emulating the effects of <strong>distance</strong> by adding a bit of low-pass filtering, and/or some <strong>echo/reverb</strong> using an impulse response. Audiomentations has a specific filter for atmospheric absorption called <code>AirAbsorption</code>.</li>
  <li>Adding a little <strong>white noise</strong> is OK (and easy) but I prefer to use ambient noise as above. Or consider pink noise, since the white noise (==equal power at all linearly-spaced frequency bands) might dominate in some &quot;quiet&quot; bands of your signal. In speech tech there's the idea of &quot;speech-shaped noise&quot; (having the same spectral profile as the target signal) - perhaps someone should implement this in the software libraries...</li>
  <li>Time masking and frequency masking -- these don't have a very obvious interpretation but are a kind of &quot;dropout&quot; of frequencies or time regions, and seem to help.</li>
  </ul>
  <p>In almost all bioacoustic scenarios the above list is unlikely to damage the bioacoustic &quot;semantics&quot; of the signal. Frequency-shifting could change the meaning of a sound; so could rearranging the units within a sequence.</p>
  <p>Be careful not to adopt methods directly from image-processing without thinking them through. The scariest example is &quot;image rotation&quot; at randomly-selected angles which would have bizarre consequences for your spectrogram data...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-18T08:08:35.007Z
Answers.Text: Pink noise can now be added with the help of `AddColorNoise` in audiomentations>=0.35.0
  
  Documentation here:
  https://iver56.github.io/audiomentations/waveform_transforms/add_color_noise/
Answers.UserDisplayName: iver56
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-04T07:11:21.943Z
Answers.Text: Mixup usually refers to the last data augmentation in your list, i.e. the combination of the features + their corresponding labels. Reference [here](https://arxiv.org/abs/1710.09412).
Answers.UserDisplayName: jul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-04T07:19:22.750Z
Answers.Text: Thank you very much for your answer! Is there any pink noise library you are using in particular?
Answers.UserDisplayName: Benjamin Cretois
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-04T07:24:27.650Z
Answers.Text: I would add time masking and frequency masking, as well as a specific filtering that has been shown to give good result: low-pass filtering, which (naively) simulates the effect of distance. More generally, in order to select the data augmentation methods, I find it useful to think about what the message should be robust to in the field (e.g. distance, environmental noise such as rain or wind, short masking noise such as lighting strikes...).
Answers.UserDisplayName: jul
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-04T10:05:08.460Z
Answers.Text: I don't see PinkNoise in standard libraries, but it's pretty easy to implement. Also in speech tech there's the idea of "speech-shaped noise" (having the same spectral profile as the target signal) - perhaps someone should implement this in the libraries.
  @jul thanks, your comments are correct - I should edit my answer when I have a moment.
Answers.UserDisplayName: Dan Stowell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-04T06:51:24.357Z
Answers.LastActivityDate: 2022-07-04T11:09:00.497Z
Answers.LastEditDate: 2022-07-04T11:09:00.497Z
Answers.UserDisplayName: Dan Stowell
Body: <p>When training a Neural Network for bioacoustics classification tasks I usually follow a data augmentation pipeline which is composed of:</p>
  <ul>
  <li><p><strong>Changes in sound to noise ratio</strong>: for instance, if my sound track is composed of a bird call and some background noise I would change the SNR so that bird call might appear louder than on the original track.</p>
  </li>
  <li><p><strong>Adding some random noise</strong>: Adding a bit of white noise to the sound track. My idea in this augmentation is that since each recorder device brands have some specific &quot;base sounds&quot; (I'm not sure of the exact term), adding random noise might reduce the chances of the model overfitting on certain devices.</p>
  </li>
  <li><p><strong>Random crop and padding</strong>: Since neural networks require a fixed-size input, I crop a part of the audio segment I am interested in and add some padding to the size of the spectrogram is coherent with the model's first layer. Usually, on a segment of 4 seconds I would crop a random segment of 3 seconds and add 1 second of padding.</p>
  </li>
  </ul>
  <p>Some recent papers I have read also use an augmentation technique most use in the visual domain, namely <strong>random flip or time inversion</strong> define as:</p>
  <blockquote>
  <p>“Inversion of a track along its time axis relates to the random flip of an image, which is an augmentation technique that is widely used in the visual domain.” <a href="https://Guzhov,%20A.,%20Raue,%20F.,%20Hees,%20J.,%20&amp;%20Dengel,%20A.%20(2022,%20May).%20Audioclip:%20Extending%20clip%20to%20image,%20text%20and%20audio.%20In%20ICASSP%202022-2022%20IEEE%20International%20Conference%20on%20Acoustics,%20Speech%20and%20Signal%20Processing%20(ICASSP)%20(pp.%20976-980).%20IEEE." rel="noreferrer">Guzhov et al., 2021</a></p>
  </blockquote>
  <p>Even though the authors suggest that this augmentation improved the performance of the model, I am unsure how much sense it make to add this augmentation type. For instance, bird calls always folow the same sequence and using <strong>random flip</strong> on a bird call might cause the call sequence to be &quot;broken&quot;.</p>
  <p>Does anyone have any thought on <strong>random flip</strong>? And more generally, is there other augmentation strategies you are using in your pipeline?</p>
  
Tags: <methods><bird><machine-learning><data-augmentation>
Title: Data augmentation strategies for bioacoustics machine learning
ThreadText: # Data augmentation strategies for bioacoustics machine learning
  
  (Asked by: username_0 on 2022-06-30)
  
  When training a Neural Network for bioacoustics classification tasks I usually follow a data augmentation pipeline which is composed of:
  
  * **Changes in sound to noise ratio**: for instance, if my sound track is composed of a bird call and some background noise I would change the SNR so that bird call might appear louder than on the original track.
  * **Adding some random noise**: Adding a bit of white noise to the sound track. My idea in this augmentation is that since each recorder device brands have some specific "base sounds" (I'm not sure of the exact term), adding random noise might reduce the chances of the model overfitting on certain devices.
  * **Random crop and padding**: Since neural networks require a fixed-size input, I crop a part of the audio segment I am interested in and add some padding to the size of the spectrogram is coherent with the model's first layer. Usually, on a segment of 4 seconds I would crop a random segment of 3 seconds and add 1 second of padding.
  
  Some recent papers I have read also use an augmentation technique most use in the visual domain, namely **random flip or time inversion** define as:
  
  > “Inversion of a track along its time axis relates to the random flip of an image, which is an augmentation technique that is widely used in the visual domain.” [Guzhov et al., 2021](https://Guzhov,%20A.,%20Raue,%20F.,%20Hees,%20J.,%20&%20Dengel,%20A.%20(2022,%20May).%20Audioclip:%20Extending%20clip%20to%20image,%20text%20and%20audio.%20In%20ICASSP%202022-2022%20IEEE%20International%20Conference%20on%20Acoustics,%20Speech%20and%20Signal%20Processing%20(ICASSP)%20(pp.%20976-980).%20IEEE.)
  
  Even though the authors suggest that this augmentation improved the performance of the model, I am unsure how much sense it make to add this augmentation type. For instance, bird calls always folow the same sequence and using **random flip** on a bird call might cause the call sequence to be "broken".
  
  Does anyone have any thought on **random flip**? And more generally, is there other augmentation strategies you are using in your pipeline?
  
  **Tags:** methods, bird, machine-learning, data-augmentation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-04. Score: 9 | Accepted answer
  
  I **do not recommend random flip or any kind of time-axis flipping**. In the context of animal sound, this could easily change the "meaning" of the signal, in a way which causes problems for your data. Temporal flipping reverses the sequence of calls, but also the temporal pattern *within* each call. In a deep learning context, the outcome might be "OK" but it might also lead to your target class being less distinctive for the classifier.
  
  [Mario Lasseck's 2018 BirdClef paper](http://ceur-ws.org/Vol-2125/paper_140.pdf) gives a very clear list of augmentations, and an evaluation of which ones give the most benefit for bird species classification.
  
  In my lab we use data augmentation a lot, most often for bird sounds but for many other terrestrial sounds. We use [the audiomentations Python library](https://github.com/username_3/audiomentations) which makes it easy to apply various modifications. The README gives a good list of the effects and what they do.
  
  For bioacoustic data augmentation I would always recommend:
  
  * **Time-shifting**
  * Mixing in some **background** "silence" recorded in the wild (i.e. ambient soundscape that doesn't contain your target classes).
  * You can even go further and **mix together *any* two sound files**, if you're using multi-label classification/detection and you take care that the resulting recording also has the right annotation (e.g. two different species presence). Some people refer to this as ["**mixup**" in image recognition](https://arxiv.org/abs/1710.09412) (see also this [twitter comment about "maxup"](https://twitter.com/mclduk/status/1268495995626807297)). Combined with temporal shifting this yields a lot of data diversity despite very little signal processing.
  * **Gentle modifications to the frequency equalisation** (this is similar to slightly changing the response characteristics of your mic), achieved e.g. by convolving with an impulse response, or some gentle use of band-stop filtering or `SevenBandParametricEQ`.
  * Closely related to frequency EQ is emulating the effects of **distance** by adding a bit of low-pass filtering, and/or some **echo/reverb** using an impulse response. Audiomentations has a specific filter for atmospheric absorption called `AirAbsorption`.
  * Adding a little **white noise** is OK (and easy) but I prefer to use ambient noise as above. Or consider pink noise, since the white noise (==equal power at all linearly-spaced frequency bands) might dominate in some "quiet" bands of your signal. In speech tech there's the idea of "speech-shaped noise" (having the same spectral profile as the target signal) - perhaps someone should implement this in the software libraries...
  * Time masking and frequency masking -- these don't have a very obvious interpretation but are a kind of "dropout" of frequencies or time regions, and seem to help.
  
  In almost all bioacoustic scenarios the above list is unlikely to damage the bioacoustic "semantics" of the signal. Frequency-shifting could change the meaning of a sound; so could rearranging the units within a sequence.
  
  Be careful not to adopt methods directly from image-processing without thinking them through. The scariest example is "image rotation" at randomly-selected angles which would have bizarre consequences for your spectrogram data...
  
  > **Comments:**
  > **username_4**: Mixup usually refers to the last data augmentation in your list, i.e. the combination of the features + their corresponding labels. Reference [here](https://arxiv.org/abs/1710.09412).  
  > **username_0**: Thank you very much for your answer! Is there any pink noise library you are using in particular?  
  > **username_4**: I would add time masking and frequency masking, as well as a specific filtering that has been shown to give good result: low-pass filtering, which (naively) simulates the effect of distance. More generally, in order to select the data augmentation methods, I find it useful to think about what the message should be robust to in the field (e.g. distance, environmental noise such as rain or wind, short masking noise such as lighting strikes...).  
  > **username_2**: I don't see PinkNoise in standard libraries, but it's pretty easy to implement. Also in speech tech there's the idea of "speech-shaped noise" (having the same spectral profile as the target signal) - perhaps someone should implement this in the libraries. @username_4 thanks, your comments are correct - I should edit my answer when I have a moment.  
  > **username_3**: Pink noise can now be added with the help of `AddColorNoise` in audiomentations>=0.35.0  Documentation here: https://username_3.github.io/audiomentations/waveform_transforms/add_color_noise/  
  
  ---
  
  ### Answer by username_1 on 2022-06-30. Score: 4
  
  I use OpenSoundscape (Python package developed by Tessa Rhinehart & the Kitzes lab at Univ of Pittsburgh), which has a lot of augmentation/pre-processing tools. Their documentation & tutorials are also amazing for the package & associated functions. If you haven't tried it out yet it's pretty nice.
  
  In particular, I've used the overlay function to "combine" 2 files for, in my specific use case, adding rain to non-rain files. I have recordings from the Malagasy rainforest that I'm using ML models for to detect lemur calls but the model is frequently confused by rain and it's challenging to find enough clips of calls in the rain for the training dataset (both because lemurs just call less frequently when it's raining and rain decreases the detection distance of calls because of the high noise levels it creates). So, I'm trying out overlaying rain onto existing lemur call clips, similar to your "adding noise" bullet point, except that the noise I'm picking is not random, it's targeted. It would be interesting to see how those different strategies worked out though.
  
  In terms of the random flip, it sounds like the spectrogram image would be inverted at random stages within a recording - am I understanding that correctly? I think it would depend on the bandwidth and frequency structure of the sound as well. If you have calls that have high power at low frequencies and low power at high frequencies I'm not sure inverting that would help the model. Perhaps more so with calls that are more variable rather than stereotyped, as you mention.
  
  
  ---
  
  