Answers.Body: <p>Dan Stowell’s answer is excellent. I will just clarify that mean-and-variance normalization he described is typically not a per-sample normalization. Instead it involves adjusting the values of each sample with fixed scaling and shifting so that the mean of the entire dataset is 0 and standard deviation is 1. This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T07:12:40.640Z
Answers.Text: Thank you Sam. If you want to, feel free to "edit" my answer
Answers.UserDisplayName: Dan Stowell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T08:25:01.737Z
Answers.Text: Welcome Sam! Thanks for your comment! On StackExchange, this kind of post should be a *comment* addressed below the answer you are talking about. If you cannot comment yet, see [here](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead).
Answers.UserDisplayName: Noil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-29T23:29:40.980Z
Answers.Text: Thanks, should I delete this answer?
Answers.UserDisplayName: Sam Lapp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-31T17:00:52.187Z
Answers.Text: @SamLapp I would say so (or a moderator), but several people upvoted your questions so you may want to wait for a bit in order not to lose some 'priviledges' (comments, etc) ;-) . A tip: tag the name of the person you are talking to in the comments (@Name) so that they receive a notification of your reply (I saw your reply by chance actually :-)
Answers.UserDisplayName: Noil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T02:45:15.613Z
Answers.LastActivityDate: 2022-07-27T02:45:15.613Z
Answers.UserDisplayName: Sam Lapp
Answers.Body: <p>Yes, you should definitely normalise audio data before supplying it to a deep learning model. But don't use &quot;peak normalisation&quot; and don't normalize each sample individually: use either (a) mean-and-variance normalisation, or (b) PCEN.</p>
  <p>More detail follows...</p>
  <p>Firstly, your question refers to &quot;ML model&quot; but it's only <strong>deep</strong> learning, not machine learning in general, where this matters, because of the gradient descent used in training. Many other algorithms (I'm especially thinking of random forests, and similar) are not affected by normalisation.</p>
  <p>The most common approach in audio DL is I think the same as the standard approach in image and other deep learning, which is to use the mean and standard deviation (measured on the training set) to normalise the data. (There's a mathematical justification for this by <a href="https://link.springer.com/chapter/10.1007/978-3-642-35289-8_3" rel="noreferrer">LeCun et al (1998), ``Efficient backprop''</a>. The main lesson is: it's easy for us to calculate-and-remove the <em>statistical moments</em> in the data, so we should do that and let the training algorithm concentrate on the harder part -- it will thus train faster.)</p>
  <blockquote>
  <p><em>Extra detail: Note that this &quot;mean and variance&quot; normalisation is typically not a per-sample normalization. Instead, it involves adjusting the values of every sample by <strong>fixed</strong> scaling and shifting constants so that the mean of the <strong>entire dataset</strong> is 0 and the standard deviation of the entire dataset is 1. (For instance, if the mean pre-normalisation value across all samples is 0.15, you would subtract 0.15 from every sample regardless of the sample's mean.) This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy</em></p>
  </blockquote>
  <p>You should not use peak normalisation. (I don't know why Audiomentations implemented it! The documentation refers to exporting audio, not to training.) Clipping is not a concern for us when doing DL, because the processing in deep learning is usually floating-point, and going beyond [-1, 1] is no problem. On the other hand, peak normalisation is likely to make gradient-based learning worse, because it does not help us to control the mean or the standard deviation of the input data--in fact it's likely to do the opposite since the max and min value can vary so aritrarily. For example, the &quot;impulse noise&quot; issue mentioned in your question.</p>
  <p>Aside from the standard method, there's also &quot;PCEN&quot; (per-channel energy normalisation), which is designed specifically for audio and takes a different approach. PCEN performs a kind of dynamic gain control (a bit like a &quot;multi-band compressor&quot; audio effect) which is designed to ensure that the energy values within each frequency band are mapped to a range that is &quot;well-behaved&quot; for ML. <a href="https://www.lostanlen.com/wp-content/uploads/2019/12/lostanlen2019spl.pdf" rel="noreferrer">Lostanlen et al (2018)</a> give a detailed analysis of this issue, arguing for PCEN. I've seen quite a few bioacoustic DL projects using PCEN as the preprocessing (and no other normalisation) and it seems to be a relatively safe low-regret option.</p>
  <blockquote>
  <p><em>Extra detail: Modern deep learning networks often have normalisation layers embedded inside them (with names such as &quot;BatchNorm&quot;, &quot;LayerNorm&quot;). If these are applied at the input layer then they might sometimes be equivalent to pre-normalising your data... but not always, so I do not recommend skipping the normalisation.</em></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T06:50:48.103Z
Answers.Text: I came across some papers that mean - std normalize the dataset on the frequency bins of the spectrogram. Is this something you consider? Or do you simply normalize the array itself?
Answers.UserDisplayName: Benjamin Cretois
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T07:12:00.207Z
Answers.Text: Yes, doing it per-frequency-bin is pretty common, and can be a good idea. Whether to do it per-bin or on the whole matrix seems to depend somewhat on the situation. (PCEN is effectively doing it per-bin)
Answers.UserDisplayName: Dan Stowell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-10T08:53:40.050Z
Answers.Text: I've been summoned! :)
  I'm upvoting this answer and will add that i have written a paper for EuroNoise 2021 explaining why i think PCEN has potential in bioacoustics, what are some known applications, and what are the next steps. 
  https://hal.archives-ouvertes.fr/hal-03381500/document
  
  I have proposed to describe the PCEN mechanism as "self-calibration" of the sensor so as to reduce the risk of ambiguity with forms of normalization whose parameters are shared across test set samples (e.g. batch norm).
Answers.UserDisplayName: lostanlen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-26T22:00:25.943Z
Answers.LastActivityDate: 2022-07-30T09:13:13.007Z
Answers.LastEditDate: 2022-07-30T09:13:13.007Z
Answers.UserDisplayName: Dan Stowell
Body: <p>In computer vision it seems that image normalization is very important to keep the gradients &quot;in check&quot; (detailed description <a href="https://stats.stackexchange.com/questions/211436/why-normalize-images-by-subtracting-datasets-image-mean-instead-of-the-current">here</a>). However, I was wondering if it was also the case in audio deep learning? Is normalization a necessary step for training efficient models?</p>
  <p>The package <a href="https://github.com/iver56/audiomentations" rel="noreferrer">Audiomentation</a> offer the possibility to do peak normalization with the <code>normalize</code> function which:</p>
  <blockquote>
  <p>Apply a constant amount of gain, so that highest signal level present in the sound becomes
  0 dBFS, i.e. the loudest level allowed if all samples must be between -1 and 1.</p>
  </blockquote>
  <p>However, I believe that peak normalization can induce some problems. For instance, if some audio files have some impulse noise present (close to 0 dBFS), the signal would end up with a very much lower-amplitude signal than expected (i.e. compared to an audio file not containing any impulse noise).</p>
  
Tags: <machine-learning><signal-processing>
Title: Should we normalize audio before training a ML model
ThreadText: # Should we normalize audio before training a ML model
  
  (Asked by: username_0 on 2022-07-26)
  
  In computer vision it seems that image normalization is very important to keep the gradients "in check" (detailed description [here](https://stats.stackexchange.com/questions/211436/why-normalize-images-by-subtracting-datasets-image-mean-instead-of-the-current)). However, I was wondering if it was also the case in audio deep learning? Is normalization a necessary step for training efficient models?
  
  The package [Audiomentation](https://github.com/iver56/audiomentations) offer the possibility to do peak normalization with the `normalize` function which:
  
  > Apply a constant amount of gain, so that highest signal level present in the sound becomes
  > 0 dBFS, i.e. the loudest level allowed if all samples must be between -1 and 1.
  
  However, I believe that peak normalization can induce some problems. For instance, if some audio files have some impulse noise present (close to 0 dBFS), the signal would end up with a very much lower-amplitude signal than expected (i.e. compared to an audio file not containing any impulse noise).
  
  **Tags:** machine-learning, signal-processing
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-26. Score: 10 | Accepted answer
  
  Yes, you should definitely normalise audio data before supplying it to a deep learning model. But don't use "peak normalisation" and don't normalize each sample individually: use either (a) mean-and-variance normalisation, or (b) PCEN.
  
  More detail follows...
  
  Firstly, your question refers to "ML model" but it's only **deep** learning, not machine learning in general, where this matters, because of the gradient descent used in training. Many other algorithms (I'm especially thinking of random forests, and similar) are not affected by normalisation.
  
  The most common approach in audio DL is I think the same as the standard approach in image and other deep learning, which is to use the mean and standard deviation (measured on the training set) to normalise the data. (There's a mathematical justification for this by [LeCun et al (1998), ``Efficient backprop''](https://link.springer.com/chapter/10.1007/978-3-642-35289-8_3). The main lesson is: it's easy for us to calculate-and-remove the *statistical moments* in the data, so we should do that and let the training algorithm concentrate on the harder part -- it will thus train faster.)
  
  > *Extra detail: Note that this "mean and variance" normalisation is typically not a per-sample normalization. Instead, it involves adjusting the values of every sample by **fixed** scaling and shifting constants so that the mean of the **entire dataset** is 0 and the standard deviation of the entire dataset is 1. (For instance, if the mean pre-normalisation value across all samples is 0.15, you would subtract 0.15 from every sample regardless of the sample's mean.) This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy*
  
  You should not use peak normalisation. (I don't know why Audiomentations implemented it! The documentation refers to exporting audio, not to training.) Clipping is not a concern for us when doing DL, because the processing in deep learning is usually floating-point, and going beyond [-1, 1] is no problem. On the other hand, peak normalisation is likely to make gradient-based learning worse, because it does not help us to control the mean or the standard deviation of the input data--in fact it's likely to do the opposite since the max and min value can vary so aritrarily. For example, the "impulse noise" issue mentioned in your question.
  
  Aside from the standard method, there's also "PCEN" (per-channel energy normalisation), which is designed specifically for audio and takes a different approach. PCEN performs a kind of dynamic gain control (a bit like a "multi-band compressor" audio effect) which is designed to ensure that the energy values within each frequency band are mapped to a range that is "well-behaved" for ML. [Lostanlen et al (2018)](https://www.username_4.com/wp-content/uploads/2019/12/username_42019spl.pdf) give a detailed analysis of this issue, arguing for PCEN. I've seen quite a few bioacoustic DL projects using PCEN as the preprocessing (and no other normalisation) and it seems to be a relatively safe low-regret option.
  
  > *Extra detail: Modern deep learning networks often have normalisation layers embedded inside them (with names such as "BatchNorm", "LayerNorm"). If these are applied at the input layer then they might sometimes be equivalent to pre-normalising your data... but not always, so I do not recommend skipping the normalisation.*
  
  > **Comments:**
  > **username_0**: I came across some papers that mean - std normalize the dataset on the frequency bins of the spectrogram. Is this something you consider? Or do you simply normalize the array itself?  
  > **username_2**: Yes, doing it per-frequency-bin is pretty common, and can be a good idea. Whether to do it per-bin or on the whole matrix seems to depend somewhat on the situation. (PCEN is effectively doing it per-bin)  
  > **username_4**: I've been summoned! :) I'm upvoting this answer and will add that i have written a paper for EuroNoise 2021 explaining why i think PCEN has potential in bioacoustics, what are some known applications, and what are the next steps.  https://hal.archives-ouvertes.fr/hal-03381500/document  I have proposed to describe the PCEN mechanism as "self-calibration" of the sensor so as to reduce the risk of ambiguity with forms of normalization whose parameters are shared across test set samples (e.g. batch norm).  
  
  ---
  
  ### Answer by username_1 on 2022-07-27. Score: 4
  
  username_2’s answer is excellent. I will just clarify that mean-and-variance normalization he described is typically not a per-sample normalization. Instead it involves adjusting the values of each sample with fixed scaling and shifting so that the mean of the entire dataset is 0 and standard deviation is 1. This avoids the issue of quiet or empty samples being normalized to have maximal values and becoming loud and noisy.
  
  > **Comments:**
  > **username_2**: Thank you Sam. If you want to, feel free to "edit" my answer  
  > **username_3**: Welcome Sam! Thanks for your comment! On StackExchange, this kind of post should be a *comment* addressed below the answer you are talking about. If you cannot comment yet, see [here](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead).  
  > **username_1**: Thanks, should I delete this answer?  
  > **username_3**: @SamLapp I would say so (or a moderator), but several people upvoted your questions so you may want to wait for a bit in order not to lose some 'priviledges' (comments, etc) ;-) . A tip: tag the name of the person you are talking to in the comments (@Name) so that they receive a notification of your reply (I saw your reply by chance actually :-)  
  
  ---
  
  