Answers.Body: <p>This idea is sometimes applied in computer vision, under the name of <a href="https://en.wikipedia.org/wiki/Whitening_transformation" rel="nofollow noreferrer">Whitening Transform</a>, or ZCA sphering transform. The name whitening comes from signal processing, since removing correlation from a signal makes it look like white noise.</p>
  <p>In images this transformation is applied to remove contrast and enhance edges, as you can see in the example below (taken from this <a href="https://ieeexplore.ieee.org/document/7808140?reload=true" rel="nofollow noreferrer">paper</a>, not open access unfortunately, but the authors made a <a href="https://github.com/kuntalkumarpal/Preprocessing-Image-Classification-CNN" rel="nofollow noreferrer">github repo</a>)</p>
  <img src="https://i.sstatic.net/mbVPv.png" width="300" />
  <img src="https://i.sstatic.net/Ahc35.png" width="305" />
  <p>I'm not aware either about application of covariance matrix in domains other than computer vision for preprocessing, but I can image few points that make this approach unpractical:</p>
  <ul>
  <li><strong>Feature selection</strong>: it's easier to select features that are already uncorrelated (checking trough chi squared test or similar) rather than turning them into uncorrelated ones. And this approach has the advantage of reducing training features and therefore speed up training and reduce memory requirements.</li>
  <li><strong>Impracticality</strong>, for thousands of training features computing the covariance matrix become basically impossible, not just inefficient. This problem becomes relevant even with less features if we want to train with batches.</li>
  <li><strong>Generalization</strong> It seems to me your also suggesting the possibility to compute a unique covariance matrix from a set of training data. This approach would force us apply the same covariance matrix on every out of training data on which we want to apply our model, meaning that after applying that covariance matrix our data will probably not be truly covariance free. This is a problem common to standardization as well (the mean and variance computed on the training data always differ from the real mean and std of the population, but it's easy to compute mean and std of millions of training instances, so usually those estimation are quite robust).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-04T21:00:08.800Z
Answers.LastActivityDate: 2022-02-04T21:00:08.800Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>For machine learning tasks, it's common to normalize input data by subtracting the mean <span class="math-container">$\mu$</span> and dividing by the standard deviation <span class="math-container">$\sigma$</span> of the dataset:</p>
  <p><span class="math-container">$$\hat{x_i} = \frac{x_i - \mu}{\sigma}$$</span></p>
  <p>—where the standard deviation is computed independently for each feature.</p>
  <p>Suppose instead we compute another multidimensional generalization of the standard deviation—the matrix square root <span class="math-container">$\Sigma$</span> of the covariance matrix:</p>
  <p><span class="math-container">$$\Sigma\Sigma^T = \frac{1}{n}\sum_i^n (x_i - \mu)(x_i - \mu)^T$$</span></p>
  <p>(Since covariance matrix <span class="math-container">$\Sigma\Sigma^T$</span> is positive semidefinite, it has a unique positive semidefinite square root <span class="math-container">$\Sigma$</span>.)</p>
  <p>And then normalize the samples by</p>
  <p><span class="math-container">$$\hat{x_i} = \Sigma^{-1}(x_i - \mu)$$</span></p>
  <p>In a sense this is a &quot;better&quot; normalization—it will not only map the values to nice ranges, but also decorrelate different features.</p>
  <p>However, I have never seen this done. Is there some non-obvious drawback to this approach I'm not seeing, aside from being more computationally intensive? Is there any research on what effect this would have on model training?</p>
  
Tags: <machine-learning><data-preprocessing><statistics><normalisation>
Title: Which generalization of standard deviation to use for multidimensional input normalization
UserDisplayName: yuri kilochek
ThreadText: # Which generalization of standard deviation to use for multidimensional input normalization
  
  (Asked by: username_0 on 2022-02-04)
  
  For machine learning tasks, it's common to normalize input data by subtracting the mean $\mu$ and dividing by the standard deviation $\sigma$ of the dataset:
  
  $$\hat{x\_i} = \frac{x\_i - \mu}{\sigma}$$
  
  —where the standard deviation is computed independently for each feature.
  
  Suppose instead we compute another multidimensional generalization of the standard deviation—the matrix square root $\Sigma$ of the covariance matrix:
  
  $$\Sigma\Sigma^T = \frac{1}{n}\sum\_i^n (x\_i - \mu)(x\_i - \mu)^T$$
  
  (Since covariance matrix $\Sigma\Sigma^T$ is positive semidefinite, it has a unique positive semidefinite square root $\Sigma$.)
  
  And then normalize the samples by
  
  $$\hat{x\_i} = \Sigma^{-1}(x\_i - \mu)$$
  
  In a sense this is a "better" normalization—it will not only map the values to nice ranges, but also decorrelate different features.
  
  However, I have never seen this done. Is there some non-obvious drawback to this approach I'm not seeing, aside from being more computationally intensive? Is there any research on what effect this would have on model training?
  
  **Tags:** machine-learning, data-preprocessing, statistics, normalisation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-04. Score: 2 | Accepted answer
  
  This idea is sometimes applied in computer vision, under the name of [Whitening Transform](https://en.wikipedia.org/wiki/Whitening_transformation), or ZCA sphering transform. The name whitening comes from signal processing, since removing correlation from a signal makes it look like white noise.
  
  In images this transformation is applied to remove contrast and enhance edges, as you can see in the example below (taken from this [paper](https://ieeexplore.ieee.org/document/7808140?reload=true), not open access unfortunately, but the authors made a [github repo](https://github.com/kuntalkumarpal/Preprocessing-Image-Classification-CNN))
  
  ![](https://i.sstatic.net/mbVPv.png)
  ![](https://i.sstatic.net/Ahc35.png)
  
  I'm not aware either about application of covariance matrix in domains other than computer vision for preprocessing, but I can image few points that make this approach unpractical:
  
  * **Feature selection**: it's easier to select features that are already uncorrelated (checking trough chi squared test or similar) rather than turning them into uncorrelated ones. And this approach has the advantage of reducing training features and therefore speed up training and reduce memory requirements.
  * **Impracticality**, for thousands of training features computing the covariance matrix become basically impossible, not just inefficient. This problem becomes relevant even with less features if we want to train with batches.
  * **Generalization** It seems to me your also suggesting the possibility to compute a unique covariance matrix from a set of training data. This approach would force us apply the same covariance matrix on every out of training data on which we want to apply our model, meaning that after applying that covariance matrix our data will probably not be truly covariance free. This is a problem common to standardization as well (the mean and variance computed on the training data always differ from the real mean and std of the population, but it's easy to compute mean and std of millions of training instances, so usually those estimation are quite robust).
  
  
  ---
  
  