Answers.Body: <p><strong>Clarifications</strong></p>
  
  <p>Before directly answering the question, there are a few things to note.</p>
  
  <ul>
  <li>The word range is actually applicable to a dependent variable.  On the input side, it is the domain that needs to be normalized.</li>
  <li>In the second method, since the minimum vector may not be the origin, the zeroth order term should be included in the normalization.  The minimum must be subtracted from the maximum to determine the domain and must also be subtracted as an offset to the independent, pre-normalized values.</li>
  <li>The standard deviation is always less than half the difference between the max and min, so there is a difference in proportion between the two methods.</li>
  </ul>
  
  <p>The crux of the question centers around the statistics of normalization and its potential impact on these methods.</p>
  
  <ul>
  <li>Multivariate linear regression (in the question introduction)</li>
  <li>Gradient descent (selected as a tag)</li>
  </ul>
  
  <p>Although these two are not technically mutually exclusive, practically they are if by linear is meant conforming to <span class="math-container">$\vec{y} = a \vec{x} + \vec{b}$</span>, where <span class="math-container">$a$</span> is a matrix dimensioned appropriately for <span class="math-container">$\vec{x}$</span> and <span class="math-container">$\vec{y}$</span>.  For that first degree polynomial cases, there is a closed form (formula) to acquire <span class="math-container">$(\vec{a}, \vec{b}, c)$</span> from a set of pairs <span class="math-container">$(\vec{x}, \vec{y})$</span>, where <span class="math-container">$c$</span> is the correlation coefficient.  It would be grossly wasteful to use iterative convergence methods, such as gradient descent, to calculate a mean squares regression result for a first degree polynomial model of the relation between <span class="math-container">$\vec{x}$</span> and <span class="math-container">$\vec{y}$</span>.</p>
  
  <p>If there is no closed form for the model or no known model at all, and the complexity of the function to be approximated is tenable in relation to the convergence methods available, the data available for training, and the computing resources and time that can be delegated to the task, then gradient descent is a method through which a least squares convergence on minimal loss may be possible.</p>
  
  <p>The significance of artificial networks is that, through the interconnection of a sequence of layers, each with an array of like activation functions, a network can approximate a useful function the mathematical model of which is not explicitly known.  Of the statistical functional approximation problems where no closed form is known and for which convergence methods are possible, artificial network training has become the appropriate device for a growing set of cases.</p>
  
  <p><strong>Normalization</strong></p>
  
  <p>The metrics of mean and standard deviation are designed specifically for data that can be approximated by a Gaussian distribution.  The term normalization originated from the belief that the Gaussian distribution was normal, however Gauss did not hold that belief.  He presented it as a distribution resulting from the limit of an accumulation of random perturbations as the mean of the square of the perturbation size decreases to zero as the length of the series of perturbations approaches infinity.</p>
  
  <p>Because of this, if the loss (or error) function is a sum of the squares of difference between the current output values of an artificial network and the labels providing the accurate and correct values, a near Gaussian distribution of input is likely to produce a faster and complete convergence.  In the multivariate case, there is an equivalent of the Gaussian distribution.</p>
  
  <p>In the real world, data distribution very close to Gaussian is not the norm.  The distribution of land surface temperatures at a specific latitude and longitude on earth for a particular month reflect two maxima in the temperature distribution, one for nighttime and one for daytime.  The model of the relationship between these two maxima and the time of year, in chaos theory, are called attractors.  Points seem attracted to those norms.</p>
  
  <p>The distribution of signal amplitudes in common circuits and natural sounds in the air are not Gaussian distributed either, but the <span class="math-container">$\log(P)$</span> of the power of common signals and sounds are close to a Gaussian distribution, which is partly why decibels are a common measure of signal strength and loudness.  The other part is that the logarithm in calculating decibels causes them to indicate relative geometric difference (ratio) in power rather than a less representative arithmatic difference.</p>
  
  <p>Normalization is often oversimplified to mean ensuring values fall within a specific domain convenient to an algorithm's numerical representation or around a mean with a given standard deviation to feed a least squares strategy for statistical analysis or correlation.  However, data is not often symmetric and nicely distributed.</p>
  
  <p>In the case of land surface temperatures the nighttime distribution has a lower standard deviation than the daytime distribution, so the two peaks have different adjacent slopes in the distribution.  How that is reflected in normalization varies depending on goals and algorithms receiving the normalized data.  Normalization strategies for signal power and loudness would include running the data through a logarithm to reflect the near exponential Gaussian distribution of power values.</p>
  
  <p><strong>Outliers</strong></p>
  
  <p>Outliers are data points that fall outside an expected trend or relationship, either because they were incorrectly measured in one or more dimensions of measurement, the measurement was corrupted, they were strongly influenced by a phenomenon outside the intended range of study, or representative of a significant but previously undiscovered phenomenon.  This last cause is why it is not wise to dismiss and exclude all outliers from functional analysis.</p>
  
  <p>When outliers are valid data points that inconveniently lie more than a few standard deviations from mean, normalizing from zero to one causes a very narrow distribution peak in one place within that input domain.  If the outlier is of the kind that should not be dismissed and legitimately representative of the phenomena under study or being controlled, then some normalization scheme other than simply a first degree polynomial translation of the data is indicated.</p>
  
  <p><strong>Housing Predictor</strong></p>
  
  <p>In the case of square footage of living space, the data is not symmetric.  An increase of 10 square meters for a large house will not produce the same price differential as 10 square meter increase for a small one.  A properly configured neural network may learn to perform something similar to a logarithm in the first few layers during training, or the engineer can explicitly perform a logarithm to un-skew the data and present it in a form with closer to a Gaussian distribution to the network for training and then also later for actual use.</p>
  
  <p>At the exit, prior to presenting the network output to the loss function, applying an exponential function may also produce faster and more reliable convergence, since a monetary unit has different values to different buyers.  A person buying a large house is less apt to be put off by cost associated with a fourth bedroom or more area than a person buying a small house.</p>
  
  <p><strong>Summary</strong></p>
  
  <p>The comparative advantages of different normalization methods are dependent upon a number of factors, including but not limited to these.</p>
  
  <ul>
  <li>The resolution of the data types used to represent forward and backward signal propagation in the network, which affects the conditions of signal saturation</li>
  <li>Skew of data set used for training</li>
  <li>The number of standard deviations to the min and max in the data set used for training</li>
  <li>Presence of multiple attractors in the distribution, which can occur in any of the features (dimensions of input) or combinations of them</li>
  </ul>
  
  <p>Method one is best for a scenario that involves a Gaussian distribution and may be best for the housing prediction.  Method two might be best if the distribution is relatively flat almost all the way to the extremes.</p>
  
  <p>Running each method and comparing them may reveal advantages for one or the other that is not predicted in this answer and further knowledge can be gained about why by performing further experiments.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-26T08:35:33.180Z
Answers.LastActivityDate: 2018-11-26T08:35:33.180Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>In the exercise <a href="http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&amp;doc=exercises/ex3/ex3.html" rel="nofollow noreferrer">Exercise 3: Multivariate Linear Regression</a>, by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero".</p>
  
  <p><span class="math-container">$$x_{n e w}=\frac{x-\mu}{\sigma}$$</span></p>
  
  <h3>Method 1</h3>
  
  <p>The author provides the following Matlab (and Octave) code to scale the inputs.</p>
  
  <pre><code>x = [ones(m, 1), x];
  sigma = std(x);
  mu = mean(x);
  x(:,2) = (x(:,2) - mu(2))./ sigma(2);
  x(:,3) = (x(:,3) - mu(3))./ sigma(3);
  </code></pre>
  
  <h3>Method 2</h3>
  
  <p>But why not simply scale the inputs between zero and one, or, divide by the maximum?</p>
  
  <pre><code>x_range=max(x)
  x(:,2) = (x(:,2)/x_range(2));
  x(:,3) = (x(:,3)/x_range(3));
  </code></pre>
  
  <p>I have done the exercise with method 2 and these are the results.</p>
  
  <p><a href="https://i.sstatic.net/fdhSG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fdhSG.png" alt="enter image description here"></a></p>
  
  <h3>Question</h3>
  
  <p>Is there a computational advantage with the first method over the second method?</p>
  
Tags: <implementation><data-preprocessing>
Title: Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?
UserDisplayName: gatorback
ThreadText: # Which of these two numerical methods for z-score normalisation is preferable, in multivariate linear regression?
  
  (Asked by: username_0 on 2018-11-21)
  
  In the exercise [Exercise 3: Multivariate Linear Regression](http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&doc=exercises/ex3/ex3.html), by Andrew Ng, the author suggests to "scale both types of inputs by their standard deviations and set their means to zero".
  
  $$x\_{n e w}=\frac{x-\mu}{\sigma}$$
  
  ### Method 1
  
  The author provides the following Matlab (and Octave) code to scale the inputs.
  
  ```
  x = [ones(m, 1), x];
  sigma = std(x);
  mu = mean(x);
  x(:,2) = (x(:,2) - mu(2))./ sigma(2);
  x(:,3) = (x(:,3) - mu(3))./ sigma(3);
  
  ```
  
  ### Method 2
  
  But why not simply scale the inputs between zero and one, or, divide by the maximum?
  
  ```
  x_range=max(x)
  x(:,2) = (x(:,2)/x_range(2));
  x(:,3) = (x(:,3)/x_range(3));
  
  ```
  
  I have done the exercise with method 2 and these are the results.
  
  [![enter image description here](https://i.sstatic.net/fdhSG.png)](https://i.sstatic.net/fdhSG.png)
  
  ### Question
  
  Is there a computational advantage with the first method over the second method?
  
  **Tags:** implementation, data-preprocessing
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-26. Score: 0
  
  **Clarifications**
  
  Before directly answering the question, there are a few things to note.
  
  * The word range is actually applicable to a dependent variable. On the input side, it is the domain that needs to be normalized.
  * In the second method, since the minimum vector may not be the origin, the zeroth order term should be included in the normalization. The minimum must be subtracted from the maximum to determine the domain and must also be subtracted as an offset to the independent, pre-normalized values.
  * The standard deviation is always less than half the difference between the max and min, so there is a difference in proportion between the two methods.
  
  The crux of the question centers around the statistics of normalization and its potential impact on these methods.
  
  * Multivariate linear regression (in the question introduction)
  * Gradient descent (selected as a tag)
  
  Although these two are not technically mutually exclusive, practically they are if by linear is meant conforming to $\vec{y} = a \vec{x} + \vec{b}$, where $a$ is a matrix dimensioned appropriately for $\vec{x}$ and $\vec{y}$. For that first degree polynomial cases, there is a closed form (formula) to acquire $(\vec{a}, \vec{b}, c)$ from a set of pairs $(\vec{x}, \vec{y})$, where $c$ is the correlation coefficient. It would be grossly wasteful to use iterative convergence methods, such as gradient descent, to calculate a mean squares regression result for a first degree polynomial model of the relation between $\vec{x}$ and $\vec{y}$.
  
  If there is no closed form for the model or no known model at all, and the complexity of the function to be approximated is tenable in relation to the convergence methods available, the data available for training, and the computing resources and time that can be delegated to the task, then gradient descent is a method through which a least squares convergence on minimal loss may be possible.
  
  The significance of artificial networks is that, through the interconnection of a sequence of layers, each with an array of like activation functions, a network can approximate a useful function the mathematical model of which is not explicitly known. Of the statistical functional approximation problems where no closed form is known and for which convergence methods are possible, artificial network training has become the appropriate device for a growing set of cases.
  
  **Normalization**
  
  The metrics of mean and standard deviation are designed specifically for data that can be approximated by a Gaussian distribution. The term normalization originated from the belief that the Gaussian distribution was normal, however Gauss did not hold that belief. He presented it as a distribution resulting from the limit of an accumulation of random perturbations as the mean of the square of the perturbation size decreases to zero as the length of the series of perturbations approaches infinity.
  
  Because of this, if the loss (or error) function is a sum of the squares of difference between the current output values of an artificial network and the labels providing the accurate and correct values, a near Gaussian distribution of input is likely to produce a faster and complete convergence. In the multivariate case, there is an equivalent of the Gaussian distribution.
  
  In the real world, data distribution very close to Gaussian is not the norm. The distribution of land surface temperatures at a specific latitude and longitude on earth for a particular month reflect two maxima in the temperature distribution, one for nighttime and one for daytime. The model of the relationship between these two maxima and the time of year, in chaos theory, are called attractors. Points seem attracted to those norms.
  
  The distribution of signal amplitudes in common circuits and natural sounds in the air are not Gaussian distributed either, but the $\log(P)$ of the power of common signals and sounds are close to a Gaussian distribution, which is partly why decibels are a common measure of signal strength and loudness. The other part is that the logarithm in calculating decibels causes them to indicate relative geometric difference (ratio) in power rather than a less representative arithmatic difference.
  
  Normalization is often oversimplified to mean ensuring values fall within a specific domain convenient to an algorithm's numerical representation or around a mean with a given standard deviation to feed a least squares strategy for statistical analysis or correlation. However, data is not often symmetric and nicely distributed.
  
  In the case of land surface temperatures the nighttime distribution has a lower standard deviation than the daytime distribution, so the two peaks have different adjacent slopes in the distribution. How that is reflected in normalization varies depending on goals and algorithms receiving the normalized data. Normalization strategies for signal power and loudness would include running the data through a logarithm to reflect the near exponential Gaussian distribution of power values.
  
  **Outliers**
  
  Outliers are data points that fall outside an expected trend or relationship, either because they were incorrectly measured in one or more dimensions of measurement, the measurement was corrupted, they were strongly influenced by a phenomenon outside the intended range of study, or representative of a significant but previously undiscovered phenomenon. This last cause is why it is not wise to dismiss and exclude all outliers from functional analysis.
  
  When outliers are valid data points that inconveniently lie more than a few standard deviations from mean, normalizing from zero to one causes a very narrow distribution peak in one place within that input domain. If the outlier is of the kind that should not be dismissed and legitimately representative of the phenomena under study or being controlled, then some normalization scheme other than simply a first degree polynomial translation of the data is indicated.
  
  **Housing Predictor**
  
  In the case of square footage of living space, the data is not symmetric. An increase of 10 square meters for a large house will not produce the same price differential as 10 square meter increase for a small one. A properly configured neural network may learn to perform something similar to a logarithm in the first few layers during training, or the engineer can explicitly perform a logarithm to un-skew the data and present it in a form with closer to a Gaussian distribution to the network for training and then also later for actual use.
  
  At the exit, prior to presenting the network output to the loss function, applying an exponential function may also produce faster and more reliable convergence, since a monetary unit has different values to different buyers. A person buying a large house is less apt to be put off by cost associated with a fourth bedroom or more area than a person buying a small house.
  
  **Summary**
  
  The comparative advantages of different normalization methods are dependent upon a number of factors, including but not limited to these.
  
  * The resolution of the data types used to represent forward and backward signal propagation in the network, which affects the conditions of signal saturation
  * Skew of data set used for training
  * The number of standard deviations to the min and max in the data set used for training
  * Presence of multiple attractors in the distribution, which can occur in any of the features (dimensions of input) or combinations of them
  
  Method one is best for a scenario that involves a Gaussian distribution and may be best for the housing prediction. Method two might be best if the distribution is relatively flat almost all the way to the extremes.
  
  Running each method and comparing them may reveal advantages for one or the other that is not predicted in this answer and further knowledge can be gained about why by performing further experiments.
  
  
  ---
  
  