<p>The assumption of normality is not a required assumption for OLS.  Further, the assumption is that the errors are normal, not the residuals.  Nonetheless, there can be a material concern when normality is violated because it depends upon why it was violated.  It only needs "fixed" if there is a reason that it was violated.</p>

<p>There are a few specific cases where it would be a problem and in those cases, you may need to switch to Bayesian methods, as mentioned above.  Let us imagine you were estimating the prices of automobiles.  There is no reason at all to believe the errors would be anything but normal.  On the other hand, if you were doing work on growth rates or financial assets then the errors couldn't possibly be normally distributed.</p>

<p>To back up a little bit, there are three primary classes of estimators, the Frequentist, the Likelihoodist and the Bayesian.  Frequentist and Likelihoodist methods are null hypothesis methods.  Bayesian methods are not.</p>

<p>You were most likely taught the Frequentist method.  It seeks the minimum variance unbiased estimator.  In the general case, the method of ordinary least squares is the minimum variance unbiased estimator.  Under assumptions violations, it may be inconsistent or inefficient.  In that case, robust methods are advised such as Theil's regression or quantile regression.</p>

<p>The main alternative in economics is Fisher's Likelihoodist method.  The assumption of normality is <em>required</em> to use ordinary least squares in regression with this school of thought.  You need to know the likelihood function to find the maximum likelihood function.  If it does not hold, then OLS is the <em>wrong</em> estimator.  If you wanted to use the method of maximum likelihood and normality did not hold, then you would need to determine the likelihood function and take the first and second derivative to determine which method was actually correct.  The Frequentist estimator is never a more accurate estimator than the MLE, but it can be worse.</p>

<p><strong>BEGINNING OF EDIT</strong></p>

<p>A good example of where ordinary least squares would not be the maximum likelihood formula would be in the logarithmic approximation of the time series $X_{t+1}=RX_t+\epsilon_{t+1},R&gt;1$.  In the log form as $x_{t+1}=rx_t+\varepsilon_{t+1}$ the likelihood function is $$\frac{1}{2\gamma}\text{sech}\left[\frac{\pi}{2}\left(\frac{x_{t+1}-\hat{\beta}x_t-\hat{\alpha}}{\gamma}\right)\right].$$</p>

<p>Ordinary least squares would be used because the MLE is not analytic and so Fisher's method isn't really viable.  Ordinary least squares is usable for any linear relationship where all the variables have a defined second moment.  An example where this would not be the case are returns on stocks which cannot have a variance, or a mean for that matter as they are the ratio of $$\frac{p_{t+1}q_{t+1}}{p_tq_t}-1,$$ where $p$ are prices and $q$ are shares. Assuming the firm did not go bankrupt or was merged out and ignoring liquidity, the likelihood function would be $$\left[\frac{\pi}{2}+\tan^{-1}\left(\frac{\mu}{\sigma}\right)\right]^{-1}\frac{\sigma}{\sigma^2+(r_t-\mu)^2}.$$</p>

<p>If you perform the integration to find the expectation you will find it does not exist.  This can be derived from the fact that equity securities are sold in a double auction and that there are many potential buyers and sellers.  You can also verify this against the disaggregated trade data in CRSP.  If you control for liquidity, this will be your likelihood function.</p>

<p><strong>END OF EDIT</strong></p>

<p>The primary circumstance one would use either method is when there is no actual prior information about the true values for the coefficients in regression.  The methods minimize the maximum loss that could happen by using the estimators in the real world.  To use an analogy, using either of these two methods would minimize the risk of burning a house completely down to the ground from having a group of teenagers come over for a party.  Bayesian methods, conversely, minimize the average damage expected to the house from inviting a bunch of teenagers over for a party.  The Bayesian method doesn't minimize the specific risk of total loss but does minimize the risk of loss from all sources.</p>

<p>Bayesian methods will require you to look at the prior research on the topic and capture this prior knowledge in a probability distribution usually called "the prior."  You would use both the information in your data and the knowledge in the literature to estimate the location of the parameters.  This tends to make Bayesian methods significantly more accurate than non-Bayesian methods, simply because there is more information included in the solution.  It also makes them difficult to use.  You should get supervision at first or do a lot of reading.</p>

<p>Bayesian methods operate in the parameter space and not the sample space as Frequentist and Likelihoodist methods do.  Data is not random, parameters are.  There is no null hypothesis.  You cannot falsify a null because there isn't one.  There can be infinitely many hypotheses and the set of hypotheses must be mutually exclusive and exhaustive of all cases.</p>

<p>Because you would be operating in the parameter space many problems that exist in non-Bayesian methods are not problems in the parameter space.  To give an example, autocorrelation is a serious issue for non-Bayesian methods and not an issue in Bayesian methods.  It doesn't make the estimate inefficient and it doesn't interfere with hypothesis testing.  Of course, as you have seen there are a different set of problems in Bayesian methods such as constructing the prior.</p>

<p>An important place where Bayesian methods shine is in prediction.  Frequentist predictions depend upon two things.  First, a pivotal value has to exist.  Second, the maximum likelihood estimator needs to be representative of the true value.  If both are true, then the Frequentist prediction can be as good as the Bayesian prediction, but if not will be far from reality.  The Bayesian estimator does not require either of these.  The Bayesian estimate is the distribution $$\Pr(\tilde{x}|\mathbf{X}).$$  </p>

<p>Notice that there are no parameter estimates in there or parameters.  It is not $\Pr(\tilde{x}|\beta;\mathbf{X})$.</p>

<p>If you have never used a Bayesian method before, begin with William Bolstad's <em>Introduction to Bayesian Statistics.</em>  You will need to know calculus through integration.  This is because the estimate of the population mean, which is $\bar{x}$ in Frequentist methods is not the estimator of the population mean in Bayesian methods.  For example, the estimate of the mean of an exponentially distributed variable is: $$\Pr(\lambda|x_1\dots{x}_n)=\frac{\prod_{i=1}^n{\lambda{e}^{-\lambda{x_i}}}\Pr(\lambda)}{\int_0^\infty\prod_{i=1}^n{{\lambda{e}^{-\lambda{x_i}}}\Pr(\lambda)}{\mathrm{d}\lambda}},\forall\lambda\in\mathbb{R}^+$$.</p>

<p>The $\Pr(\lambda),\forall\lambda\in\mathbb{R}^+$ is determined by your research about prior information on the location of $\lambda$.  As you can tell, this is not plug-n-play like you get for ordinary least squares.</p>

<p>If I were you, until my skills in other methods developed, I would use quantile regression.  It is plug-n-play.</p>

<p>If you want to provide a little more information about your problem, it might also be possible to tell you if the absence of normality is an issue.</p>

CC BY-SA 3.0
2018-04-07T16:25:32.770Z
You write "The assumption of normality is required to use ordinary least squares in regression with this school of thought." That's simply wrong.  Least squares is by construction a distribution-free estimation method, and it can be implemented on its own,irrespective of likelihood considerations.
Alecos Papadopoulos
CC BY-SA 3.0
2018-04-08T19:55:54.777Z
*"There is no null hypothesis. You cannot falsify a null because there isn't one."* I've said the same myself before, and then people have pointed out to me cases of people doing hypothesis testing with Bayes. I'm not sure that it should be a thing, but it turns out it is a thing.
410 gone
CC BY-SA 3.0
2018-04-08T22:24:34.070Z
@EnergyNumbers  Bayesian methods have hypotheses, but no null hypothesis.  No hypothesis is special.  Instead, you assign positive probabilities that a specific hypothesis is true.  For example, you could have three hypotheses such as $\mu<5,5\le\mu<10,\mu\ge{10}$.  Each would have a probability of being true assigned to it, given the data.
Dave Harris
CC BY-SA 3.0
2018-04-08T23:02:35.287Z
@DaveHarris I know. I agree, And yet people have pointed me to people doing just that. Yes, I completely agree with you that it's utterly missing the point of Bayes. But it happens.
410 gone
CC BY-SA 3.0
2018-04-07T03:05:43.977Z
2018-04-08T18:16:18.143Z
2018-04-08T18:16:18.143Z
Dave Harris
<p>Use Bayesian regression. It's much more flexible on its assumptions, and much more powerful in its outputs. You'll find that you have to be much more explicit about your assumptions, and you'll have to quantify them.</p>

<p>It's much harder work, and all the work you do, will be valuable.</p>

<p>Normal regression makes it very easy for anyone to do really bad science.</p>

<p>Bayesian regression makes it harder to do anything, but makes it much easier to do good science.</p>

CC BY-SA 3.0
2018-04-06T22:38:51.943Z
Could you perhaps also recommend some books on Bayesian regressions?
Giskard
CC BY-SA 3.0
2018-04-08T20:42:01.087Z
I am pretty sure that it's also easy do bad science with Bayesian regression, especially because it's much harder work.
emeryville
CC BY-SA 3.0
2018-04-08T23:06:45.843Z
@emeryville I agree that it's quite easy to arrive at invalid answers with Bayesian regression, but I'd still argue that it's still slightly better science, because at least the analyst has been more explicit about (some of) their assumptions. It's by no means a panacea, and it is certainly possible to abuse it, but the very nature of demanding the analyst be quantitatively explicit about assumptions, I'd argue that it's easier to do good science.
410 gone
CC BY-SA 3.0
2018-04-09T05:36:56.507Z
Right. Could you perhaps also recommend some (introductory) books on Bayesian regressions?
emeryville
CC BY-SA 3.0
2018-04-09T10:08:18.593Z
What exactly are you saying about normal (= frequentist?) regression?  That it's often or easily misused, leading to bad science?  Or that even if properly used, there's something inherent about it that leads to bad science?
Adam Bailey
CC BY-SA 3.0
2018-04-09T16:13:32.423Z
@AdamBailey both of those.
410 gone
CC BY-SA 3.0
2018-04-06T22:29:46.730Z
2018-04-06T22:29:46.730Z
410 gone
<p>One of the key assumption of the OLS is that the normality of the residual.</p>

<p>If this assumption is violated, how can we fix it generally? </p>

<p>Especially, I want to make predictions</p>

<econometrics>
Normality of residual is violated, How to fix it properly?
Tom
# Normality of residual is violated, How to fix it properly?

(Asked by: username_0 on 2018-04-06)

One of the key assumption of the OLS is that the normality of the residual.

If this assumption is violated, how can we fix it generally?

Especially, I want to make predictions

**Tags:** econometrics


---

## 2 answers

### Answer by username_3 on 2018-04-06. Score: 2 | Accepted answer

Use Bayesian regression. It's much more flexible on its assumptions, and much more powerful in its outputs. You'll find that you have to be much more explicit about your assumptions, and you'll have to quantify them.

It's much harder work, and all the work you do, will be valuable.

Normal regression makes it very easy for anyone to do really bad science.

Bayesian regression makes it harder to do anything, but makes it much easier to do good science.

> **Comments:**
> **username_4**: Could you perhaps also recommend some books on Bayesian regressions?  
> **username_5**: I am pretty sure that it's also easy do bad science with Bayesian regression, especially because it's much harder work.  
> **username_3**: @username_5 I agree that it's quite easy to arrive at invalid answers with Bayesian regression, but I'd still argue that it's still slightly better science, because at least the analyst has been more explicit about (some of) their assumptions. It's by no means a panacea, and it is certainly possible to abuse it, but the very nature of demanding the analyst be quantitatively explicit about assumptions, I'd argue that it's easier to do good science.  
> **username_5**: Right. Could you perhaps also recommend some (introductory) books on Bayesian regressions?  
> **username_6**: What exactly are you saying about normal (= frequentist?) regression?  That it's often or easily misused, leading to bad science?  Or that even if properly used, there's something inherent about it that leads to bad science?  
> **username_3**: @username_6 both of those.  

---

### Answer by username_1 on 2018-04-07. Score: 1

The assumption of normality is not a required assumption for OLS. Further, the assumption is that the errors are normal, not the residuals. Nonetheless, there can be a material concern when normality is violated because it depends upon why it was violated. It only needs "fixed" if there is a reason that it was violated.

There are a few specific cases where it would be a problem and in those cases, you may need to switch to Bayesian methods, as mentioned above. Let us imagine you were estimating the prices of automobiles. There is no reason at all to believe the errors would be anything but normal. On the other hand, if you were doing work on growth rates or financial assets then the errors couldn't possibly be normally distributed.

To back up a little bit, there are three primary classes of estimators, the Frequentist, the Likelihoodist and the Bayesian. Frequentist and Likelihoodist methods are null hypothesis methods. Bayesian methods are not.

You were most likely taught the Frequentist method. It seeks the minimum variance unbiased estimator. In the general case, the method of ordinary least squares is the minimum variance unbiased estimator. Under assumptions violations, it may be inconsistent or inefficient. In that case, robust methods are advised such as Theil's regression or quantile regression.

The main alternative in economics is Fisher's Likelihoodist method. The assumption of normality is *required* to use ordinary least squares in regression with this school of thought. You need to know the likelihood function to find the maximum likelihood function. If it does not hold, then OLS is the *wrong* estimator. If you wanted to use the method of maximum likelihood and normality did not hold, then you would need to determine the likelihood function and take the first and second derivative to determine which method was actually correct. The Frequentist estimator is never a more accurate estimator than the MLE, but it can be worse.

**BEGINNING OF EDIT**

A good example of where ordinary least squares would not be the maximum likelihood formula would be in the logarithmic approximation of the time series $X\_{t+1}=RX\_t+\epsilon\_{t+1},R>1$. In the log form as $x\_{t+1}=rx\_t+\varepsilon\_{t+1}$ the likelihood function is $$\frac{1}{2\gamma}\text{sech}\left[\frac{\pi}{2}\left(\frac{x\_{t+1}-\hat{\beta}x\_t-\hat{\alpha}}{\gamma}\right)\right].$$

Ordinary least squares would be used because the MLE is not analytic and so Fisher's method isn't really viable. Ordinary least squares is usable for any linear relationship where all the variables have a defined second moment. An example where this would not be the case are returns on stocks which cannot have a variance, or a mean for that matter as they are the ratio of $$\frac{p\_{t+1}q\_{t+1}}{p\_tq\_t}-1,$$ where $p$ are prices and $q$ are shares. Assuming the firm did not go bankrupt or was merged out and ignoring liquidity, the likelihood function would be $$\left[\frac{\pi}{2}+\tan^{-1}\left(\frac{\mu}{\sigma}\right)\right]^{-1}\frac{\sigma}{\sigma^2+(r\_t-\mu)^2}.$$

If you perform the integration to find the expectation you will find it does not exist. This can be derived from the fact that equity securities are sold in a double auction and that there are many potential buyers and sellers. You can also verify this against the disaggregated trade data in CRSP. If you control for liquidity, this will be your likelihood function.

**END OF EDIT**

The primary circumstance one would use either method is when there is no actual prior information about the true values for the coefficients in regression. The methods minimize the maximum loss that could happen by using the estimators in the real world. To use an analogy, using either of these two methods would minimize the risk of burning a house completely down to the ground from having a group of teenagers come over for a party. Bayesian methods, conversely, minimize the average damage expected to the house from inviting a bunch of teenagers over for a party. The Bayesian method doesn't minimize the specific risk of total loss but does minimize the risk of loss from all sources.

Bayesian methods will require you to look at the prior research on the topic and capture this prior knowledge in a probability distribution usually called "the prior." You would use both the information in your data and the knowledge in the literature to estimate the location of the parameters. This tends to make Bayesian methods significantly more accurate than non-Bayesian methods, simply because there is more information included in the solution. It also makes them difficult to use. You should get supervision at first or do a lot of reading.

Bayesian methods operate in the parameter space and not the sample space as Frequentist and Likelihoodist methods do. Data is not random, parameters are. There is no null hypothesis. You cannot falsify a null because there isn't one. There can be infinitely many hypotheses and the set of hypotheses must be mutually exclusive and exhaustive of all cases.

Because you would be operating in the parameter space many problems that exist in non-Bayesian methods are not problems in the parameter space. To give an example, autocorrelation is a serious issue for non-Bayesian methods and not an issue in Bayesian methods. It doesn't make the estimate inefficient and it doesn't interfere with hypothesis testing. Of course, as you have seen there are a different set of problems in Bayesian methods such as constructing the prior.

An important place where Bayesian methods shine is in prediction. Frequentist predictions depend upon two things. First, a pivotal value has to exist. Second, the maximum likelihood estimator needs to be representative of the true value. If both are true, then the Frequentist prediction can be as good as the Bayesian prediction, but if not will be far from reality. The Bayesian estimator does not require either of these. The Bayesian estimate is the distribution $$\Pr(\tilde{x}|\mathbf{X}).$$

Notice that there are no parameter estimates in there or parameters. It is not $\Pr(\tilde{x}|\beta;\mathbf{X})$.

If you have never used a Bayesian method before, begin with William Bolstad's *Introduction to Bayesian Statistics.* You will need to know calculus through integration. This is because the estimate of the population mean, which is $\bar{x}$ in Frequentist methods is not the estimator of the population mean in Bayesian methods. For example, the estimate of the mean of an exponentially distributed variable is: $$\Pr(\lambda|x\_1\dots{x}\_n)=\frac{\prod\_{i=1}^n{\lambda{e}^{-\lambda{x\_i}}}\Pr(\lambda)}{\int\_0^\infty\prod\_{i=1}^n{{\lambda{e}^{-\lambda{x\_i}}}\Pr(\lambda)}{\mathrm{d}\lambda}},\forall\lambda\in\mathbb{R}^+$$.

The $\Pr(\lambda),\forall\lambda\in\mathbb{R}^+$ is determined by your research about prior information on the location of $\lambda$. As you can tell, this is not plug-n-play like you get for ordinary least squares.

If I were you, until my skills in other methods developed, I would use quantile regression. It is plug-n-play.

If you want to provide a little more information about your problem, it might also be possible to tell you if the absence of normality is an issue.

> **Comments:**
> **username_2**: You write "The assumption of normality is required to use ordinary least squares in regression with this school of thought." That's simply wrong.  Least squares is by construction a distribution-free estimation method, and it can be implemented on its own,irrespective of likelihood considerations.  
> **username_3**: *"There is no null hypothesis. You cannot falsify a null because there isn't one."* I've said the same myself before, and then people have pointed out to me cases of people doing hypothesis testing with Bayes. I'm not sure that it should be a thing, but it turns out it is a thing.  
> **username_1**: @EnergyNumbers  Bayesian methods have hypotheses, but no null hypothesis.  No hypothesis is special.  Instead, you assign positive probabilities that a specific hypothesis is true.  For example, you could have three hypotheses such as $\mu<5,5\le\mu<10,\mu\ge{10}$.  Each would have a probability of being true assigned to it, given the data.  
> **username_3**: @username_1 I know. I agree, And yet people have pointed me to people doing just that. Yes, I completely agree with you that it's utterly missing the point of Bayes. But it happens.  

---

