<blockquote>
<p>If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data?</p>
</blockquote>
<p>Aside from special cases no, OLS fits a line that is created based on minimization of sum of squared errors:</p>
<p><span class="math-container">$$\min \sum e_i^2 = \min \sum (y_i-\hat{y_i})=\min \sum (y_i - \mathbf{x}_i^{\prime}  \beta)^2 .$$</span></p>
<p>If you remove some observations then you will likely change the objective function above leading to some other line having the best fit through data (save for special cases like when all observations are on a single line).</p>
<blockquote>
<p>If not, wouldn't that show that the data doesn't have a linear relationship?</p>
</blockquote>
<p>Not necessarily, the linear relationship is assumed to exist between the variables in the 'background', but since we cannot directly observe the data generating process and there will always be some disturbance term even if we are sure there should be a linear relationship between the two variables the observations will not necessarily all fall on a single line.</p>
<p>Non-linearity cannot be detected just from the fact that line you fit changes when you remove some data (although that is an indicator of robustness of a result - robust results should not change too much when sample is expanded/cut slightly). However, you can detect non-linearity in other ways. For example, you can look for patterns in residual plot. If residual plot shows some curved pattern then that is an indicator of potential non-linearity. There are also rigorous tests for non-linearity (like Ramsey test etc) but to go over all of such ways would be beyond the scope of an SE answer - you can read more about them in econometrics textbooks such as Wooldridge Introductory Econometrics: A Modern Approach.</p>

CC BY-SA 4.0
2020-11-07T16:38:47.823Z
how to do the ramsey test of non-linearity
develarist
CC BY-SA 4.0
2020-11-07T16:40:52.413Z
@develarist that would require separate answer. If that is what you are interested in post it as a new question on the site. Or you can just google Ramsey test for non-linearity
1muflon1
CC BY-SA 4.0
2020-11-07T18:31:01.930Z
it says Ramsey RESET test is for testing model mis-specification
develarist
CC BY-SA 4.0
2020-11-07T18:33:11.827Z
@develarist right but non-linearity is a form of mis-specification. If model is supposed to be $y= a + b_1 x_1 + b_2 x_1^2$ (which is non-linear) but you estimate $y = a + b_1 x_1$ (which is linear) you are mis-specifying your model. Hence the RESET test can test for non-linearity if properly set up
1muflon1
CC BY-SA 4.0
2020-11-07T18:45:15.770Z
non-linearity in the variables or non-linearity in the coefficients?
develarist
CC BY-SA 4.0
2020-11-07T18:46:39.420Z
@develarist here I refer to the non-linearity of the relationship that is non-linearity in relationship between $y$ and $x$ of course both of the above models are still linear in parameters.
1muflon1
CC BY-SA 4.0
2020-11-07T22:31:26.083Z
Thank you for the answer @1muflon1 . About robustness - would you say that the OLS here is un-robust? I can see that in this particular dataset, the difference between the ~20% and ~7% predictive effects is not reflected in the standard error for $\beta_1$  which is only 0.0068 (0.68%). Am I still able to say then that "there is a 20% (0.68%) predictive effect" or would that be misleading and I should instead say "it's inconclusive" or "it depends on marital status"? Is it in general typical to check if control variables substantially affect predictive effect of treatment variables?
llllvvuu
CC BY-SA 4.0
2020-11-07T22:35:00.410Z
@LawrenceWu you are welcome, if you think this answered your Q consider accepting it. Regarding the robustness question, if beta across different subsamples changes quite a lot then it would not be robust but you would not gauge this from standard errors but from how beta changes in different subsamples. Here there are also not really any hard rules for what is considered robust, more like rules of thumb. For example, if in one sample reg show that 1y of educ increases income by 100e and second that by 101e that would be considered robust but if it would be 1000e it would not be robust anymore
1muflon1
CC BY-SA 4.0
2020-11-07T11:11:53.310Z
2020-11-07T11:17:32.163Z
2020-11-07T11:17:32.163Z
1muflon1
<p>There is a possible ambiguity in &quot;fit the data&quot;.  Suppose first that a sample set of data points lie <em>exactly</em> on a straight line, so that the line estimated by linear regression with linear functional form provides a <em>perfect</em> fit (all residuals are zero).  Suppose you take a sub-sample (of at least 2 non-coincident data points) and run the regression again.  That will yield precisely the same regression line.  This is true, moreover, whether the sub-sample is selected randomly from the original sample or whether it is selected so as deliberately to exclude data points associated with certain characteristics.</p>
<p>In an empirical study the above scenario is of course unlikely.  Typically, there is a degree of randomness in the distribution of data points. Therefore, even if a line estimated by linear regression with linear functional form provides a good fit, the fit will be less than perfect (there will be some non-zero residuals).  Because of the randomness, running the regression again with a sub-sample of the original sample probably will not yield precisely the same regression line.</p>
<p>If the original sample was large and a fairly large sub-sample is randomly chosen, then the likelihood is that the two regression lines will not differ very much. If however the sub-sample is deliberately chosen (eg as in your example to exclude married individuals) then it is possible that the two regression lines will differ considerably, because there may be on average a real difference between included and excluded individuals in the relationship between the regression variables.  However, even if the two regression lines differ considerably, this does not imply that the linear functional form of the original line is inappropriate.  It merely shows that the sample to which the original line was fitted includes sub-samples which differ in relevant ways.</p>

CC BY-SA 4.0
2020-11-07T22:46:12.673Z
Thanks for the answer - is there a point at which it would be more appropriate to present the OLSs over the subsamples rather than the OLS over the whole dataset (i.e. does the full-sample OLS relatively lose meaning at some point)?
llllvvuu
CC BY-SA 4.0
2020-11-08T13:21:15.417Z
@LawrenceWu  I'm not sure I understand exactly what you are asking in your comment, but the following may help.  Whenever you run a regression to estimate parameters, it is important to be clear as to what is the population of interest.  If you have data for a random sample from your population of interest, there is nothing to be gained by running a regression on a sub-sample rather than on the whole sample (it would just tend to increase the standard errors of the parameter estimates).  Running a regression on a sub-sample would be worthwhile in two circumstances: a) if you are interested ...
Adam Bailey
CC BY-SA 4.0
2020-11-08T13:24:42.533Z
... in estimating the parameters for a sub-population as well as for the whole population; b) if the population from which the original sample was drawn is not of interest to you (eg because you are using a dataset collected for another purpose), but a subset of that population is of interest to you.
Adam Bailey
CC BY-SA 4.0
2020-11-07T16:08:40.697Z
2020-11-07T17:45:12.107Z
2020-11-07T17:45:12.107Z
Adam Bailey
<p>The relationship may be different on a sub-sample if you purposefully exclude some demographics. The linear relationship between the education of males and wages and education of females and wages might not be the same. This is why wage equations control for the effect of gender. If you exclude gender the relationship might change even if the relationship is still linear for both males and females.</p>

CC BY-SA 4.0
2020-11-07T12:32:07.923Z
2020-11-07T12:32:07.923Z
csilvia
<p>If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data? If not, wouldn't that show that the data doesn't have a linear relationship?</p>
<p>Example: <a href="https://ocw.mit.edu/courses/economics/14-382-econometrics-spring-2017/lecture-notes/MIT14_382S17_lec1.pdf" rel="nofollow noreferrer">https://ocw.mit.edu/courses/economics/14-382-econometrics-spring-2017/lecture-notes/MIT14_382S17_lec1.pdf</a>. On page 10, in the gender wage gap example, it finds a predictive effect of ~20% in the full sample and ~7% in the never married sample. But then if dWage/dGender depends on Marital Status, how can the functional form be right?</p>

<econometrics><regression>
Why would regression coefficients be different on a subsample?
llllvvuu
# Why would regression coefficients be different on a subsample?

(Asked by: username_0 on 2020-11-07)

If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data? If not, wouldn't that show that the data doesn't have a linear relationship?

Example: <https://ocw.mit.edu/courses/economics/14-382-econometrics-spring-2017/lecture-notes/MIT14_382S17_lec1.pdf>. On page 10, in the gender wage gap example, it finds a predictive effect of ~20% in the full sample and ~7% in the never married sample. But then if dWage/dGender depends on Marital Status, how can the functional form be right?

**Tags:** econometrics, regression


---

## 3 answers

### Answer by username_1 on 2020-11-07. Score: 2 | Accepted answer

> If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data?

Aside from special cases no, OLS fits a line that is created based on minimization of sum of squared errors:

$$\min \sum e\_i^2 = \min \sum (y\_i-\hat{y\_i})=\min \sum (y\_i - \mathbf{x}\_i^{\prime} \beta)^2 .$$

If you remove some observations then you will likely change the objective function above leading to some other line having the best fit through data (save for special cases like when all observations are on a single line).

> If not, wouldn't that show that the data doesn't have a linear relationship?

Not necessarily, the linear relationship is assumed to exist between the variables in the 'background', but since we cannot directly observe the data generating process and there will always be some disturbance term even if we are sure there should be a linear relationship between the two variables the observations will not necessarily all fall on a single line.

Non-linearity cannot be detected just from the fact that line you fit changes when you remove some data (although that is an indicator of robustness of a result - robust results should not change too much when sample is expanded/cut slightly). However, you can detect non-linearity in other ways. For example, you can look for patterns in residual plot. If residual plot shows some curved pattern then that is an indicator of potential non-linearity. There are also rigorous tests for non-linearity (like Ramsey test etc) but to go over all of such ways would be beyond the scope of an SE answer - you can read more about them in econometrics textbooks such as Wooldridge Introductory Econometrics: A Modern Approach.

> **Comments:**
> **username_2**: how to do the ramsey test of non-linearity  
> **username_1**: @username_2 that would require separate answer. If that is what you are interested in post it as a new question on the site. Or you can just google Ramsey test for non-linearity  
> **username_2**: it says Ramsey RESET test is for testing model mis-specification  
> **username_1**: @username_2 right but non-linearity is a form of mis-specification. If model is supposed to be $y= a + b_1 x_1 + b_2 x_1^2$ (which is non-linear) but you estimate $y = a + b_1 x_1$ (which is linear) you are mis-specifying your model. Hence the RESET test can test for non-linearity if properly set up  
> **username_2**: non-linearity in the variables or non-linearity in the coefficients?  
> **username_1**: @username_2 here I refer to the non-linearity of the relationship that is non-linearity in relationship between $y$ and $x$ of course both of the above models are still linear in parameters.  
> **username_0**: Thank you for the answer @username_1 . About robustness - would you say that the OLS here is un-robust? I can see that in this particular dataset, the difference between the ~20% and ~7% predictive effects is not reflected in the standard error for $\beta_1$  which is only 0.0068 (0.68%). Am I still able to say then that "there is a 20% (0.68%) predictive effect" or would that be misleading and I should instead say "it's inconclusive" or "it depends on marital status"? Is it in general typical to check if control variables substantially affect predictive effect of treatment variables?  
> **username_1**: @LawrenceWu you are welcome, if you think this answered your Q consider accepting it. Regarding the robustness question, if beta across different subsamples changes quite a lot then it would not be robust but you would not gauge this from standard errors but from how beta changes in different subsamples. Here there are also not really any hard rules for what is considered robust, more like rules of thumb. For example, if in one sample reg show that 1y of educ increases income by 100e and second that by 101e that would be considered robust but if it would be 1000e it would not be robust anymore  

---

### Answer by username_3 on 2020-11-07. Score: 1

There is a possible ambiguity in "fit the data". Suppose first that a sample set of data points lie *exactly* on a straight line, so that the line estimated by linear regression with linear functional form provides a *perfect* fit (all residuals are zero). Suppose you take a sub-sample (of at least 2 non-coincident data points) and run the regression again. That will yield precisely the same regression line. This is true, moreover, whether the sub-sample is selected randomly from the original sample or whether it is selected so as deliberately to exclude data points associated with certain characteristics.

In an empirical study the above scenario is of course unlikely. Typically, there is a degree of randomness in the distribution of data points. Therefore, even if a line estimated by linear regression with linear functional form provides a good fit, the fit will be less than perfect (there will be some non-zero residuals). Because of the randomness, running the regression again with a sub-sample of the original sample probably will not yield precisely the same regression line.

If the original sample was large and a fairly large sub-sample is randomly chosen, then the likelihood is that the two regression lines will not differ very much. If however the sub-sample is deliberately chosen (eg as in your example to exclude married individuals) then it is possible that the two regression lines will differ considerably, because there may be on average a real difference between included and excluded individuals in the relationship between the regression variables. However, even if the two regression lines differ considerably, this does not imply that the linear functional form of the original line is inappropriate. It merely shows that the sample to which the original line was fitted includes sub-samples which differ in relevant ways.

> **Comments:**
> **username_0**: Thanks for the answer - is there a point at which it would be more appropriate to present the OLSs over the subsamples rather than the OLS over the whole dataset (i.e. does the full-sample OLS relatively lose meaning at some point)?  
> **username_3**: @LawrenceWu  I'm not sure I understand exactly what you are asking in your comment, but the following may help.  Whenever you run a regression to estimate parameters, it is important to be clear as to what is the population of interest.  If you have data for a random sample from your population of interest, there is nothing to be gained by running a regression on a sub-sample rather than on the whole sample (it would just tend to increase the standard errors of the parameter estimates).  Running a regression on a sub-sample would be worthwhile in two circumstances: a) if you are interested ...  
> **username_3**: ... in estimating the parameters for a sub-population as well as for the whole population; b) if the population from which the original sample was drawn is not of interest to you (eg because you are using a dataset collected for another purpose), but a subset of that population is of interest to you.  

---

### Answer by username_4 on 2020-11-07. Score: 0

The relationship may be different on a sub-sample if you purposefully exclude some demographics. The linear relationship between the education of males and wages and education of females and wages might not be the same. This is why wage equations control for the effect of gender. If you exclude gender the relationship might change even if the relationship is still linear for both males and females.


---

