Standardization and test prediction in linear regression I am considering a linear regression model to predict a target $y\in\mathbb{R}^n$ from a data matrix $X\in\mathbb{R}^{n\times d}$. Let $X_1,\ldots,X_d\in\mathbb{R}^n$ be the columns of $X$. The prediction of $y$ is $\hat{\alpha}1_n+X\hat{\beta}$, where $$\hat{\alpha},\hat{\beta}\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y-\alpha 1_n-X\beta\|_2^2.$$

*

*It is common practice to standardize the columns of $X$ to obtain a new matrix $X'$, for, e.g., interpretability: the columns of $X'$ are, for all $j\in\{1,\ldots,d\}$, $$X_j'=\frac{1}{\sigma_j}(X_j-\mu_j1_n),\quad\text{where}\quad\mu_j=\frac{1}{n}\sum_{i=1}^nX_{i,j}\quad\text{and}\quad\sigma_j=\frac{1}{n}\sum_{i=1}^nX_{i,j}^2-\mu_j^2.$$ Then the prediction of $y$ is $\hat{\alpha}'1_n+X'\hat{\beta}'$, where $$\hat{\alpha}',\hat{\beta}'\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y-\alpha 1_n-X'\beta\|_2^2.$$ While I understand that this does not affect the prediction accuracy of $y$, what about the prediction of a test response? Given a test data matrix $X_\text{test}\in\mathbb{R}^{m\times d}$, the predicted response is $\hat{\alpha}'1_m+X_\text{test}'\hat{\beta}'$, where $X_{\text{test},j}'=(1/\sigma_j)(X_{\text{test},j}-\mu_j1_m)$. Without standardization, the predicted response is $\hat{\alpha}1_m+X_\text{test}\hat{\beta}$, which seems more direct and probably more accurate. Are there situations where we are still better off using standardization for test prediction?


*Suppose I also standardize $y$ to obtain a new target $y'$, which is not necessarily useful (or is it?) but not incorrect either. Then the prediction of $y'$ is $\hat{\alpha}''1_n+X'\hat{\beta}''$, where $$\hat{\alpha}'',\hat{\beta}''\in\arg\min_{\alpha\in\mathbb{R}\\\beta\in\mathbb{R}^d}\|y'-\alpha 1_n-X'\beta\|_2^2,$$ and the prediction of $y$ is $\sigma_y(\hat{\alpha}''1_n+X'\hat{\beta}'')+\mu_y1_n$. By the way, we know that $\hat{\alpha}''=0$ (Why does the y-intercept of a linear model disappear when I standardize variables?). Is the test prediction $\sigma_y(\hat{\alpha}''1_m+X_\text{test}'\hat{\beta}'')+\mu_y1_m$? How does its accuracy compare to that in 1.?
Thanks!

Without loss of generality, a simple linear regression is assumed. Write the sample mean of the regressor as $\overline{x} = \frac{1}{n}X^\top 1_n$, and its standard deviation as $s$. Then, the centered and scaled regressor is $X^* = (X - \overline{x})/s$. Now, observe the relationship between the regression models with nonstandardized and standardized regressors. The nonstandardized one is
$$
Y = \alpha + \beta X + \epsilon,
$$
whereas with the standardized covariate vector, it becomes
$$
Y = \alpha^* + \beta^* X^* + \epsilon = \alpha^* + \beta^* \frac{X - \overline{x}}{s} + \epsilon = \left(\alpha^* - \frac{\beta^* \overline{x}}{s} \right) + \frac{\beta^*}{s}X + \epsilon.
$$
We see that $\alpha = \alpha^* - \frac{\beta^*}{s} \overline{x}$ and $\beta = \beta^*/s$. Since $\widehat{\alpha} = \overline{y}-\widehat{\beta}\overline{x}$, $\widehat{\alpha}^* = \overline{y}$ and $\widehat{\beta}^* = s\widehat{\beta}$. This shows you that standardizing the regressor doesn't really change the problem much.
But if you look at the covariance between $\widehat{\alpha}$ and $\widehat{\beta}$, this has a greater implication than just interpretability: $\mathrm{Cov}(\widehat{\alpha},\widehat{\beta}) = -\frac{\overline{x}}{s}$. If the regressor is standardized, $\overline{x}^* = 0$, and thus, $\mathrm{Cov}(\widehat{\alpha}^*,\widehat{\beta}^*) = 0$. This is often very important for accelerating estimation, but I digress. Just note that this is conducive to inference, rather than prediction.
Now, let's say you have a new regressor value $x_\text{new}$. It should be standardized using the same $\overline{x}$ and $s$ as before: $x_\text{new}^* = (x_\text{new}-\overline{x})/s$. Observe that the prediction with this standardized predictor is equivalent to using the unstandardized linear regression and unstandardized regression coefficient estimates.
$$
y_\text{new}^* = \widehat{\alpha}^* + \widehat{\beta}^*x_\text{new}^* = \widehat{\alpha}^* + \widehat{\beta}^*\frac{x_\text{new}-\overline{x}}{s} = \left(\widehat{\alpha}^* - \frac{\beta^*\overline{x}}{s} \right) + \frac{\widehat{\beta}^*}{s}x_\text{new}=\widehat{\alpha}+\widehat{\beta}x_\text{new}.
$$
The final takeaway is that as long as you transform predictors correctly, the prediction value does not change.
