What metrics can I use to compare the predictions of two linear regression models on an independent data set other than RMSE? Model A was fit using 213 predictors, Model B with 252 predictors, on a data set with n=431. I test both on an independent data set (n=121) and am currently using RMSE to determine which model performs better.
I am looking for more measures to compare the predictions of the two models. I looked up mean absolute deviation, but am not sure what exactly what that tells me or if it is useful for a comparison of predictions between both models. Is it the variation of the predictions? Can I use r-squared (r2) or do I need an adjusted r2 since both models have a different number of predictors? I'm not sure how I would calculate the adjusted r2 for the test set because n=121 and p>n for both models. Am i misinterpreting the adjusted r2 calculation?
I have also created a plot of actual vs predicted values. Any suggestions are appreciated.

Mean squared error and mean absolute error are common. Then there are some that arise from time series forecasting but might make sense for you. The easiest to understand is mean absolute percentage error (MAPE), despite its problems. That page also mentions four alternatives to MAPE:

*

*Mean Absolute Scaled Error (MASE)


*Symmetric Mean Absolute Percentage Error (sMAPE)


*Mean Directional Accuracy (MDA)


*Mean Arctangent Absolute Percentage Error (MAAPE)
Regarding $R^2$ and adjusted $R^2$, there is a natural way to use the idea of $R^2$ for out-of-sample data. Regular $R^2$ compares the MSE of your model to the MSE of a model that guesses the mean of $y$ every time, regardless of what the features are.
$$
R^2 = 1 - \dfrac{
\sum_{i=1}^n \big(y_i - \hat y_i\big)^2
}{
\sum_{i=1}^n \big(y_i - \bar y\big)^2
}
$$
Apply this same idea to out-of-sample truth values $y_i$ and predicted values $\hat y_i$, making sure to still use a denominator model that always guesses the same $\bar y$ that you calculated on the in-sample data.
$$
R^2_{out} = 1 - \dfrac{
\sum_{i=1}^n \big(y_i - \hat y_i\big)^2
}{
\sum_{i=1}^n \big(y_i - \bar y_{in}\big)^2
}
$$
Adjusted $R^2$ on test data makes less sense, since the reason for having the penalty for a high parameter count in adjusted $R^2$ is to catch if the model only fits the data used to train the model, rather than being able to generalize. Checking the ability to generalize, rather than just fitting the training data, is exactly what an out-of-sample metric checks.
