What metrics can I use to compare the predictions of two linear regression models on an independent data set other than RMSE? Model A was fit using 213 predictors, Model B with 252 predictors, on a data set with n=431. I test both on an independent data set (n=121) and am currently using RMSE to determine which model performs better.
I am looking for more measures to compare the predictions of the two models. I looked up mean absolute deviation, but am not sure what exactly what that tells me or if it is useful for a comparison of predictions between both models. Is it the variation of the predictions? Can I use r-squared (r2) or do I need an adjusted r2 since both models have a different number of predictors? I'm not sure how I would calculate the adjusted r2 for the test set because n=121 and p>n for both models. Am i misinterpreting the adjusted r2 calculation?
I have also created a plot of actual vs predicted values. Any suggestions are appreciated.

If you are mainly interested in metric for prediction in linear regression. Two of the most common metrices are Mean Square Error(MSE) and Mean Absolute Error (MAE). (Note: RMSE is square root of MSE)
The main difference between the two is they penalize difference between predicted value and actual value (i.e. residual) differently. Recall MSE is $\frac{1}{n} \sum_{i=1}^n (\hat{y}_i-y_i)^2$, where $\hat{y}_i$ is the predicted value, $y_i$ is the actual value and $n$ is the sample size. This indicates the residual $(\hat{y}_i-y_i)$ is penalized quadratically. On the other hand, MAE is defined as $\frac{1}{n} \sum_{i=1}^n |\hat{y}_i-y_i|$, which indicates the residual is penalized linearly. So the deciding factor is do you want to penalized more on larger residuals/ differences? If so, use MSE. If not, use MAE.
$R^2$ and adjusted-$R^2$ (from lm output) are used for model selection and they have nothing to do with predictive power of the model. Therefore, they are not useful based on your title and description.
