Selecting the best model parameters from grid search SARIMA I ran a manual gridsearch of SARIMA across several parameters and now I have 7875 rows of scores (RMSE, MAE, MAPE each) from it. These were the parameters (30k+ permutations) I ran a grid search over-
p = [0 to 10]
d = [0,1,2]
q = [0 to 12]
P = [0 to 5]
D = [0,1]
Q = [0,1,2]
S = [0,7]
These are the top 20 rows of the results sorted by RMSE in ascending. Parameters are in the order ((p,d,q),(P,D,Q,S)). How should I go about selecting the best model from this? Should I do it based on the lowest RMSE or should I select a higher one in which the model parameters are not as high?

import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/vyaduvanshi/helper-files/master/df_timeseries.csv')
train_df, test_df, val_df = df[:-60], df[-30:], df[-60:-30]
result_df = pd.read_csv('https://raw.githubusercontent.com/vyaduvanshi/helper-files/master/metrics_timeseries.csv')
result_df = result_df.rename(columns={'Unnamed: 0':'parameters'})

The model is fitted on train_df and predictions made on val_df
Edit:


It's good that you are looking at out-of-sample metrics.
That said, as you see, the errors among your top 20 models do not vary a lot. I would assume the differences are not statistically significant. Thus, whether model A that performs better than model B in this holdout exercise will also perform better on new data is a toss-up. In such a situation, I would definitely use a simpler model rather than a more complex one.
Note that the gold standard in automatic ARIMA fitting only uses orders up to (5,2,5), and with good reason. (Any particular reason why you want to roll your own model selection, rather than go with an established tool?)
You could compare models with the same differencing using information criteria, like AIC, AICc or BIC, choose the optimal model in each case and then use your errors to decide which differencing order is the one you want to go with. (Note that you can't compare information criteria between models with different orders of differencing.)
