Evaluating a multi-step forecasting model? The literature is a bit confusing for me on this one, from what I understand, a great deal of papers evaluate multi-step forecasting models on a single forecasting horizon on the hold out set. It seems a bit silly to me to evaluate the whole model on a single forecasting horizon.
But say I'm interested in evaluating a model on a 24 step forecasting horizon. My current way of doing it is training and hyperparameter tuning on a training set and then rolling window forecasting on the test set.
My main question is, is this a valid way to evaluate my model? If not could somebody point me to a paper with a different evaluation scheme?
A side question would be about a proper forecasting metric or more precisely, say you've chosen a metric for evaluating a single forecast horizon (MAE in my case, percentage errors are problematic for me since I have a large number of zero values in the time series), and then calculating the mean of the chosen metric over all forecast horizons. I've seen that approach used in a few places, alongside other MAE statistics, but I'm struggling to find a consensus in the literature.
I know this question might be unfocused, but these two questions come in the same package and it would be nice to have them both answered in the same place.

Q1: Training and hyperparameter tuning on a training set and then rolling window forecasting on the test set for forecast evaluation sounds fine.
Q2: How you define the overall accuracy across horizons by a single number depends on your use case. E.g. if you are making bets of equal size on each horizon, the overall loss will be the sum of losses across horizons. If you are making a bet that depends on the worst misprediction across horizons alone, the overall loss will be the maximum of the losses across horizons. Etc., etc.
The question is somewhat similar to how you choose the evaluation metric (loss function) for a single horizon; there is no generally correct answer there, all depends on the particular user's sensitivity to the various possible forecast errors.
