Time Series Forecasting with LSTMs in keras - convergence problem
I am trying to forecast a time series with multivariate input and multi output (multi step forecast). Since some of my input features are known for future time steps, wheras others are not, naturally including the target variable itself, I am facing the problem that the input will not fit the 3-D shape needed by an LSTM. I tried different approaches to handle this. What seems to work quite well is to split the data by the last day on that all features are known and build two different 3-d-inputs. input_past is a matrix of shape (x, y1, z1) with x: record count y1: context window (time steps to use in lstm layer) z1: feature count
input_future is a matrix of shape (x, y2, z2) with x: record count y1: time steps to forecast z1: count of features known at future time steps
input_past is passed to a LSTM Layer, input future to a Dense Layer. These layers are being concatenated and densified afterwards.
The results are quite good and outperform different other models including naive models, lightgbm models and arima models in some cases. Which leeds me to my two questions:
1. Is the described model reasonable or is there a better way of handling partly known future features in time series forecasting using LSTMs?
2. The results of a NN is non-deterministic. 856
As described above, the model is able to outperform different other approaches, but not every time the model is evaluated. Evaluating the model on the same input with the exact same configuration may lead to extremely different forecasting performance. The results may vary from best scores (MAPE ~10%, RMSE ~60k) to (MAPE ~30%, RMSE ~200k). I have not been able to make my model converge to nearly the best score in every time it is being run. I tried pretty much everything from varying the learning rate of the ADAM optimizer (even exponential decay schedules) to make the network deeper and/or wider or experimenting with the batch_size or different modes of initial weights of the layers. The only thing that made the scores a little more reliable was to increase epsilon parameter of the optimizer but that led to very poor forecasting performance. With this varying scores on multiple runs of the same model, how do I know, which version of my model is the best to use in production? Any help would be greatly appreciated, since I am quite a newbie using keras.
I am explicitly NOT asking for reproducability. I dont want to set a random seed. What I want is a way the model converges to at least near optimum weights in every run. 650