title: Keras LSTM: a time-series multi-step multi-features forecasting - poor results
body: <p>I have a time series dataset containing data from a whole year (date is the index). The data was measured every 15 min (during whole year) which results in 96 timesteps a day. The data is already normalized. The variables are correlated. All the variables except the VAR are weather measures.</p>
  
  <p>VAR is seasonal in a day period and in a week period (as it looks a bit different on weekend, but more less the same every weekend). VAR values are stationary. 
  I would like to predict values of VAR for next two days (192 steps ahead) and for next seven days (672 steps ahead).</p>
  
  <p>Here is the sample of the dataset:</p>
  
  <pre><code>DateIdx               VAR       dewpt       hum         press       temp
  2017-04-17 00:00:00   0.369397  0.155039    0.386792    0.196721    0.238889
  2017-04-17 00:15:00   0.363214  0.147287    0.429245    0.196721    0.233333
  2017-04-17 00:30:00   0.357032  0.139535    0.471698    0.196721    0.227778
  2017-04-17 00:45:00   0.323029  0.127907    0.429245    0.204918    0.219444
  2017-04-17 01:00:00   0.347759  0.116279    0.386792    0.213115    0.211111
  2017-04-17 01:15:00   0.346213  0.127907    0.476415    0.204918    0.169444
  2017-04-17 01:30:00   0.259660  0.139535    0.566038    0.196721    0.127778
  2017-04-17 01:45:00   0.205564  0.073643    0.523585    0.172131    0.091667
  2017-04-17 02:00:00   0.157650  0.007752    0.481132    0.147541    0.055556
  2017-04-17 02:15:00   0.122101  0.003876    0.476415    0.122951    0.091667
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/3DgL2.png" rel="noreferrer">Input dataset plot</a></p>
  
  <p>I have decided to use LSTM in Keras. Having data from the whole year, I have used data from past 329 days as a training data and the rest for a validation during the training.
  train_X -> contains whole measures including VAR from 329 days
  train_Y -> contains only VAR from 329 days. The value is shifted one step ahead.
  The rest timesteps goes to test_X and test_Y.</p>
  
  <p>Here is the code I prepare train_X and train_Y:</p>
  
  <pre><code>#X -&gt; is the whole dataframe
  #Y -&gt; is a vector of VAR from whole dataframe, already shifted 1 step ahead
  
  #329 * 96 = 31584
  train_X = X[:31584]
  train_X = train_X.reshape(train_X.shape[0],1,5)
  train_Y = Y[:31584]  
  train_Y = train_Y.reshape(train_Y.shape[0],1)
  </code></pre>
  
  <p>To predict next VAR value I would like to use past 672 timesteps (whole week measures). For this reason I have set <code>batch_size=672</code>, so that the ‘fit’ command look like this:</p>
  
  <pre><code>history = model.fit(train_X, train_Y, epochs=50, batch_size=672, validation_data=(test_X, test_Y), shuffle=False)  
  </code></pre>
  
  <p>Here is the architecture of my network:</p>
  
  <pre><code>model = models.Sequential()
  model.add(layers.LSTM(672, input_shape=(None, 672), return_sequences=True))
  model.add(layers.Dropout(0.2))
  model.add(layers.LSTM(336, return_sequences=True))
  model.add(layers.Dropout(0.2))
  model.add(layers.LSTM(168, return_sequences=True))
  model.add(layers.Dropout(0.2))
  model.add(layers.LSTM(84, return_sequences=True))
  model.add(layers.Dropout(0.2))
  model.add(layers.LSTM(21, return_sequences=False))
  model.add(layers.Dense(1))
  model.compile(loss='mae', optimizer='adam')
  model.summary()
  </code></pre>
  
  <p>From the plot below we can see that the network has learn ‘something’ after 50 epochs:</p>
  
  <p><a href="https://i.stack.imgur.com/ZhtS9.png" rel="noreferrer">Plot from the learning process</a></p>
  
  <p>For the prediction purpose I have prepared a set of data containing last 672 steps with all values and 96 without VAR value – which should be predicted. I also used autoregression, so I updated VAR after each prediction and used it for next prediction. </p>
  
  <p>The predX dataset (used for prediction) looks like this:</p>
  
  <pre><code>print(predX['VAR'][668:677])
  
  DateIdx            VAR
  2017-04-23 23:00:00    0.307573
  2017-04-23 23:15:00    0.278207
  2017-04-23 23:30:00    0.284390
  2017-04-23 23:45:00    0.309118
  2017-04-24 00:00:00         NaN
  2017-04-24 00:15:00         NaN
  2017-04-24 00:30:00         NaN
  2017-04-24 00:45:00         NaN
  2017-04-24 01:00:00         NaN
  Name: VAR, dtype: float64
  </code></pre>
  
  <p>Here is the code (autoregression) I have used to predict next 96 steps:</p>
  
  <pre><code>stepsAhead = 96
  historySteps = 672
  
  for i in range(0,stepsAhead):
      j = i + historySteps
      ypred = model.predict(predX.values[i:j].reshape(1,historySteps,5))
      predX['VAR'][j] = ypred
  </code></pre>
  
  <p>Unfortunately the results are very poor and very far from the expectations:</p>
  
  <p><a href="https://i.stack.imgur.com/BI3LW.png" rel="noreferrer">A Plot with predicted data</a></p>
  
  <p>Results combined with a previous day:</p>
  
  <p><a href="https://i.stack.imgur.com/fbx4t.png" rel="noreferrer">Predicted data combined with a previous day</a></p>
  
  <p>Except from the ‘What have I done wrong‘ question, I would like to ask a few questions:</p>
  
  <p><strong>Q1.</strong> During model fifing, I have just put the whole history in batches of 672 size. Is it correct? How should I organize the dataset for the model fitting? What options do I have? Should I use the “sliding window” approach (like in the link here: <a href="https://machinelearningmastery.com/promise-recurrent-neural-networks-time-series-forecasting/" rel="noreferrer">https://machinelearningmastery.com/promise-recurrent-neural-networks-time-series-forecasting/</a> )?</p>
  
  <p><strong>Q2.</strong> Is the 50 epochs enough? What is the common practice here? Maybe the network is underfitted  resulting in poor prediction? So far I tried 200 epoch with the same result.</p>
  
  <p><strong>Q3.</strong> Should I try a different architecture? Is the proposed network ‘big enough’ to handle such a data? Maybe a “stateful” network is the right approach here?</p>
  
  <p><strong>Q4.</strong> Did I implement the autoregression correctly? Is there any other approach to make a prediction for many steps ahead e.g. 192 or 672 like in my case?</p>