Looking for an advice based on an output of a LSTM model
I have fitted a LSTM model first using keras.
Data:I have a time series with 560 observations. From that I trained the model using first 500 observations and then evaluated the model using last 60 observations.
I have only one predictor(X) where X is the lag price of the time series. So basically I am trying to predict the current price from the previous days price by fitting a LSTM model.
I used the following setup to fit the model:
model = Sequential()
model.add(LSTM(32, activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.add(Dropout(0.3))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)
Then I evaluated the model for test data. The comparison between the actual price and the predicted price for test data looks like this:
Based on the above plot, It seems that the model is kind of capturing the shape of the actual price. 856
But it is underestimating the price all the time.
Can anyone suggest anything to improve this model?
I am beginner to model fitting in deep learning. So any help will be highly useful.
Thank you.
*Update*** I tried following setup and the results improved drastically:
model = Sequential()
model.add(LSTM(50, activation = 'relu', return_sequences = True,input_shape=(dataXtrain.shape[1], dataXtrain.shape[2])))
model.add(Dense(1))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(dataXtrain, dataYtrain, epochs=1000, batch_size=16)
However it seems that the predicted price capturing the actual trend after some delay. Will it be possible to fix this?
Any advice will be really helpful.
*Edit**
I am updating the question with the code that I used to obtain the fitted values and make the plot.
First I Stored the predictions based on test data:
y_pred = model.predict(dataXtest)
y_pred.shape
When I am fitting the model, I reshaped the data so that it is a subsequence of 2 observations. (For an example 500 training observations are reshaped as (250,2,1). Also the 60 test data reshaped as (30,2,1) )
Then I reshaped the predictions to the original form.
y_pred=y_pred.reshape(60,1)
dataYtest=dataYtest.reshape(60,1)
pred_data=pd.concat([dataYtest,y_pred],axis=1)
pred_data.head(10)
<PHONE_NUMBER>
1 5991 5851.091797
2 6023 <PHONE_NUMBER>
3 5972 5939.657227
4 6028 <PHONE_NUMBER>
5 5429 <PHONE_NUMBER>
6 5697 5390.026367
7 5841 <PHONE_NUMBER>
8 5980 <PHONE_NUMBER>
9 6548 5897.926758
10 6928 <PHONE_NUMBER>
This is how I plotted the data:
plt.figure(figsize=(14,5))
plt.plot(dataYtest, color = 'red', label = 'Actual Price')
plt.plot(y_pred, color = 'blue', label = 'Predicted Price')
plt.title('Test data')
plt.xlabel('Time')
plt.ylabel('Price')
plt.legend()
plt.show() 856