First, you have too little data. 50 samples. Think of it. Of all possible 2001 sequences of that kind, you are only feeding 50 (less than 2.5%). Your question is actually a very good case to show the importance of big data for training a neural network.
And second, it is the problem stated in this arxiv article made by Uber empolyees - even a deep neural network with nonlinear units often cannot generalize to solution of a simple linear problem.
Your problem still can be approximately solved with
1. Reducing batch size
It is well known that increasing batch size leads to faster but poorer convergence. 716
Particularly in your case, you should use line:
python model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1, batch_size=1, shuffle=True)
instead of
python model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)
even knowing that it will be slooooow...
1. Modifying LSTM layer: removing unit_forget_bias=True and adding orthogonal initializer with small gain - explained in this article:
python from keras.initializers import orthogonal model.add(LSTM(10, activation='tanh', #unit_forget_bias=True, kernel_initializer=orthogonal(gain=.01), ))
1. Increasing coverage of the space of all possible samples
In the code below SEQ_LEN=2000 and N_SAMPLES=50 is your case. Here we generate random sequences like ones that your code above (although not the same sequences as yours). Now, try experimenting with values SEQ_LEN=200 and N_SAMPLES=80 and then SEQ_LEN=200 and N_SAMPLES=130. See if with N_SAMPLES<SEQ_LEN/2 your net converges at all, while in second case it will!
```python
import numpy as np
SEQ_LEN = 2000 N_SAMPLES = 50 n_test_samples = int(.1*N_SAMPLES) if N_SAMPLES>SEQ_LEN+1: raise ValueError("Can't create more samples of this type than SEQ_LEN+1 is.") ns = np.random.randint(0,SEQ_LEN,size=(N_SAMPLES,)) all_Xs = np.triu ( np.ones(shape=(SEQ_LEN+1,SEQ_LEN)) ) np.random.shuffle(all_Xs)
Xs = all_Xs[:N_SAMPLES, ...] Ys = Xs.mean(axis=1) Xs = Xs.reshape(N_SAMPLES, SEQ_LEN, -1) trainX, testX, trainY, testY = Xs[:-n_test_samples], Xs[-n_test_samples:], Ys[:-n_test_samples], Ys[-n_test_samples:]
import tensorflow # To prevent import tensorflow as tf # to declarations to prevent cryptic errors on my Windows laptop
from keras import Sequential from keras.layers import Masking, LSTM, Dense from keras.optimizers import adam from keras.initializers import orthogonal from keras.layers import Flatten
model = Sequential() model.add(Masking(mask_value=0.00, input_shape=(trainX.shape1,trainX.shape2))) model.add(LSTM(10, activation='tanh', #unit_forget_bias=True, input_shape=(trainX.shape1,trainX.shape2), kernel_initializer=orthogonal(gain=.01), )) model.add(Dense(1,activation='relu')) model.compile(optimizer=adam(),loss='mse',metrics=['mae']) model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1, batch_size=1, shuffle=True)
testY_PRED = model.predict(testX) print (np.concatenate(( testY_PRED, testY.reshape(-1,1)), axis=1) )
```
I haven also tested case with 300 samples and sequence length of 500 (converges). 392