Remedies to CNN-LSTM overfitting on relatively small image dataset
Notes
Using a pretrained model, trying data augmentation (not possible knowing nature of images, lowering number of parameters in the network, all didn't help)
Context
I have a sequence of images. Target is a multivariate continuous time series. I am trying LSTM on CNN without using a pretrained model. Training a CNN model didn't got me a satisfying results. A very sure reason is that train is only on one year. 650
While predicting with test images present is on several months.
Any image augmentation is nearly impossible from nature of images, satellite images on a fixed geo-location, tracking passing clouds.
Along with images, I have time features, and trend, seasonality of target which is known for test set, as it can be scientifically calculated (it's about GHI, estimated by the <PERSON> and <PERSON> model).
Problem
The problem is with over-fitting, tracking best model on validation set is done by early stopping.
Validation set is a small fraction from train, .9 for train, so train set is furthermore made fewer.
Train is a set of 8804 images, and target variable. Timed Model layers take series of 31 sequences. For example train takes (255,31) and validation takes (29,31).
The model, I came up with is the following:
losses_weights=[[1, .4]];
main_input__ = Input(shape=(31, 120, 120, 1), name='main_input__')extraction
x__ = TimeDistributed(
Conv2D(8, kernel_size=(3, 3), strides=(1, 1) , activation='relu')
)(main_input__)
x__ = TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2)))(x__)
x__ = (TimeDistributed(BatchNormalization()))(x__)
x__ = TimeDistributed(Conv2D(8, (2,2), strides=(1, 1), activation='relu'))(x__)
x__ = TimeDistributed(MaxPooling2D((2, 2), strides=(2, 2)))(x__)
# extract features and dropout
x__ = TimeDistributed(Flatten())(x__)
x__ = (TimeDistributed(Dense(8, activation='relu')))(x__)
x__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(x__)
lstm_out__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(x__)
auxiliary_output__ = Dense(8, name='aux_output')(lstm_out__)
auxiliary_input__ = Input(shape=(31, 10), name='aux_input')
z__ = keras.layers.concatenate([lstm_out__, auxiliary_input__])
# We stack a deep densely-connected network on top
# z__ = (LSTM(lstm, return_sequences=True, dropout=0.4))(z__)
z__ = Bidirectional(LSTM(lstm, return_sequences=True, dropout=0.3))(z__)
main_output__ = Dense(8, name='main_output')(z__)
################################################################################
loss=[loss_mse_warmup, loss_mse_warmup];
#;
model__ = Model(inputs=[main_input__, auxiliary_input__], outputs=[main_output__, auxiliary_output__])
model__.compile(loss=loss_mse_warmup, optimizer='adam', loss_weights=loss_weights)
history__ = model__.fit(x=[x_train, aux_train],y=[y_train, y_train], epochs=100, batch_size=2, validation_split=.9, callbacks=callbacks)
loss_mse_warmup is just a mean_squared_error that ignores 5 first training input signals.
Tries
1. Several Batch-size lengths: [32, 16, 8, 2].
2. loss weights ranging in [[1, .4], [1, .3], [1, .2]].
3. Different variants of number of nodes in CNN: [8,16,32].
4. 695