Stack Exchange
Tensorflow LSTM; using TimeseriesGenerator in prediction evaluation - strange prediction dimensions shape
I have trouble on using TimeseriesGenerator once a Tensorflow LSTM has been correctly trained. As it often happens with Tensorflow, I cannot find good and exhaustive references on using a generator for the predictions evaluation.
The data has been prepared in this way
expansion_factor = 8
batch_size = 512 #256
timesteps = 8

generator = TimeseriesGenerator(X_train_s, pd.DataFrame.to_numpy(Y_train[['TARGET_call_ITM']]), length=timesteps, batch_size=batch_size)

validation_generator = TimeseriesGenerator(X_test_s, 
 pd.DataFrame.to_numpy(Y_test[['TARGET_call_ITM']]), length=timesteps, 
batch_size=batch_size) 

The model is something of simple
loss = 'binary_crossentropy' # binary_crossentropy categorical_crossentropy
activation = 'sigmoid' #softmax sigmoid

early_stop = EarlyStopping(monitor = 'val_accuracy', mode = 'max', verbose = 1, patience = 200)

# Define the checkpoint file path
checkpoint_path = path +  'LSTM ' + underlying_name + further_filename_data + 'OPTIONS MODEL.h5'

# Crea un modello sequenziale
model = Sequential()

# Define the ModelCheckpoint callback
checkpoint_callback = ModelCheckpoint(checkpoint_path, 
                                      monitor='val_accuracy', 
                                      mode='max', 
                                      verbose=1, 
                                      save_best_only=True)

# Aggiungi il primo livello LSTM
model.add(LSTM(units=X_train_s.shape[1]*expansion_factor, 
               return_sequences=True, 
               input_shape=(timesteps, X_train_s.shape[1]),
               activation = 'tanh'))

# Aggiungi il secondo livello LSTM
model.add(LSTM(units=int(X_train_s.shape[1]*expansion_factor/4), return_sequences=True, activation = 'tanh'))

# Aggiungi un livello completamente connesso (Dense) per l'output
model.add(Dense(units=1, activation = activation))  # 'output_dim' dovrebbe essere il numero di classi nel tuo problema

# Compila il modello
model.compile(optimizer = tensorflow.keras.optimizers.Adam(learning_rate=0.0003),                      
                     loss = loss, 
                     metrics = ['accuracy'])


# Stampa una rappresentazione del modello
model.summary()

# model fit
model.fit(generator,
          epochs=120,
          validation_data=validation_generator,
          callbacks=[early_stop, checkpoint_callback])

Troubles arise on prediction evaluation; I would like to replicate something I already have developed for evaluating ANN model, as follow
# Prediciotns calculation

predictions = model.predict(validation_generator)
predictions_df = pd.DataFrame(predictions)

# Cut off predictions with low probability

predictions_cutoff = pd.DataFrame()

min_probability_call_for_1 = 0.57
min_probability_call_for_0 = 0.0008

# predictions_01_3_2['PRED_call_ITM'] = predictions_df_12[0].apply(lambda x: 1 if x > min_probability_call else 0 ).values
predictions_cutoff['PRED_call_ITM'] = predictions_df[0].apply(lambda x: 1 if x > min_probability_call_for_1 else (0 if x < min_probability_call_for_0 else -1) ).values

# Show summary report - Training
predictions_cutoff['PRED_call_ITM']))
print('PRED_call_ITM CLASSIFICATION REPORT')
print(classification_report(Y_test[['TARGET_call_ITM']], predictions_cutoff['PRED_call_ITM']))

I think there are some troubles with the predictions shape given by the generators. The first evidence I had on this came with an error when trying to convert predictions into a dataframe
ValueError                                Traceback (most recent call last)
Cell In[12], line 4
      1 # Prediciotns calculation
      3 predictions = model.predict(validation_generator)
----> 4 predictions_df = pd.DataFrame(predictions)

ValueError: Must pass 2-d input. shape=(883395, 8, 1)

Printing the shape of predictions i get
predictions.shape
(883395, 8, 1)

and printing the first element
predictions[1]
array([[0.5422559 ],
       [0.6785833 ],
       [0.8133916 ],
       [0.8472937 ],
       [0.8740182 ],
       [0.8937476 ],
       [0.9085225 ],
       [0.91809547]], dtype=float32)

This is different than what I was expecting, but maybe I am wrong. With Timesteps = 8, my expectation was that for each chronologically serialized group of 8 rows of features, the model gave as output 1 prediction; instead here I apparently have 8 predictions for each group of 8 rows of features.
I tried also a different approach by creating new data structures with the correct dimensions for feeding the LSTM network as follows
n_seq_train = len(X_train_s) - timesteps + 1
X_train_s_LSTM = np.array([X_train_s[i:(i+timesteps)] for i in range(n_seq_train)])
X_train_s = []
gc.collect()

n_seq_test = len(X_test_s) - timesteps + 1
X_test_s_LSTM = np.array([X_test_s[i:(i+timesteps)] for i in range(n_seq_test)])
X_test_s = []
gc.collect()

Y_train_LSTM = Y_train[timesteps-1:]

Y_test_LSTM = Y_test[timesteps-1:]

but this regretfully saturates the memory in my PC, in kaggle and also Google Collab. I imagine that the TimeseriesGenerator offers a smarter way of creating the correct dimensionality on the fly avoiding replicating data structures.
Anyway, the final part was just for evaluating the precision and recall of the forecasts compared with the Y_test targets, passing through some thresholds. Maybe with TimeseriesGenerator there are other ways? Any suggestion? Thank you.
