Precision, recall and accuracy metrics significantly different between training/validation and actual predictions
I have two sequential models built with Keras that train on data from a CSV file. This is how they are built ```python
First model
model = Sequential() model.add(Input(shape=X_1.shape)) model.add(Dense(128, activation='relu', kernel_initializer='he_uniform')) model.add(Dense(64, activation='relu', kernel_initializer='he_uniform')) model.add(Dropout(0.7)) model.add(Dense(32, activation='relu', kernel_initializer='he_uniform')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer=Adam(learning_rate=0.001), metrics=[recall, prec, ba])
Second Model
model_2 = Sequential() model_2.add(Input(shape=X_1.shape)) model_2.add(Dense(64, activation='relu', kernel_initializer='he_uniform')) model_2.add(Dropout(0.5)) model_2.add(Dense(32, activation='relu', kernel_initializer='he_uniform')) model_2.add(Dense(1, activation='sigmoid')) model_2.compile(loss='binary_crossentropy', optimizer=Adam(learning_rate=0.001), metrics=[recall, prec, ba]) ```
My dataset is split in half, one for training/validation and one for testing, also I have about 47k rows of data across 19 columns. Most of them are categorical, just one is numerical, all of the categorical data is one hot encoded and the numerical is normalized using MinMaxScaler, When training the model I use the built-in metrics from Keras for Recall, Precision, Accuracy and I get decent numbers of above 0.7 for these. 392
I calculate the F1 score of my results manually.
I then test the trained model on the resting data (the second half of the dataset) and calculate the metrics using Sklearn as I wasn't able to find some built-in functions for that in Keras. I use the below code.
```
predict crisp classes for test set
yhat_classes = (model.predict(X_2) > 0.5).astype("int32")
Calculate F1 score
accuracy = accuracy_score(y_2, yhat_classes) precision = precision_score(y_2, yhat_classes) recall = recall_score(y_2, yhat_classes) f1 = f1_score(y_2, yhat_classes) ```
On these metrics, I get considerably lower numbers across all of them. Again my validation numbers from the .fit method of the model are exactly what I need. I also use EarlyStopping on my model, watching val_recall when training.
Why are my resulting numbers from the prediction so different than training and how can I improve them? Is there a difference between how Sklearn and Keras calculate Precision and Recall? 392