I am trying to do experiments on multiple data sets. Some are more imbalanced than others. Now, in order to assure fair reporting, we compute F1-Score 3 on test data. In most machine learning models, we train and validate the model via accuracy measure metric. However, this time, I decided to train and validate the model on an F1-score metric measure. Technically, there should be no problems, in my opinion. However, I am wondering if this is the correct approach to go.
Second, when I use this method (training, validation on F1-score), I receive a higher loss error and a lower F1-score on training data than on validation data. I’m not sure why.