Threshold optimization with cross validation I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the optimal hyperparameters. I'm unsure how to determine the optimal threshold with cross-validation.
Does the following process make sense:

*

*loop through each fold

*train the model

*determine the optimal threshold on the train set

*calculate the f1 score on the held-out set using the threshold obtained from step 3.

The above process leads to 5 thresholds. I select the threshold with the best f1 score on the hold-out sets. Lastly, finalize the model assessment on the test set.

The correct and efficient steps should be as follows:

*

*loop through each fold

*train the model

*try each threshold value you want to test on the validation fold

*report the threshold that gives the best performance metric (in your case F1) when averaged over all folds

The only difference compared to a typical hyper-parameter is that you do not need to train the model for trying different thresholds.
I'd also like to note that the thresholds are not fully regarded as hyper-parameters of the underlying model as they do not change the model flexibility but only reflect a measure of your classifier's trade-off.
