Hi,
okay, thank you. But why would it introduce a data leak? I mean I never see the test data during training or hyperparameter tuning.
Anyways, I have tried to just train with the train data and it works fine (probably because the validation set is only 10% of the training set, so it does not make a big difference). However, what surprises me, is that if I do not learn thresholds, but just use 0.5 as threshold for every label, I get better test results than taking the middle of the averages of positive and negative samples as a threshold. How can that be possible? Is sigmoid intended to be used with a threshold of 0.5 and there is no need to finetune the thresholds? I find it hard to find any resoruces on how to binarize sigmoid outputs, so I am just trying and guessing right now.