The validation set includes few positive labels I'm training a classifer on an unbalanced dataset. The test dataset's positive proportion is 0.02%.
For that reason, the validation data set labels proportions are the same. Because the validation set size is much smaller than the test dataset, it contains less than ten positive labels. The test set includes 25 positive labels. I tune the model hyperparameters by using the F-beta score.
I'm not sure that a sample with less than ten positive labels, is a valid sample for tuning and evaluating the classifer. Indeed, the classifer has terrible results when applied to the validation and test sets.
Since the training set is more balanced from the validation and test sets, I can move positive labels from the train set to the validation set (and test set). However, in that way, they will not represent the real data.
What do you recommend me to do?

If the training data is more balanced, why do you not consider adding more positive samples to the validation and test set from the training set? If training data is also not balanced, maybe using representation learning first to learn discriminative features better and then taking these features to apply classification can be an option.
