Performance evaluation with non-representative data I am currently trying to apply some models for text classification on a binary task. The core two approaches that I follow are, on the one hand, using word2vec vector representations on a Random Forest and, on the other hand, using a fined tuned BERT.
My issue is that my training data has been labelled by me in some sort of active learning fashion: I started labelling some examples, then checked the performance on the top classification, relabel and feed back into the training data. After several rounds, this process led me with a dataset in which the class unbalancedness is definitely not even close to the one that the algorithms actually see when I feed in new data. To give you some quick numbers, the proportion of 1s on my training data is around 40% whereas the one resulting from a normal classification can be around 10% or lower.
This interferes with my performance testing. Since I test different metrics on a test set which is way less imbalanced than the "real" distribution, I can't be sure that this is indeed a valid procedure to do comparisons across models. Augmenting the training data to make it more realistic is technically infeasible ( I would need to classify way too many examples atm).
Is there any method to produce reliable performance metrics when this happens? I thought about synthetic sampling to augment the volume of 0s but I fear this might just be producing similar results...
EDIT: Scheme of the process:

Train and test set are taken from the labelled data with class distribution 40-60%. This does not reflect the distribution of the unlabelled data (much more imbalanced), and hence evaluation metrics can't be trusted since this is not what the algorithm faces when it has to do predictions.

As @Björn explains, your test set is biased because you created it by deterministically choosing the "top examples" to label. This introduces bias in the evaluation that you cannot correct for.
However, you don't necessarily need to sample examples randomly to evaluate machine learning models correctly if you are more strategic in how you select examples to label.
This is called active testing; you can learn more about it in [1].
The gist: It's okay to introduce bias by actively selecting test examples to label as long as it is possible to remove the bias afterwards. Obviously this makes for more efficient evaluation; it can also reduce the variance of the performance estimator.
The procedure outlined in [1] uses a surrogate model to estimate acquisition probabilities, which indicate how valuable an example is to label next. So instead of choosing an example deterministically, you sample it according to its acquisition probability. The idea behind using the surrogate is to (a) account for uncertainty over the outcomes; (b) make predictions that are diverse to the model f(x) under evaluation, and (c) incorporate information from all available data.
[1] J. Kossen, S. Farquhar, Y. Gal, and T. Rainforth. Active testing: Sample-efficient model evaluation. In Proceedings of the 38th International Conference on Machine Learning, 2021. https://doi.org/10.48550/arXiv.2103.05331
