When is an unbalanced dataset large enough for calculating a decision threshold? I have a (large i.e. >1M rows) very unbalanced (1% event label, binary classification) dataset with data from various institutions. At the moment, I train an XGBoost model on this data and get good performance on average (in production) across all institutions. The decision threshold is calculated on the test dataset to give a model with a defined target level of precision (we care about false positives).
This means, however, that for some institutions the performance is subpar and being more or less strict i.e. having a lower or higher decision threshold would be beneficial. I understand I'm walking into overfitting territory, but at the same time the individual institutions complain they're not getting good performance.
So my question is: how would you determine if a dataset (in this case, the data relative to a certain institution) is large and diversified enough (enough rows, but also enough event labels) to undertake that threshold optimisation (the model is the same, I'm only asking about the decision threshold)?
I understand the answer is usually "depends on the data", but I don't have a heuristic to determine what factors to look into.

1% of $10^6$ is 10,000, which is a pretty good sample size. You could be having problems because 10,000 samples isn't enough to fill the feature space you're dealing with. If I had to guess, though, I'd say that your problem isn't the imbalance, it's that the imbalance varies from institution to institution. If you're wedded to XGBoost, or similar, I would recommend adding the institution to the list of labels you train on, and see if that helps.
If you're allowed to change model, though, I would recommend training a model that outputs probabilities (i.e. trained via logistic regression). Then you can use the balance of the overall set as a Bayesian prior probability to remove from the prediction and replace with a prior reflective of the individual institution.
You can then do a ROC analysis to set decision boundary, do some form of expected value calculation, or do game theory if multiple people are involved.
If the 2008 financial crash taught us anything, it's that the data can have unexpected correlations. That is, there may be some important time-series information in here. For instance, the classes of the previous $k$ samples may have relevant information about whether the next sample is of the rare class (examples: a machine on the line broke, so the usually rare defective part is now common; or a good experience by one person who falls in the rare class means they've spread good word-of-mouth among their friends who are more likely than average to be from the rare class, etc).
