Binning Calibrated probability scores for business use Context:
We have a model that outputs calibrated probability scores for a binary classification problem (events/nonevents). There is a general business requirement that we bin these outputs further to make the predictions easier for business usage. For example, it is encouraged that we overlay the probability scores with scores of 1 to 5, where 5 will represent prob. scores of say 10%-100%, score 4 will do the same for scores of say 5%-10%...etc. And say one arrives at the bin edge 10-100% because the area contains 90% of the minority class (events) in the test sample and also is 2% of the entire population, and similarly one arrives at bin edge 5-10% because the area contains an additional 5% of the minority class (events) in the test sample which is also 25% of the total population, so by reviewing bins 4 and 5 (5% to 100%  prob. scores) one reviews only 27% of the population but captures 95% of the events.
The question is:
This binning process is extremely qualitative and subject to criticism (why 5 bins, why should we believe 90% of the event population will remain in the 10%-100% in production setting...), does anyone know of a robust method to post-process calibrated probabilities into bins for decision making, without breaking this fundamental rule of machine learning "In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps" ESL II."?
Any resources and links would be appreciated.

If you need to bin the predictions for business reasons, you probably want to do something like treating the bins differently, e.g. assigning them to different marketing campaigns. In such a case, you have in mind optimizing some objective (money, clicks, subscriptions, churn, etc). What you need to do is find such bins that are optimal given the objective. This is an optimization problem. Any other binning strategy would be arbitrary.
