Hunting "Sweetspots" In Predictive Models I have three separate predictive models that each accept a set of inputs about an upcoming event and produce a probability of that event being true.
I have domain knowledge that leads me to suspect that some models have a sweet spot relationship with the true probability of the event.
For example, Model A seems to be a good all around predictor, it's the closest approximate to true probability that I have.
I suspect that when Model A predicts a very low probability, Model B is actually slightly better.
Likewise, I suspect that when the true probability of an event is very high that model C is slightly better.
How could I go about testing and quantifying this?

Here's the approach I'm thinking of, and I have no doubt that's it crude.
I have a database of about 3000 historic events with predictions from all three models. I'm thinking about putting the predictions into buckets of 2% and counting the number of events and correct predictions in each bucket. For example (in theory) if there are 50 events in the 40-42% probability bucket then there should be around 21 correct predictions.
If there are significantly more or less then that would likely indicate a range that the model does poorly over. I would of course need to smooth the line formed by such a bucketing a approach, I'm sure there will be outliers, but it sounds like it would give a good idea of the general trend. I could probably throw some errors bars on there too to get a feel for the confidence associated with any given range.
Here's a quick and dirty sketch of what my idea looks like.

The diagonal line indicates an imaginary perfect model, where if the model forecasts a 50% correct probability, then over many many events the actual correct rate is 50%. Anywhere our buckets are close to or touching this line is a sweetspot.
This is one admittedly rough idea. Are there any common approaches to figuring out if a predictive model has a sweetspot? I have some hazy ideas about how logistic regressions could be used here but this question is long enough without any more half formed thoughts.

You seem to be interested in what is called calibration - an important characteristic that is too often ignored. It can be also assessed using LOESS plots, which can be a bit more stable then somewhat arbitrary binning. When a model is not well-calibrated various techniques like Platt-scaling can be tried to remedy this.
When you have multiple models that perform well and do different mistakes, then combining them/stacking is often an interesting idea (see e.g. the ensembling chapter of this book). If you think you can somehow predict from some features which model will work better, then you probably want to use predictors of one model being better than anohter in the stacking model. If your stacked model performs better in a sensible evaluation (e.g. some suitable form of cross-validation or whatever else makes sense in the context), then combining the model is probably a good idea.
