Unexpected probability distribution from xgboost binary classification I am testing different a couple of different binary classification models using xgboost to predict the likelihood to convert. The difference between the 2 probability distributions shown below is based on different fields being used to train the models.
The first image shows a distribution that makes sense to me given my understanding of the industry. There's a large number of predictions that are unlikely to convert, a handful that is very likely to convert, and the rest fall somewhere in between.  Note that the first image has had the probabilities rounded and converted to a scale falling between 0 to 100 instead of 0 to 1.

The second image shows the distribution from a different model where the list of available features was restricted, and not only is the range between the lowest and the highest distribution very narrow (lowest ~0.48, highest ~0.51), but the distribution is very heavy on just a handful of specific probabilities.

My questions are:

*

*Does a probability distribution like the latter give reason to
disregard that model as a candidate?

*Is it more typical to see
results like the first or second distribution for these types of
models?

*What kind of features would typically cause the results to
be more like one or the other?

Apologies if these are dumb questions - I don't have a stats background just a very rudimentary understanding of the basics. Any pointers to additional reading are appreciated.

Let me try to answer your question.
First of all, I wanted to highlight that what you are observing here in terms of distribution of the probability of the model is a common effect especially for boosted models, SVM or even naive bayes. You tend to see a sigmoid distortion of the probabilities more prominent towards the extremes rather than an even distribution. If you look at the logic behind these models, you will see it makes sense in the way that they do the classification.

*

*Whether to use the model or not is a decision that ideally should be taken based on the metrics that you are looking for like ROC etc. Unless what you really intend to use are the probability. In that case you should check out something called platt scaling. Check out this link to get more details on it. https://www.analyticsvidhya.com/blog/2016/07/platt-scaling-isotonic-regression-minimize-logloss-error/

*I hope I have answered your concern already.

*This trend is not a function of features but the result of the model you are using.

