XGBoost Classifier not capturing extreme probabilities I'm using XGBoost for a binary classification task—trying to predict whether team A will beat team B given the score of the game and the time left. I know for certain score-time combinations, the probability of a success (team A wins) is essentially 0 or 1 (e.g. up 20 points with a minute left, it should be ~0.999). This is true empirically (in previous games) regardless of the other features (team quality, pace of play, etc.)
The problem: When I look at fitted values given these score-time combos, the output tends to be around 0.03 or 0.97, rather than .0001 or .9999. I've tried loads of different combinations of parameters and can't get the model to output something close to 0 or 1.
A few more details:

*

*I have about 630k observations, about 55% of which are successes. I also have at least tens of thousands where I believe the probability should be > .99 or < .01.

*Parameters I've messed with + the ranges I've tried: max_depth (from 4 to 12), n_estimators (up to a couple hundred), eta (.001 to .3), min_child_weight (up to a couple hundred), reg_lambda (1-5), gamma (0-5), colsample_byX (0.7-1), and subsample (0.7-1)

*I have not messed with scale_pos_weight since my understanding is that this parameter is for imbalanced datasets. My dataset isn't imbalanced overall, it's just imbalanced at certain places in the feature space

*Loss function is binary:logistic
Any thoughts? Thanks a ton!

This is what I would focus on.

*

*Limitations on max_depth might cause terminal nodes to group together observations with very small probabilities with other observations where probabilities aren't that small, so the effect is to move the leaf weights away from extreme values. Likewise, something similar with large probability observations. Try increasing max_depth.

*lambda penalizes the absolute value of the weights. You want weights with large absolute value, because these weights allow for probabilities closer to 0 and 1, so try setting lambda smaller.

*Column subsampling could omit the important features (time left in the game sounds important), so I wouldn't use it.

*Increasing the maximum number of trees dramatically and using early stopping could help.

*Tuning the learning rate alongside these parameters is important.

Since your question is basically about calibration of probabilities, something to know is that XGBoost is notorious for producing poorly-calibrated predicted probabilities. It's unclear if this is the culprit in your case; usually, the poor calibration arises from predictions that are too close to 0 or 1, but you have the opposite finding here. This is why I think you might be able to close the gap using different hyper-parameters.

I wonder if an XGBoost model is the best approach, because your data are arranged sequentially in time (60, 50, ... 10 minutes remaining, etc.). I would investigate alternative models that can account for this temporal dependency.
If you think about each game as a sequence, the probability of Team A winning should have a wide band around it at the start of the game, and then that band should narrow as the clock runs out. I don't know how to model that, but intuitively, that seems like what you're looking for.
