Why does the calibration curve looks like this? Context
I am currently working on a heavily imbalanced classification model that predicts if someone is going to list their property in the market in 6 months or not, there are 30 million properties in the UK so potentially a dataset of 30M rows to train on times the amount of times each property has had a listing. However, the model has been trained on data from only one year worth of historical transactions with LightGBM and oversampling for the minority class to decrease the imbalance a little bit (by no means is a balanced dataset, now it is just 4% for the minority class instead of 1%). The training set is divided train/val and  Finally, the model is scored and evaluated in the 30 million properties on the last 6 months.
Now, the problem is not with the modelling, this is just in case I'm missing something, the problem is with the model calibration. I have run the calibration on 10 bins/deciles for 5 different versions of the model (using different points in time) and it looks very well except for the top decile. As you can see in the picture, it is a non decreasing line except for the top decile where it drastically drops. I was wondering if you have any ideas as to why this might happen.


It would help if you could tell more about the descriptors you used, and label the axes.
But what I think we are seeing is that the top scoring cases include many false positives. This is a common problem in classifiers that are designed to classify an minority class. In the PR curve, it shows up as a vertical line segment down from recall=0, precision=1.
You need to examine a sample of the high scored negatives, and try to think of some feature engineering that would help classify them better.
Given that the rest of the calibration curve looks good, you might succeed. But the risk in such work is always that the available descriptors are simply not predictive enough. Imagine that you had to predict a person's weight, based only on nationality and sex - you could make a model, but it could not be a great model.
