Why do we need separate data for probability calibration? Why do we need separate data for probability calibration?
Scikit learn documentation says:
The samples that are used to fit the calibrator should not be the same samples used to fit the classifier, as this would introduce bias. This is because performance of the classifier on its training data would be better than for novel data. Using the classifier output of training data to fit the calibrator would thus result in a biased calibrator that maps to probabilities closer to 0 and 1 than it should.
Can someone provide more details? I can't decide if I want a biased calibrator trained on a massive amount of data or if I want an un-biased calibrator trained on little data.

If you first fit your model to some data and then calibrate it using the same data, you will follow the idiosyncrasies of this dataset too closely and overfit. It makes much more sense to split your training data into two parts: fit the model on one part and calibrate on the other. (Or use a method that gives you unbiased probabilistic classifications on the entire training sample.)

I can't decide if I want a biased calibrator trained on a massive amount of data or if I want an un-biased calibrator trained on little data.

If you have a biased model, it is systematically wrong. If you have trained this on a large amount of data, you will probably be very certain of your wrong model. This is not a good place to be. Better to have an unbiased model with more uncertainty.
