How to calibrate with multiclass classification problem? I am training a model to predict the label (target) based on loan status e.g. 0,1,2,3. So i have 4 classes. I have so far trained a model as follows:
 from HyperclassifierSearch import HyperclassifierSearch

X = data.iloc[:, :-1]
y = data.label

    
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.2, 
random_state=42)
# Create a hold out dataset to train the calibrated model to prevent overfitting
X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, 
stratify=y_train, test_size=0.2, random_state=42)
categorical_transformer = OneHotEncoder(handle_unknown='ignore')
              
numeric_transformer = Pipeline(steps=[('imputer',SimpleImputer(missing_values=np.nan, fill_value=0) ),('scaler', StandardScaler())])
   
preprocessor = ColumnTransformer(transformers=[('num', numeric_transformer, numeric_cols),
                                    ('cat', categorical_transformer, cat_cols)])

#then i use hyperclassifer library
models = {  'xgb': Pipeline(steps=[('preprocessor', preprocessor),('clf', XGBClassifier(objective='multi:softprob'))]),
                       'rf': Pipeline(steps=[('preprocessor', preprocessor),('clf', RandomForestClassifier(criterion = 'entropy', random_state = 42))]) }


search = HyperclassifierSearch(models, params)
best_grid = search.train_model(X_train, y_train, cv=3, n_jobs=-1, scoring='accuracy')
results = search.evaluate_model()
fitted_model = best_grid.best_estimator_

pred = fitted_model.predict_proba(X_test)
labels = fitted_model.predict(X_test)

**note i have omitted alot of the imported libs and params dict since large and only included hyperclassifier since it is large **
my pred is a matrix containing 4 columns where each is related to the class of the loan. Generally i know it is good pracitce to calibrate the probabilities and particularly from tree based algorithms the output is a score not really a probability. I am however confused as to how to calibrate these probabilities.
Usually i would calibrate using the holdout validation set but am unsure how to do it with multiclass
Update
Should i ammend the above xgbclassifier by doing the following:
OneVsRestClassifier(CalibratedClassifierCV(XGBClassifier(objective='multi:softprob'), cv=10))

https://stackoverflow.com/questions/31617530/multiclass-linear-svm-in-python-that-return-probability
My question is what is the correct way to calibrate probabilities from multiclass model?

I think Guillem's answer is most helpful, but I want to point out what seems to be an error or at least an oversight in the Cardoso paper.
The suggestion there is that conditional probabilities are used. For example, in a three-class problem we might have three classes:
$$C_1=low$$
$$C_2=medium$$
$$C_3=high$$
One would run two binary classification models:
Model 1: $p_1 = p(C_x>C_1)$
and
Model 2: $p_2 = p(C_x>C_2)$
Cardoso's method would then say
$$p_{C_1}=  1-p(C_x>C_1)=1-p_1$$
$$p_{C_2}=  (1-p_2)*p(C_x>C_2)=(1-p_2)*p_1$$
$$p_{C_3}=  p(C_x>C_2)=p_2*p_1$$
Let's say you ran a well-behaved model, and it found:
$$p_1= p(C_x>C_1)=0.8$$
$$p_2 = p(C_x>C_2)= 0.4$$
Using the formulas suggested by Cardos0, this would give
$$p_{C_1}=1-0.8=0.2$$
$$p_{C_2} = (1-0.4)*0.8=0.48$$
$$p_{C_3}=0.8*0.4=0.32$$
Using Frank & Hall's formulas, you get:
$$p_{C_1}=0.2$$
$$p_{C_2}= 0.4$$
$$p_{C_3}=0.4$$
Frank and Hall's seems like the correct answer to me.
It seems to me that Cardoso's method is meant to be used where the higher-level models are trained excluding data from the lower classes. In the example above, Model 2 would be given as:
Model 2: $p_2 = p(C_x>C_2|C_x>C_1)$
So in practice you would train model 1 to discriminate $C_1$ vs ($C_2$ and $C_3$) and model 2 to discriminate between $C_3$ and $C_2$.
