Cost-sensitive Logloss for XGBoost
I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.
$$ \begin{array} \ p &= \frac{1}{1+e^{-x}} \ \hat{y} &= min(max(p, 10^{-7}, 1-10^{-7}) \ FN &= y \times log(\hat{y}) \ FP &= (1-y) \times log(1-\hat{y}) \ Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP \end{array} $$ I have calculated the gradient and hessian for this loss function: $$ \begin{array} \ \frac{dLoss}{dx} &= 4py + p - 5y \ \frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p) \end{array} $$
And my code:
def logistic_obj(y_hat, dtrain):
y = dtrain.get_label()
p = 1.0 / (1.0 + np.exp(-y_hat))
grad = 4 * p * y + p - 5 * y
hess = (4 * y + 1) * (p * (1.0 - p))
return grad, hess
def err_rate(y_hat, dtrain):
y = dtrain.get_label()
y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
loss_fn = y*np.log(y_hat)
loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)
xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic',
'max_depth': 6, 'tree_method': 'hist', 'seed': 42}
model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'),
(d_val, 'vld')], obj=logistic_obj, feval=err_rate)
Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective directive, it apparently defaults to RMSE which makes matters worse.
[0] trn-error:0.065108 vld-error:0.056749 trn-error:<PHONE_NUMBER> vld-error:<PHONE_NUMBER>
[1] trn-error:0.064876 vld-error:0.056645 trn-error:<PHONE_NUMBER> vld-error:<PHONE_NUMBER>
[2] trn-error:0.064487 vld-error:0.05651 trn-error:<PHONE_NUMBER> vld-error:<PHONE_NUMBER>
[3] trn-error:0.064573 vld-error:0.056553 trn-error:<PHONE_NUMBER> vld-error:0.64864
[4] trn-error:0.064484 vld-error:0.056514 trn-error:<PHONE_NUMBER> vld-error:0.649974
[5] trn-error:0.064483 vld-error:0.056514 trn-error:<PHONE_NUMBER> vld-error:<PHONE_NUMBER>
[6] trn-error:0.064470 vld-error:0.056507 trn-error:<PHONE_NUMBER> vld-error:<PHONE_NUMBER>
[7] trn-error:0.064466 vld-error:0.056506 trn-error:<PHONE_NUMBER> vld-error:0. 509