Extensive hyperparameter tuning yields nothing, XGBoost classifier I'm looking for an educated reasoning concerning the following.
I have several time run extensive hyperparameter tuning sessions for an XGBoost classifier with Optuna applying large search spaces on n_estimator (100-2000), max_depth(2-14)´and gamma(1-6). In the meantime, I've had set a fixed low learning rate to 0.03 and fixed stochastic sampling (subsample, colsample_bttree and colsample_bylevel, set to 0.6, 0.6, 0.8).
However, the result doesn't improve at all compared to the default model (sklearn XGBClassifier()). Of course, the default settings perhaps are the optimal in my case. I'm using 22 features and the dataset has 200 000 observations. The dataset is imbalanced 1:20 and I'm not using scale_pos_weight.
My initial thought on this is that the dataset isn't complex enough to aloud improvements from hyperparameter tuning. However, that is just my guess. What says the experts on a situation where hyperparameter tuning doesn't yield any improvements at all for an XGBoost classifier?

Tunning the hyperparameters does not have to lead to improvement.

*

*It can be the case that the default parameters were good enough or that different values of parameters do not play an important role in training (e.g. the data is especially good or bad). Maybe you simply reached the limit of what XGBoost can achieve here.

*XGBoost is not the only machine learning model, it is still possible that other models can have better performance.

*If "no improvement" means very bad results, it always can be the case that the problem cannot be solved with machine learning.

*Check your code for bugs, if there are no changes, maybe you are doing something incorrectly. Did you make sure that you are exploring a broad enough region of the hyperparameter space?

