h2o, different stopping metric leads to different optimal for hyperparameters
I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package
# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
max_depth = c(5,10,20,30),
min_rows = c(5,10,20,30),
learn_rate = c(0.01,0.05,0.08,0.1),
balance_classes=c(T,F))
# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "mean_per_class_error")
# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
seed = 42, distribution = "multinomial",
training_frame = td.train.hyper.h2o, nfolds = 3,
hyper_params = hyper_params, search_criteria = search_criteria)
# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "mean_per_class_error", decreasing=FALSE)
hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)
This gives as optimal combination for the hyperparameters
learn_rate max_depth min_rows ntrees
0.<PHONE_NUMBER>
Then I am trying to do the same but with different stopping_metric. So in the above i use mean_per_class_error, and in the following i use logloss, so i run the following code:
# create hyperparameter grid
hyper_params = list(ntrees = c(10,20,50,100,200,500),
max_depth = c(5,10,20,30),
min_rows = c(5,10,20,30),
learn_rate = c(0.01,0.05,0.08,0.1),
balance_classes=c(T,F))
# random subset of hyperparameters
search_criteria = list(strategy = "RandomDiscrete", max_models = 192, seed = 42,
stopping_rounds = 15, stopping_tolerance = 1e-3, stopping_metric = "logloss")
# run grid
gbm.grid <- h2o.grid("gbm", grid_id = "gbm.grid", x = names(td.train.h2o)[!names(td.train.h2o)%like%"s_sd_segment"], y = "s_sd_segment",
seed = 42, distribution = "multinomial",
training_frame = td.train.hyper.h2o, nfolds = 3,
hyper_params = hyper_params, search_criteria = search_criteria)
# sort results
gbm.sorted.grid <- h2o.getGrid(grid_id = "gbm.grid", sort_by = "logloss", decreasing=FALSE)
hyperparameters_gbm <- as.data.table(gbm.sorted.grid@summary_table)
which gives as optimal combination for hyperparameters
learn_rate max_depth min_rows ntrees
0.1 20 5 500
I know that i use as argument strategy = "RandomDiscrete", but still for instance the optimal combination for gbm using stopping_metric = "mean_per_class_error" is the "50th optimal combination" for the gbm using stopping_metric = "logloss", and the "2nd optimal combination" for gbm using stopping_metric = "logloss", is the "14th optimal combination" for gbm using stopping_metric = "mean_per_class_error"
Why could that happen ? 613