I should set num_threads = 2?

Yes. And make sure this is faster (otherwise revert back) 

Tuner

Good point. I see what I can do. Maybe TF-DF could catch some of these issues (maybe caused by incompatible configuration) and give informative error messages…
A nice example was created by a user.

confidence bounds or t-test

Confidence intervals (CI) on the accuracy can be computed with the Wilson Score Interval and CI on AUC can be computed with Hanley et al method. Alternatively, CI can be computed on any metrics using bootstrapping (i.e. you sample with replacement in your predictions, and estimate the CI empirically; I am a big fan of this  ).
Note that those CI will not contain the training noise (unless you use some form of repeated training / cross-validation).
McNemar test 2 is probably suited for accuracy on pairwise data (which should be the case i.e. you use the same test dataset on both candidate models).