"General" domain hyperparameter tuning across classes I am conducting experiments which requires me to create a somewhat well performing model while avoiding (as much as possible) the computational expensive search over sets of hyperparameters.
My experiments involved further pretraining on a pretrained language model, using a masked language modelling objective on extra data, and are conducted over a number of domains, each with various constituent class labels, e.g. Amazon product reviews with 30 or so product categories.
Since I want to avoid the expense of hyperparameter tuning my language modelling tasks across each of the 30 product categories, could I then effectively capture a reasonable set of hyperparameters by conducting a grid search over randomly sampled documents for the whole domain?
For instance, randomly sampling 50k documents over all 30 product categories, with the goal of capturing the general language and document structure of the domain? I understand that I won't find the best settings when then testing these tasks in isolation, but I was wondering if this was a generally theoretically sound approach for I guess, "domain" tuning, since the masked language modelling task does not take into account class labels?

It is to be expected that your tuned BERT model, MLM tuned over the complete domain (containing all the product categories (classes)), performs better than a vanilla off-the-shelf BERT model. Of course, you should use some hyperparameter tuning software like keras tuner or optuna which can use Bayesian optimization to tweak the search.
Also, you might want to compare your results with that of a vanilla BERT to see how much effect this additional work has.
