How to run Bayesian optimization experiments in parallel? Suppose I have the following hyperparameters for tuning:

learning_rate: [0.00001, 0.1]
epochs: [200,300,400,....,1000]
batch_size: [16,32,64,128]

If I want to run experiments using 4 parallel jobs, can I just divide one parameter space into 4 subspaces to achieve parallelism? For example,

1st job:
learning_rate: [0.00001, 0.0001]
epochs:[200,300,400,....,1000]
batch_size: [16,32,64,128]


2nd job:
learning_rate: [0.0001, 0.001]
epochs:
[200,300,400,....,1000]
batch_size: [16,32,64,128]

and so on.
Does this make sense?

The problem with partitioning the search space yourself upfront and then running Bayesian optimization independently on each of them is that it is quite likely that most of your partitions don't contain any parameters that would be remotely optimal. That's why you use Bayesian optimization which is supposed to quickly figure out where to concentrate search effort and which regions can be abandoned. It does that usually much better than we humans.
Thus, if you want to benefit from parallel execution, you should employ implementations of Bayesian optimization that use parallel execution internally. The Bayesian optimization method knows best where to look next and where to use parallel compute power. Thus, your multiple cores are used to investigate many promising parameters in parallel instead of wasting them in irrelevant regions.
Such parallel implementations of Bayesian optimization are available, e.g. in R: ParBayesianOptimization, and in python: GPyOpt.
