Doing hyperparameter tuning in parts For a regression problem, where I'm doing hyperparameter tuning for LGBMRegressor, I was wondering whether I can first tune max_depth and n_estimators and then (in another GridSearchCV round) the regularisation parameters (reg_lambda, learning_rate, reg_alpha)? This will help me save time.

You won't necessarily reach the optimal hyperparameters that way.
The n_estimators and learning_rate are maybe the easiest to see: as you reduce the learning rate, you need more trees to reach the same level of model capacity. In my experience, lowering the learning rate pretty far improves performance, but requires more trees.
Other hyperparameters will generally have the same issue: the optimum value of one depends on the values of others. I don't have enough experience playing with all of them to give confident advice, but max_depth is likely one of the most important hyperparameters, and since its optimal value is usually fairly small and it only takes integer values, optimizing it first might not hurt too much.  The regularization parameters tend to have less impact on performance, so tuning those last may be OK.
