Doing hyperparameter tuning in parts For a regression problem, where I'm doing hyperparameter tuning for LGBMRegressor, I was wondering whether I can first tune max_depth and n_estimators and then (in another GridSearchCV round) the regularisation parameters (reg_lambda, learning_rate, reg_alpha)? This will help me save time.

This is actually a common strategy implemented in some software (e.g. in optuna) and commonly used in practice. As indicated in the other answer, it may not lead to the best result given infinite training time, but may actually be better than alternatives if you only have a fixed amount of time available.
The argument for why it might work reasonably well to tune certain parameters in turn is that some of them depend not so much on the other parameters (e.g. in this order: 1. depth of trees 2. proportion of records you subsample per tree 3. minimum child weight 4. proportion to sub-sample predictor columns). Some examples discussing such strategies are this Kaggle forum post, this tutorial on Kaggle. Some people are less than convinced, as e.g. seen in this discussion.
