One drawback could be a slightly confusing code, as the optimizer defines another learning rate than the scheduler is allowing. Besides that the scheduler will drive the learning rate, so I don’t think there are other issues.