I have found the issue, and it is a very subtle one:
When returning a scheduler to Lightning using the dict format like I do in this line:
        return {'optimizer': optimizer, 'scheduler': scheduler}

the keyword for the scheduler needs to be lr_scheduler, otherwise it will not be picked up and the learning rate will stay high.