The scheduler is ReduceLROnPlateau 2, it is used to update the learning rate based on a metric (in my case validation accuracy).
Because val_acc is not a model parameter, I would assume it to be different on every process (because every process has its own mini-batch). Therefore, I need to synchronise it so every process changes the learning rate at the same time.
When I find some time, I will try and work out a minimum example.