Could you try to change the learning rate using one optimizer and changing its param_groups[0]['lr'] or alternatively torch.optim.lr_scheduler 3?
Currently you are re-initializing it in each epoch, thus your optimizer will lose its accumulated momentum etc.
Maybe it won’t have a big effect, but it’s worth a first shot!