Okay so I left it at the default LR that Adam provides. I was earlier using Adadelta and switched to Adam because it adds an additional momemtum term and is more stable.
This is the optimizer snippet. The whole point is to skip using a scheduler.
optimizer = Adam(parameters, weight_decay=opt.weight_decay)