Hi @ThaiThien
here 58 is an example of using another LR scheduler (PiecewiseLinear), but the idea is the same. In the example, a helper function is used to attach scheduler to the trainer and internally it does the following : https://github.com/pytorch/ignite/blob/master/ignite/contrib/engines/common.py#L113 56
So, in your code, probably you need to do the following:
# Should be attached in any case: loading or not loading
# else:  
#        print("do not load, keep training")
trainer.add_event_handler(Events.ITERATION_STARTED, lr_scheduler)

HTH