I think what wrong is I should use
trainer.add_event_handler(Events.EPOCH_STARTED, lr_scheduler)
instead of
trainer.add_event_handler(Events.ITERATION_STARTED, lr_scheduler)
Because in 1 epoch there are so many ITERATION (batch size of 1 for example) that make learning rate drop so fast with my step_size=100