If you check the code in xla/xla_model.py at 815197139b94e5655ed6b347f48864e73dc73011 · pytorch/xla · GitHub 9, you will find that optimizer_step will call mark_step after the optimizer.step. This means if you call xm.optimizer_step at the step end, you don’t need another call to xm.mark_step().