The relation between backward and optimize seems unrealistic: Are you on cuda? Did you use torch.cuda.synchronize() each time you start or stop the timer?
I have to ask because it’s a very common cause of invalid performance analyses.
Best regards
Thomas