Thanks for your reply. Sorry I did not.
Here are the results of the run after adding torch.cuda.synchronize():

Screenshot_20181226_151512.png326×707 27.1 KB

Now loss.backward() is the slowest step. I can understand forward calculation and loss.backward having similar time complexity, but what’s the reason for optimize step being ~30x faster? Are all the weight updates doing simultaneously, while loss.backward is sequential?