To clarify the question, the speed starts picking up steadily in epoch 2 but the impact doesn’t start to be visible until later in the epoch.
I found that loss.backward() calls are taking the most time to run. Do you have any quick guesses where it might be wrong? I tried detaching the learned state and the threshold (a learned variable defined in the model) in each epoch, but doesn’t seem to help. Should I try detaching all other variables in the model as well?