ChrisLiu2:

I found that loss.backward() calls are taking the most time to run. Do you have any quick guesses where it might be wrong? I tried detaching the learned state and the threshold (a learned variable defined in the model) in each epoch, but doesn’t seem to help. Should I try detaching all other variables in the model as well?


You could try to check all tensors for a valid .grad_fn and detach every tensor, which is not needed anymore. If the backward pass yields the slowdown this might indeed point towards a growing computation graph.
Do you see an increase in memory usage while you are training the model?
Just a wild guess, but could you also detach the loss when you are passing it to the summary_writer? I don’t know, how variables are handled by it.