mzimmer:

What could be the problem here? Do I accumulate history? I am unsure why this would get slower and slower over time, in the beginning it is fairly fast but the runtime is monotonically increasing.


That could be the case. Do you also see an increased memory usage?
If so, could you check, if your momentum buffer has a valid .grad_fn after a couple of iterations?
If so, it could store the computation graph. However, since you have already used the no_grad context manager this should not be the case.
How did you isolate that the momentum calculation is responsible for the slowdown?