From what I understand is that when you do regular ML, you do something like this after unrolling the training loop:
# Iteration 1
loss = f(w0)
g = grad(loss, w0)
w1 = w0 - lr * g
# Iteration 2
loss = f(w1)
g = grad(loss, w1)
w2 = w1 - lr * g
# Iteration 3
loss = f(w2)
g = grad(loss, w2)
w3 = w2 - lr * g

So you only ever backprop to the latest w value.
In this case, you can actually ask for gradients for older “versions” of w like for the loss at iteration 3, you can do grad(loss, w0). And in that case, you can see that the gradient update by SGD is actually part of the differentiable operations that the autograd need to differentiate through.