Phenomenal update. You anticipated the problems I had and helped me.
I learned much from your code.
I noticed that this also works:
    with torch.no_grad():
          a.sub_(lr * a.grad)

once replaced this code:
a = (a.data - lr * a.grad).requires_grad_(True)