The error is because the weight of the linear layer has changed (through optimizer.step).
One might add that here, you have the gradient computation for addmm (which powers linear) pretend it would also want to compute the input derivative for which it would need the weight which, in this trivial use, is not actually the case. If you have a multiple layers, all but the first need the input gradient and you cannot actually change the weight and then compute the (correct) backward, not even with retain_graph.
Best regards
Thomas