retain_graph basically keeps the forward(or backward) path after the backward call, however, torch will use buffers to store temporary data in some forward calls, and these data are utilized in their respective backward pass, their version will be stored so that any update on these buffers will be noticed by torch and an error will be thrown to notify users of this. This mechanism also applies to parameters. Parameters and Buffers are basically tensors with special wraps.
when you call “step”, the optimizer will perform parameter updates, thus their version number will be increased, therefore you got this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 1]], which is output 0 of TBackward, is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

“retain_graph” is mainly used to perform multiple backward pass on networks with multiple outputs (i.e.: multi-head output) before a final call to step().
Detach stops gradient from flowing through that path, so you will not get those errors.