Fixed it. Key takeaways in fixing autograd problems like these:

Use torch.autograd.set_detect_anomaly(True) to isolate the line with the problem.
Call .clone() on the variables on the right hand side of the update.

had to replace the line:
dx = 2.0 * torch.stack([torch.rfft(x_r[0], 2), torch.rfft(x_r[1], 2)], dim=0)

with:
dx = 2.0 * torch.stack([torch.rfft(x_r[0].clone(), 2), torch.rfft(x_r[1].clone(), 2)], dim=0)

Found the following threads / blogs useful:


https://www.yongfengli.tk/2018/04/13/inplace-operation-in-pytorch.html


Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation


RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead