Apologies for not coming back to this thread. If I recall correctly, in the end, the problem was not with rfft() but with my multiplication operation.
I analyzed my tensors and found that after this operation, NaNs were present. This was because I was trying to multiply by zero. To get around this, I added a small “eps”/epsilon to the data before multiplying it, i.e. eps = 1e-15.
I have also since moved onto the most recent version of PyTorch which deprecated the rfft function for fft.fftn().