For anyone using, I believe this is a more elegant implementation
torch.rfft(imgs, signal_ndim=2, normalized=True)

As torch.rfft() should be running from the last dimension, meaning that anything before imgs[2] will be considered as a batch size. Hence [52, 3] is treated as a whole and the fft is performed only at [128, 128].