So the Fourier transform works on intensities and an RGB image won’t have a defined intensity given a pixel. You could try and splitting the image in the rgb channels and then running torch.fft on them?
image = torch.randn([52,3,128,128])
r =torch.rfft(image[:,0:1].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])
g =torch.rfft(image[:,1:2].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])
b =torch.rfft(image[:,2:3].permute([0,2,3,1]), signal_ndim=1).permute([0,3,1,2,4])

image_fft = torch.cat([r,g,b], dim=1)
image_fft.shape
>>torch.Size([52, 3, 128, 128, 2])