I found the issue. I had to rename the CUDA kernel function to RgbGradientForwardKernel since I was using RgbGradientForward for the CPU version. This shouldn’t be an issue though since torchvision uses the same function names for the CPU and GPU kernel.
Maybe a bug somewhere?