I have tried your suggestion, but unfortunately, it did not solve the issue. Even with random tensors on the GPU it still switches to the CPU for the the testing.
I have profiled the code and it seems the bottleneck is for <method 'sort' of 'numpy.ndarray' objects>