Could you post the shapes of both tensors, so that we could profile it?
Using these shapes:
x = torch.randn(100, 100, 100)
idx = torch.randint(0, x.nelement(), (1000,))

yield approx. the same timing for CPU and CUDA runs in the latest nightly binary.