OK, I see. Maybe your actual workload is just small compared to data moving (idx = torch.randperm(n)[:1000].cuda()) and expanding (x = m.unsqueeze(1).expand(n, n, d)).
Have you tried to run it on the CPU? If the workload is really small (I cannot estimate it without shapes), probably the data transfer overhead is bigger than the performance advantage on the GPU.