Same here, seems to be related to randperm()
Reproducible code:
>>> import torch
>>> device = torch.device("cuda:0")
>>> torch.randperm(29999, device=device)
tensor([13324, 19251, 23333,  ..., 18540, 14502, 26766], device='cuda:0')
>>> torch.randperm(30000, device=device)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDevice: invalid device ordinal

pytorch version: 1.8.0+cu111