Not knowing precisely the use case, I’d suggest something like: (i) transform your undesired indexes in a set, (ii) construct a list of the complement of your indexes.
 python -m timeit --setup="import torch;x = torch.rand(10 ** 5);not_idx = set(torch.randint(10 ** 5, (10 ** 5,)).tolist())" "x[[e for e in range(10 ** 5) if e not in not_idx]]"

The step e not in not_idx is very fast in python.