You could get all occurrences of the current number you would like to draw and randomly permute these indices:
x = torch.tensor([2, 3, 2, 1, 2, 3, 2])
to_sample = torch.as_tensor(2)
idx_all = (x == to_sample).nonzero()
idx = idx_all[torch.randperm(len(idx))[0]]