Would it be possible to compute the invalid indices before and pass them as a list to your Dataset?
If so, you could just add an offset to an index vector using the invalid indices.
E.g. if the invalid indices are at 2 and 4, the indices list could be:
self.indices = [0, 1, 3, 5, ...]
index = self.indices[index]
# use index as before in __getitem__

Alternatively, you could also pass these indices to a Subset, which will apply the same logic.
If you cannot compute these indices beforehand, you could return a pre-defined invalid value, check for it in the training loop, and skip this sample.