If you are fine with some repeated images, you could check in the __getitem__ method of your Dataset, if the current file exists, and if not just sample a new random index.
This approach will repeat as many valid images as there are missing ones.
Would that work for you?