Thank you @the-dharma-bum for your reply.
It seems that a custom sampler is exactly what I require in this case. This will allow me to check that all samples in the [idx] folder have been processed correctly before sending to the dataloader.
This is a great solution for me as the dataset is being processed while I am experimenting with some model architectures. Identifying incompete samples during init wouldn’t be possible due to the dataset size.
Thank you for your help!