While this approach can be used to create a Dataset with an unknown size (which is good!), it’s still not “lazy enough” just like that because when combined with Samplers and DataLoader I need to make sure to only generate indices of segments within the current file the DataSet has loaded and split.
It now seems possible to perhaps do this by coupling the Dataset and the Sampler, but I think for simplicity i’ll just pre-process all the files and split them in advance.
Thanks for your help