Thanks. Interesting ideas.
Let’s say I can look at the folder, read some metadata about each file and calculate a rough upper bound K of the number of segments in all files combined.
The problem is, that given an index in the range [0,K-1], it’s not possible to know which file to look at to find that index. I could again do some estimations to just pick a file based on the index in a deterministic way, load it, process it and use the % to get a segment inside it even if it actually doesn’t have enough segments.
However this would cause lots of re-loading of the files and re-running the signal processing algorithms that split it… Each file would be processed many times. It would cause a non-negligible overhead. Probably I could fix it with some caching, but even that might still be slow since I have a lot of data. The reason I was looking for some DataLoader trick here was to do all the loading and processing lazily, on demand…
Thanks again.