Thanks for the reply!
I want to pay the price of loading the file once in memory and using the offsets (which I already have done), but the use case requires many of these large files, ie. the dataset could consist of 100 files of 1GB each. So what I imagined was always loading the next 1GB while the current 1GB is being served.
I don’t mind the penalty of waiting to load the first file, as long as subsequent loads do not affect training times - I want to make sure the GPU is utilised fully.
I will have a look at the bg_iterator, which may work!
edit: How would I have found audio/utils without going through the source/getting this recommendation?
Is there perhaps some documentation I missed?
edit 2: bg_iterator did it! Fairly simple too, just had to create a generator for all big files and specify the generator with maxsize=1 so it will buffer the next item. All my other logic still works, generating the correct frames/batches. Thanks!