Hi everyone,
I have a specific thing I want to achieve, and was wondering if it’s already possible using the current Dataset/DataLoader implementation. I saw a PR for a ChunkDataset API that may serve my needs, but it isn’t there yet.
Data: Audio, with lots of small (1-10s) sound files.
I want to process this audio in terms of frames, but also incorporate a hop parameter (ie. take the first 1024 samples, then the next frame will start at 256 instead of 1024)
What I want to do is concatenate all the short audio examples into long .wav files, of which two can fit into memory. I wrote code to index individual sounds and their respective frames and it works well.
The idea is to serve frames from one (long, e.g. 1GB) .wav file, and have another one loaded in the background. When all frames from the first file have been served, I replace the “current” file with the one that was loaded in the background, and load a new file.
Everything works, except for the fact that the IO on loading a new file will block the getitem call, interrupting training. I was thinking of some async io structure, but lack some experience there in getting it to interop with the Dataset/Loader classes.
How can I do a non-blocking IO call to replace the current “buffered” file while keep serving frames?