Hi,
I have implemented a custom pytorch Dataset.
The dataset is initialized with a folder. Each file in the folder contains a long signal that I do some signal processing on to split it into M segments.
Right now for each of the files, the __getitem__() method generates an NxM matrix where N is a known number of features, and M is the number of extracted segments from that file. I have no way of knowing M in advance, i.e. not until I analyze the entire signal and it’s different per file.
The __len__() method currently returns the number of files in the folder the Dataset was initialized with.
What I actually want is to work with these segments as individual samples for my model, not the entire signal.
In other words, I want batch sizes of say 128 segments (so a batch would be of shape 128xN).
Ideally, I would give my dataset to some customized DataLoader and it would create these batches on the fly by loading one file (with M>128 segments) and taking random batches of 128 segments. When there aren’t enough segments left in the loaded file to fill a batch, a new file should be loaded (in random order) and so on.
I tried looking into the Sampler and BatchSampler classes and also at the custom collate_fn that can be provided to a DataLoader, but I haven’t found a way to achieve this… Everything seems to expect the number of samples to be known in advance.
So, is there some trick I can use?
Or do I need to resort to simply saving each segment to a separate file and loading them using the standard 1file=1sample based approach?
Thanks.