Hi,
I’m new using PyTorch. I am implementing and testing a new paper called Sound of Pixels 21.
In short it’s a net which works with a 2-tower stream. One tower is fed with a stack of images and the other one is fed with audio spectrograms.
I’m using a private dataset, in which each sample is a numpy binary file which contains a python dictionary with both, audio and images.
I’ve created my own dataset subclass in which I’m enlisting all the files inside a folder and using this list to call
__getitem__ function.
The matter is I have to combine features of N samples (since the net is trained by summing audio samples). As far as I understand, I have to  combine these N samples inside the __getitem__ function. However, this function is designed to deal with one sample and pass the info to the dataloader.
I considered the option of doing a post-processing of the batch doing what I need or making my own dataloader using one of the implemented samplers. Before doing that I would like to know if there is a more efficient way of dealing with this in pytorch because I have to compute fourier transforms of audio and I don’t want to bottleneck the training with a bad data preprocessing.
Thank you very much
Dataset class designed for just one sample

class BinaryData(torch.utils.data.Dataset):

    def __init__(self, root_dir,transform):
        self.input_list = []
        self.transform = transform
        for path, subdirs, files in os.walk(root_dir):
            for name in files:
                self.input_list.append(os.path.join(path, name))

    def __len__(self):
        return len(self.input_list)

    def __getitem__(self, idx):
        dic = data2dic(np.load(self.input_list[idx]))
        audio = dic['audio']
        audio = Sound_standarize(audio)
        frames = dic['frames']
        size = np.shape(frames)
        images = []
        for i in range(size[3]):
            images.append(self.transform(Image.fromarray(frames[:,:,:,i])))
        frames = torch.stack(images)
        return audio,frames