Dear folks,
hello, I am a quite newbie in deep learning with PyTorch…
I am trying to build an image classification model using PyTorch,
but in the process of pre-processing dataset, I’ve stuck with a problem.
Problem)


I want to read a bunch of .npy files in a directory (e.g. ‘/training’)


One of the .npy files consists of tensors with shape of (N, C, H, W), i.e. mini-batch of images
(e.g. file1: (1592, 3, 224, 224), file2: (1100, 3, 224, 224), …)


The size of mini-batch is different from each other (1592, 1100, 683, …)


Questions)


AFAIK, DatasetFolder or ImageFolder module of torchvision.datasets are used for the directory having a list of single image files. (e.g. 1592 independent .npy files, of which shape is (1, 3, 224, 224))


In my case, is there any recommended way to deal with the situation?
(I finally want to use DataLoader after constructing a (custom) DatasetFolder/ImageFolder module)


I want to avoid the situation of reading each mini-batch .npy file, split them by a sample (1, C, H, W), and save them again in a separate file…


Thank you, in advance.
Have a good day!