Hi,
I’m building a image classifier using trail camera data. I’m trying to create a custom dataset by loading images from a directory with many sub folders organized by capture location and date and not by classes.
for example,.
/data
    /COL
            /COLE
                        /CO_LE_05_190211
                                     /CLE-COLE05_00001_1-6-2019.jpg
                                     /CLE-COLE05_00001_1-25-2019.jpg
                                     .
                                     .

                        /CO_LE_05_190227
                                     /CLE-COLE05_00002_2-5-2019.jpg
                                     /CLE-COLE05_00002_2-6-2019.jpg
                                     .
                            
                        /CO_LE_05_190404
                                     /CLE-COLE05_00003_4-7-2019.jpg
                                     /CLE-COLE05_00003_4-13-2019.jpg
                                     .   
                                     .

I have a csv file with image paths and their respective label so I could take a subset of the dataset and structure sub-folders based on each class, however, doing so would limit accuracy, inferencing, etc.
I have over 3 million images and have found that re-organizing images by class would not be possible considering my memory limitations and time needed to do so.
I was wondering if anyone could suggest a custom dataset loader that would allow me to reference the csv with image path and label info, while retaining the current folder structure?
any help is greatly appreciated.
cheers,
mkutu