I’m not sure I understand the data structure completely, but would recommend to use e.g. pandas and to make sure you can load all images with their label.
Once you’ve created the mapping between the image paths, their metadata, and targets, I would create a csv (or any other format you like), such that indexing this file will give you everything you need to load the data sample and create or load the target.
If that’s done, you could load this new csv in Dataset.__init__, and lazily load each sample in its __getitem__.