I have asked something similar to your question and I find out writing a custom dataset based on Dataset abstract class is the best idea and fastest!
This is a sample as an answer to my question:

  
    
    
    Custom Dataset with some preprocessing vision
  
  
    When should I do this based on  Dataset  class? Should I do my processing step in  __getitem__() ? If yes, would it be parallel and fast? 

Yes. the __getitem__ calls are the ones run in parallel. Here’s some example: 
class CelebaDataset(Dataset):
    """Custom Dataset for loading CelebA face images"""

    def __init__(self, txt_path, img_dir, transform=None):
    
        df = pd.read_csv(txt_path, sep=" ", index_col=0)
        self.img_dir = img_dir
        self.txt_path = txt_path
        …
  


And if you are in full implemented of my work, here is the link to my Github 71 of a simple CNN for edge detection on some custom dataset with custom loss and custom transforms.
And my dataset structure is exactly same as you. So I hope it helps you.
Thanks for any star