In your case the dataset can be fit into RAM and you are taking advantage of that. The initial slowdown that you are facing is a one time slowdown as you don’t hve to load the images again (the biggest bottleneck in all cases).
The choice to load the entire dataset on RAM depends on what is the bottleneck CPU or GPU. If GPU is the bottleneck than no need to load images in RAM.