ImageFolder will lazily load and process each sample to save memory.
Based on your code snippet you are preloading the entire dataset, which would of course cost a startup time, but could be faster in each iteration. This approach is often not possible, as datasets are often too large to fit into the RAM.
Also, if you store the data on a local SSD and use multiple workers in the DataLoader you might also be able to hide the data loading latency (each sample will be loaded and processed in the background, while the GPU is busy with the training).