The main difference between the ImageFolder and your approach would be the lazy vs. eager loading.
While the ImageFolder dataset loads each sample lazily in the __getitem__ method, it seems you are preloading all samples in the __init__ method.
You could try to refactor your code such that you are only creating all valid paths (and targets, if it’s cheap to calculate them) in the __init__ and execute the loading using these paths in __getitem__.
Let me know, if this would help.