ImageFolder already creates the data-target mapping internally and loads each sample lazily in its __getitem__.
I think the cleanest approach would be to write a custom Dataset by reusing parts of DatasetFolder 160 and add your target tensor manually to your custom class.