Your approach sounds alright, if you want to use the ImageFolder datasets.
Alternatively, you could write a custom Dataset and return the corresponding targets for both datasets, which wouldn’t rely on creating empty folders.