If the dataset is static and you are using the same labels every time, you could create a new root folder (for the ImageFolder) and create symbolic links of the desired class folders in this new root.
The ImageFolder will then use only these classes and will assign them the right class index.