Your code looks alright. Especially since you used the folder and class names directly, which doesn’t leave much room for errors.
If you don’t remap the validation targets, is the model worse than a random prediction?
This is indeed the first time I’m seeing such an issue and usually there is something wrong with the data.
Also, could you load manually some validation data, check the class manually (I assume you can classify the images), and check the predictions again? While the splitting seems to be alright, there might still be some mismatches in the overall data loading pipeline. E.g. are you using a custom sampler or collate_fn?