ImageFolder will create the labels based on the passed folders, so the labels should also mismatch in the original Dataset before the splitting. Could you check that and see if some images might be stored in the wrong folder?