Your approach should work.
Could you check, if you have any additional folders (maybe hidden) in your root folder?
I created fake folders like:
root
    - folder0
        - image0.jpg
    - folder1
        - image0.jpg
    - folder2
        -image0.jpg

These are the classes:
dataset = datasets.ImageFolder(root='root')

print(dataset.classes)
> ['folder0', 'folder1', 'folder2']
print(dataset.class_to_idx)
> {'folder2': 2, 'folder1': 1, 'folder0': 0}

Your targets shouldn’t be images, but long integers. Are you sure you are subtracting from the right value?