I would recommend to create a custom Dataset and maybe just reuse some parts of the ImageFolder dataset from here 59.
Changing this attribute after the ImageFolder was created seems to be wrong, since class_to_idx will be used to create the dataset as seen here 40.