Just to understand the use case correctly:
you are not using the torchvision.dataset.CIFAR10 data, but downloaded the cifar.python dataset from another source.
After changing the path, the accuracy of your model is constantly lower using exactly the same setup and different seeds?