Is this behavior reproducible, i.e. are you seeing a better performance using the torchvision dataset if you rerun the code multiple times with a different seed?
If so, are you using the “exact” same dataset (same number of images are might some be missing.corrupt), the same transformation, and the same model?