ImageFolder will build classes based on names of the folders you have under the root, so if the underlying folders and files are exactly the same you can expect a one-to-one match between datasets made with test and groundtruth root folders. You can also check .imgs attribute of each dataset which will contain full path and label for each file and see if the one-to-one match is preserved. So if you iterate over each dateset separately with shuffle=False, you should get what you need.
However, I’m not sure how these two datasets need to be iterated in your testing pipeline.