To understand the issue completely: you are calling model.eval() and check the accuracy on the validation set. The validation set indices (passed to Subset) are definitely the same, but the accuracy changes for sequential runs?