I just fixed this actually. I was originally instantiating my dataset class only once and subsetting my train and test datasets from there. Instantiating my dataset class twice (one object for my training set and one for my test set) fixed my issue. I guess some part of evaluation was altering the dataset object therefore changing my training dataset.