What initially comes to my mind is that you’re still holding on to your training set data in memory somewhere, so when you go to use your validation set you increase the total memory (as you’re adding new data but haven’t cleared the training data that you’re no longer using)!
(Might be wrong, so best to get a dev’s opinion but I have a feeling it could be something like this!)