I want to validate my model using only a fixed subset of the whole dataset  without affecting the training loop. I want my code to look something like

for epoch in range(num_epochs):
for batch in train_dataloader:
     train_step()  #  loss, optimize, etc ..

torch.manual_seed(0) # This fixes the training data too!

for repetition in range(segments_per_speaker) # sampling multiple segments per speaker
    for batch in valid_dataloader:
         valid_step() # choose best model


Placing torch.manual_seed(0) after the training loop somehow fixes the training data in different epochs. What am I missing? Any recommendations to solve my issue?
Thanks in advance!