There is quite a lot of discussion on WeightedRandomSampler here, but to your point – you should use the random sampler only on the training data. While training, you are updating each parameter to minimize the loss between predicted y and actual y. And you are trying to do this in a generalized manner so we are learning our weights training, and then seeing how they perform on our validation set (which should just be shuffled).
If you want to penalize the model for making mistakes in finding your imbalanced minority class, you can also play with adding weights to nn.BCEWithLogitsLoss().