Hey, I figured this out. Seems that even though I am zipping my weights with the samples while shuffling at the epoch end, the weights with the sampler don’t get shuffled. The weighted random sampler just allocated the probabilities to all the samples once and has no effect of my custom shuffling.
My accuracies are still 1-2% off as compared to my Keras code and I think the reason is ‘no-shuffle’ after each epoch. If I set shuffle=True in the data loader, it throws the error that sampler and shuffle are mutually exclusive options.
Is there a way to shuffle properly with a weighted random sampler?