You can get both behaviors in torchtext (I believe it’s shuffle=True) but also I think it’s still unclear which is better for a given task – I’ve seen significant effects in both directions but haven’t carried out any sort of systematic analysis.