It feels odd that I have to iterate over the past steps while using a map-style dataset. Couldn’t the batch sampler just fast-forward to the desired step?
I am loading a huge dataset for language modelling. The operation is IO-bounded and it is taking a few hours to go over the steps.