Solved by sparseinference in post #3 
              
              
                OK thanks. 
I see the idea - generate a random permutation of the indexes ahead of time, then just pass that to a SequentialSampler. 
Since the order of indexes is fixed I can just extract batch_size indexes at a time from that list.