I guess what you need is a sampler that can sort the sparse dataset by batch dim and efficiently return mini-batch according to batch dim. But there is a harder problem, that is to have batch op supports (e.g. bmm). I will take a look at this.