I do some hacks to get this working, see https://github.com/jramapuram/datasets/blob/master/loader.py#L178 18 and an older example using a different submodule commit: https://github.com/jramapuram/LifelongVAE_pytorch/blob/master/main.py#L317 6
The high level idea is you want to implement a sampler that filters your labels.