It would require rewriting the whole data loading part. It would need to go over all of it to gather all the tokens, and then lazily load only the batches you request. You’d need to add a proper torch.utils.Dataset subclass that does it.