I’ve been using PyTorch to estimate discrete choice models with large choice sets. The data are “sessions” where each session consists of ~10k rows corresponding to the choice set.
Currently, I’m storing the data in an h5 indexed by a unique session key, then reading in sessions in batches with a DataLoader. This works, but is very slow (the IO time is ~10x the computation time). On disk, the h5 file is about 500gb.
Is there a better option? I’ve seen WebDataset, which I’m planning to try next barring any better suggestions!