I am going to be working with many files, that all can’t fit into memory at the same time.
Since the data is text that I would like to be loaded quickly, I was hoping to use ether pytorch.load/pickle files, or pandas/parquet files.
However, it looks like ConcatDatastet can’t work for my case, since there’s no way to load specific lines only, like with csv files and linecache.getline, so I’ll have to load all the files at once.
So now I’m thinking I’ll just load a fraction of the files, as much as will fit in memory, and do a ConcatDatastet for those files. Perhaps using pytorch multiprocessing to load those several files at once. And then use ConcatDatastet of that chunk.
I’ll probably have to code additional shuffling code for shuffling among the chunks.
Does this strategy sound like the best approach? Or are there some additional Pytorch tools that can help me around this?