I would recommend dataset as follows



in init process and compute full data length. This is required for iterator
Don’t confuse this with actual data size.
I will go ahead and store in chunk more than 64 to minimize disk/file access. e.g. read 4096 lines everytime and store in buffer. Actual datasize will be overall length and we maintain a pointer modulo to 4096 which will iterate till last chunk.


getitem: If still in current chunk, directly return what you already have and increment local pointer
If local index equals to 64 (64 8 64 = 4096) then you are at the end of buffer. Now load next 4096 lines.



Now, control 4096 (number of lines to cache) depending on how much RAM you want to spend in buffer (it’s a tradeoff)  You will have to maintain a list of file name to iterate over