Hi There,
I was facing an practical issue for using pytorch to training over text data. my training data was about 100Gb and cannot be load into memories. I couldn’t create an custom dataset over my training data because of the implementation of getitem and len could be very time intense.
I was wondering if there any practical solution for loading HUGE text data by using pytorch ?