I have been trying to use this method to go through my dataset of around one thousand csv files, each containing about 300k lines (one training sample per line).
This method leads to a slow buildup of CPU memory as I iterate through the batches until it crashes.
I have tried putting in linecache.clearcache() at the end of each batch but it isnt clearing up the memory.