@lhoestq on a similar note, then I have been preparing a large corpus of medical data for T5 denoising fine-tuning with adding all the random sentinel tokens and spans to the input and target columns.
I find that on trying to load larger files of the data using the run_summarisation.py example I get a memory overrun. So first question - is loading from a dataset much more efficient with memory usage? So instead of loading the whole file to hold the whole dataset in memory, it allows access but brings spans of data into memory only when needed? (as is suggested on this page?)
Second question, if the loading as a dataset is what I need to do to minimise memory usage, what is the best/easiest route to get this done? I have the csv file with the two columns for input and target (and other metadata in other columns such as the original text). So this is a very simple file. Just to be clear, there isn’t an available web resource to load the original data from, it is data I have generated and have myself on my PC. Can I simple load the csv data in a python script into a dataset using ‘load_dataset()’ and then just use your new `my_dataset.push_to_hub()’ method which would then host/store the data on the hub for retrieval by me/others that wanted to try it?
And thanks in advance for any help, new to this so pointers would be very much appreciated!