Hi !

I find that on trying to load larger files of the data using the run_summarisation.py example I get a memory overrun. So first question - is loading from a dataset much more efficient with memory usage?

In summarization.py the dataset is loaded using something similar to load_dataset("csv", data_files=["path/to/my/data.csv"]). I converts your CSV into an Arrow file which can be loaded without filling your RAM. Therefore I’m not sure how it could cause memory issues in your case, can you provide more details about what happened ?

Second question, if the loading as a dataset is what I need to do to minimise memory usage, what is the best/easiest route to get this done?

The data_files argument to pass to load_dataset works for both local and remote data. In your case, since your CSV is local you can just do
from datasets import load_dataset

my_dataset = load_dataset("csv", data_files=["local/path/to/my/data.csv"])

Ultimately you will be able to do my_dataset. push_to_hub() to share it with other people.
As I mentioned in my previous message, we also plan to letting users upload their data files (csv, json etc.) directly in a dataset repository on the Hugging Face Hub and allow them to load it with load_dataset("username/dataset_name"). I expect the PR to get merged next week and a new release of datasets to happen right after