BramVanroy:

Let’s say that I do not need attention masks and type_token_ids. If I use set_format that information will still be present in the dataset somewhere, but it just won’t be returned. That would mean that the on-disk dataset also contains those columns, even if they might not be necessary. Isn’t it better to remove those completely, then?


Well for most of these operations we go with the fastest option, i.e. not updating the underlying storage but only the in-memory representation.
If you need to be vary careful about HD memory, you can map the dataset again to update the HD repr or you can use the load_dataset(keep_in_memory=True) option to keep the dataset always in RAM (see here 2).



 BramVanroy:

I also found that the caching does not seem to work as I expected. The “mapping” happens at each new run of the script. For now I resorted to my earlier suggestion of saving the dataset after all operations, and checking beforehand whether the processed directory exists. If so, just load it. If not, process and save.


Ok, this should not happen if you can share a snippet to reproduce it we can investigate why caching is not working.