No, the Python process just dies. I have 120 GB of RAM on this machine and 500 GB of disk space. If I could shard this large dataset before saving that would work too. I would just save these shards one by one. But by doing sharding I get copies of of dataset.arrow in each dir and it quickly adds up.