Thanks for the reply! I should have mentioned that the VM has 100GB of disk space in total and the dataset in question is larger than this. By a factor of 5 or so. There is the option of making a new one but I’d like that to be a last resort if possible.
However, now that you mention it, there’s no reason why I can’t just map smaller batches of the data frame it originates from and save_to_disk each local table before deleting it - then using concatenate_datasets afterwards. Very messy though, seeing as each mapping process already uses 30 workers (along with concatenate_datasets).
The error I get for writing to the mounted folder is along the lines of IOError: [Errno 5] Input/output error which is also common when writing large files to a mounted Google Drive folder when using Colab. The slow-downs refer to timeouts when reading the arrow files. The kernel won’t stop but I need to run some of the cells again which means having to continuously monitor the process (which takes a long time).
Thanks!