Yes exactly in this case the dataset is loaded in memory.
The issue might come from multiprocessing. There is a limitation in pyarrow that makes it impossible to send more that 4GB of data from one process to the other. To workaround that in datasets, it writes the data temporarily on disk and the other process reads the data from disk to put it in RAM. The temporary files are removed after map concatenates all the results from all the processes, but it looks like your disk gets full before that.
Let me know if you still have issues with this behavior so we can take a look