I assume you’ll be applying tokenizers or other processing to that large dataset using .map()? In that case, what was your batch_size and writer_batch_size? I found it very difficult to find the right numbers that do not consume all my RAM and was fast enough (my dataset is >100GB).