Hey this isn’t working for me. Anytime I try to load in the Wiki it takes 17 gb.
I tried a few iterations of moving split or a different cofig no luck its still too large. It’s strange because with bookcorpus it works fine.
code:
from nlp import load_dataset
mem_before = psutil.Process(os.getpid()).memory_info().rss >> 20
dataset_wiki = load_dataset(“wikipedia”, “20200501.en”)[“train”]
mem_after = psutil.Process(os.getpid()).memory_info().rss >> 20
print(f"RAM memory used: {(mem_after - mem_before)} MB")