nlp 86 is what you are looking for . It’s the new datasets library from HF which allows you to memory map huge datasets and load them lazily. It also caches all the pre-processing you do, so the next time you load the data with same processing it just uses the cache instead of recomputing.
Just to see how efficient this is, it can load 17 GB+ English wikipedia dataset in just 9 MB of RAM. 

  
      twitter.com
  
  
    

Thomas Wolf (Thom_Wolf) 57

 There is a bit of magic in the new 🤗nlp library besides giving dead-simple access to 120+ datasets🧙‍♂️

We've tested it with @qlhoest and loading a 17GB+ dataset like English Wikipedia only takes... 9MB in RAM🐣

And you can iterate over the data at 2-3 Gbit/s🚀

Try it yourself👇


  5:55 AM - 15 Jun 2020
    
      
        
       1.1K
    
    
      
        
       238