Memory mapping and streaming. In this video we'll take a look at two core features of the Datasets library that allow you to load and process huge datasets without blowing up your laptop's CPU. 

Nowadays it is not uncommon to find yourself working with multi-GB sized datasets, especially if you’re planning to pretrain a transformer like BERT or GPT-2 from scratch. In these cases, even *loading* the data can be a challenge. For example, the C4 corpus used to
pretrain T5 consists of over 2 terabytes of data!

To handle these large datasets, the Datasets library is built on two core features: the Apache Arrow format and a streaming API. Arrow is designed for high-performance data processing and represents each table-like dataset with an in-memory columnar format. As you can see in this example, columnar formats group the elements of a table in consecutive blocks of RAM and this unlocks fast access and processing. Arrow is great at processing data at any scale, but some datasets are so large that you can't even fit them on your hard disk. For these cases, the Datasets library provides a streaming API that allows you to progressively download the raw data one element at a time. The result is a special object called an IterableDataset that we'll see in more detail soon. Let's start by looking at why Arrow is so powerful. The first feature is that it treat every dataset as a memory-mapped file. Memory mapping is a mechanism that maps a portion of a file or an entire file on disk to a chunk of virtual memory. This allows applications to access can access segments in an extremely large file without having to read the entire file into memory first. Another cool feature of Arrow's memory mapping capability is that it allows multiple processes to work with the same large dataset without moving it or copying it in any way. This "zero-copy" feature of Arrow makes it extremely fast for iterating over a dataset. In this example you can see that we iterate over 15 million rows in about a minute using a standard laptop - that's not too bad at all! Let's now take a look at how we can stream a large dataset. The only change you need to make is to set the streaming=True argument in the load_dataset() function. This will return a special IterableDataset object, which is a bit different to the Dataset objects we've seen in other videos. This object is an iterable, which means we can't index it to access elements, but instead iterate on it using the iter and next methods. This will download and access a single example from the dataset, which  means you can progressively iterate through a huge dataset without having to download it first. Tokenizing text with the map() method also works in a similar way. We first stream the dataset and then apply the map() method with the tokenizer. To get the first tokenized example we apply iter and next. The main difference with an IterableDataset is that instead of using the select() method to return example, we use the take() and skip() methods because we can't index into the dataset. The take() method returns the first N examples in the dataset, while skip() skips the first N and returns the rest. You can see examples of both in action here, where we create a validation set from the first 1000 examples and then skip those to create the training set.