In a lot of our examples, you're going to see DataCollators popping up over and over. They're used in both PyTorch and TensorFlow workflows, and maybe even in JAX, but no-one really knows what's happening in JAX. We have a research team working on that, so maybe they'll tell us soon. But what are data collators? Data collators collate data. More specifically, they put together a list of samples into a single training minibatch. For some tasks, the data collator can be very straightforward. For example, when you're doing sequence classification, all you really need from your data collator is that it pads your samples to the same length and concatenates them into a single Tensor. But for other workflows, data collators can be more complex, as they handle some of the preprocessing needed for that particular task. For PyTorch users, you usually pass the DataCollator to your Trainer object. In TensorFlow, the easiest way to use a DataCollator is to pass it to the to_tf_dataset method of your dataset. You'll see these approaches used in the examples and notebooks throughout this course. In both cases, you end up with an iterable that's going to output collated batches, ready for training. Note that all of our collators take a return_tensors argument - you can set this to "pt" to get PyTorch Tensors, "tf" to get TensorFlow Tensors, or "np" to get Numpy arrays. For backward compatibility reasons, the default value is "pt", so PyTorch users don't even have to set this argument most of the time, and so are often totally unaware that this option exists. This is a valuable lesson about how the beneficiaries of privilege are often the most blind to its existence. So now let's see some specific DataCollators in action, though remember that if none of them do what you need, you can always write your own! First, we'll see the "basic" data collators. These are DefaultDataCollator and DataCollatorWithPadding. These are the ones you should use if your labels are straightforward and your data doesn't need any special processing before being ready for training. Most sequence classification tasks, for example, would use one of these data collators. Remember that because different models have different padding tokens, DataCollatorWithPadding will need your model's Tokenizer so it knows how to pad sequences properly! So how do you choose one of these? Simple: As you can see here, if you have variable sequence lengths then you should use DataCollatorWithPadding, which will pad all your sequences to the same length. If you're sure all your sequences are the same length then you can use the even simpler DefaultDataCollator, but it'll give you an error if that assumption is wrong! Moving on, though, many of the other data collators are often designed to handle one specific task, and that's the case with DataCollatorForTokenClassification and DataCollatorForSeqToSeq. These tasks need special collators because the labels are variable in length. In token classification there's one label for each token, and that means the length of the labels can be variable, while in SeqToSeq the labels are also a sequence of tokens that can have variable length. In both of these cases, we handle that by padding the labels too, as you can see here. Inputs and the labels will need to be padded if we want to join samples of variable length into the same minibatch, and that's exactly what the data collators will do. The final data collator I want to show you is the DataCollatorForLanguageModeling. It's very important, firstly because language models are so foundational to everything we do in NLP, and secondly because it has two modes that do two very different things. You choose which mode you want with the mlm argument - set it to True for masked language modeling, and False for causal language modeling. Collating data for causal language modeling is actually quite straightforward - the model is just making predictions for what token comes next, so your labels are more or less just a copy of your inputs, and the collator handles that and ensures your inputs and labels are padded correctly. When you set mlm to True, though, you get quite different behaviour! That's because masked language modeling requires the labels to be, well... masked. So what does that look like? Recall that in masked language modeling, the model is not predicting "the next word"; instead we randomly mask out multiple tokens and the model makes predictions for all of them at once. The process of random masking is surprisingly complex, though - that's because if we follow the protocol from the original BERT paper, we need to replace some tokens with a masking token, other tokens with a random token and then keep a third set of tokens unchanged. This isn't the lecture to go into *why* we do that - you should check out the original BERT paper if you're curious. The main thing to know here is that it can be a real pain to implement yourself, but DataCollatorForLanguageModeling will do it for you. And that's it! That covers the most commonly used data collators and the tasks they're used for.