---
local: data-collator
sections:
- local: transformers.default_data_collator
  title: Default data collator
- local: transformers.DefaultDataCollator
  title: DefaultDataCollator
- local: transformers.DataCollatorWithPadding
  title: DataCollatorWithPadding
- local: transformers.DataCollatorForTokenClassification
  title: DataCollatorForTokenClassification
- local: transformers.DataCollatorForSeq2Seq
  title: DataCollatorForSeq2Seq
- local: transformers.DataCollatorForLanguageModeling
  title: DataCollatorForLanguageModeling
- local: transformers.DataCollatorForWholeWordMask
  title: DataCollatorForWholeWordMask
- local: transformers.DataCollatorForPermutationLanguageModeling
  title: DataCollatorForPermutationLanguageModeling
title: Data Collator
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="data-collator">Data Collator</h1>

Data collators are objects that will form a batch by using a list of dataset elements as input. These elements are of
the same type as the elements of `train_dataset` or `eval_dataset`.

To be able to build batches, data collators may apply some processing (like padding). Some of them (like
[DataCollatorForLanguageModeling](/docs/transformers/pr_15592/en/main_classes/data_collator#transformers.DataCollatorForLanguageModeling)) also apply some random data augmentation (like random masking)
on the formed batch.

Examples of use can be found in the [example scripts](../examples) or [example notebooks](../notebooks).


<h2 id="transformers.default_data_collator">Default data collator</h2>

<div class="docstring">

<docstring><name>transformers.default\_data\_collator</name><anchor>transformers.default_data_collator</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L48</source><parameters>[{"name": "features", "val": ": typing.List[InputDataClass]"}, {"name": "return_tensors", "val": " = 'pt'"}]</parameters></docstring>

Very simple data collator that simply collates batches of dict-like objects and performs special handling for
potential keys named:

- `label`: handles a single value (int or float) per object
- `label_ids`: handles a list of values per object

Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs
to the model. See glue and ner for example of how it's useful.


</div>

<h2 id="transformers.DefaultDataCollator">DefaultDataCollator</h2>

<div class="docstring">

<docstring><name>class transformers.DefaultDataCollator</name><anchor>transformers.DefaultDataCollator</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L74</source><parameters>[{"name": "return_tensors", "val": ": str = 'pt'"}]</parameters><paramsdesc>- **return_tensors** (`str`) --
  The type of Tensor to return. Allowable values are "np", "pt" and "tf".</paramsdesc><paramgroups>0</paramgroups></docstring>

Very simple data collator that simply collates batches of dict-like objects and performs special handling for
potential keys named:

- `label`: handles a single value (int or float) per object
- `label_ids`: handles a list of values per object

Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs
to the model. See glue and ner for example of how it's useful.

This is an object (like other data collators) rather than a pure function like default_data_collator. This can be
helpful if you need to set a return_tensors value at initialization.




</div>

<h2 id="transformers.DataCollatorWithPadding">DataCollatorWithPadding</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorWithPadding</name><anchor>transformers.DataCollatorWithPadding</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L212</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "padding", "val": ": typing.Union[bool, str, transformers.file_utils.PaddingStrategy] = True"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_to_multiple_of", "val": ": typing.Optional[int] = None"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters><paramsdesc>- **tokenizer** ([PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) or [PreTrainedTokenizerFast](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast)) --
  The tokenizer used for encoding the data.
- **padding** (`bool`, `str` or [PaddingStrategy](/docs/transformers/pr_15592/en/internal/file_utils#transformers.file_utils.PaddingStrategy), *optional*, defaults to `True`) --
  Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
  among:

  - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence
    if provided).
  - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
    acceptable input length for the model if that argument is not provided.
  - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
    lengths).
- **max_length** (`int`, *optional*) --
  Maximum length of the returned list and optionally padding length (see above).
- **pad_to_multiple_of** (`int`, *optional*) --
  If set will pad the sequence to a multiple of the provided value.

  This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
  7.5 (Volta).
- **return_tensors** (`str`) --
  The type of Tensor to return. Allowable values are "np", "pt" and "tf".</paramsdesc><paramgroups>0</paramgroups></docstring>

Data collator that will dynamically pad the inputs received.




</div>

<h2 id="transformers.DataCollatorForTokenClassification">DataCollatorForTokenClassification</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorForTokenClassification</name><anchor>transformers.DataCollatorForTokenClassification</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L264</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "padding", "val": ": typing.Union[bool, str, transformers.file_utils.PaddingStrategy] = True"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_to_multiple_of", "val": ": typing.Optional[int] = None"}, {"name": "label_pad_token_id", "val": ": int = -100"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters><paramsdesc>- **tokenizer** ([PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) or [PreTrainedTokenizerFast](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast)) --
  The tokenizer used for encoding the data.
- **padding** (`bool`, `str` or [PaddingStrategy](/docs/transformers/pr_15592/en/internal/file_utils#transformers.file_utils.PaddingStrategy), *optional*, defaults to `True`) --
  Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
  among:

  - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence
    if provided).
  - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
    acceptable input length for the model if that argument is not provided.
  - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
    lengths).
- **max_length** (`int`, *optional*) --
  Maximum length of the returned list and optionally padding length (see above).
- **pad_to_multiple_of** (`int`, *optional*) --
  If set will pad the sequence to a multiple of the provided value.

  This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
  7.5 (Volta).
- **label_pad_token_id** (`int`, *optional*, defaults to -100) --
  The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions).
- **return_tensors** (`str`) --
  The type of Tensor to return. Allowable values are "np", "pt" and "tf".</paramsdesc><paramgroups>0</paramgroups></docstring>

Data collator that will dynamically pad the inputs received, as well as the labels.




</div>

<h2 id="transformers.DataCollatorForSeq2Seq">DataCollatorForSeq2Seq</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorForSeq2Seq</name><anchor>transformers.DataCollatorForSeq2Seq</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L514</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "model", "val": ": typing.Optional[typing.Any] = None"}, {"name": "padding", "val": ": typing.Union[bool, str, transformers.file_utils.PaddingStrategy] = True"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_to_multiple_of", "val": ": typing.Optional[int] = None"}, {"name": "label_pad_token_id", "val": ": int = -100"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters><paramsdesc>- **tokenizer** ([PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) or [PreTrainedTokenizerFast](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast)) --
  The tokenizer used for encoding the data.
- **model** ([PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel)) --
  The model that is being trained. If set and has the *prepare_decoder_input_ids_from_labels*, use it to
  prepare the *decoder_input_ids*

  This is useful when using *label_smoothing* to avoid calculating loss twice.
- **padding** (`bool`, `str` or [PaddingStrategy](/docs/transformers/pr_15592/en/internal/file_utils#transformers.file_utils.PaddingStrategy), *optional*, defaults to `True`) --
  Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
  among:

  - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence
    is provided).
  - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
    acceptable input length for the model if that argument is not provided.
  - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
    lengths).
- **max_length** (`int`, *optional*) --
  Maximum length of the returned list and optionally padding length (see above).
- **pad_to_multiple_of** (`int`, *optional*) --
  If set will pad the sequence to a multiple of the provided value.

  This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
  7.5 (Volta).
- **label_pad_token_id** (`int`, *optional*, defaults to -100) --
  The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions).
- **return_tensors** (`str`) --
  The type of Tensor to return. Allowable values are "np", "pt" and "tf".</paramsdesc><paramgroups>0</paramgroups></docstring>

Data collator that will dynamically pad the inputs received, as well as the labels.




</div>

<h2 id="transformers.DataCollatorForLanguageModeling">DataCollatorForLanguageModeling</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorForLanguageModeling</name><anchor>transformers.DataCollatorForLanguageModeling</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L607</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "mlm", "val": ": bool = True"}, {"name": "mlm_probability", "val": ": float = 0.15"}, {"name": "pad_to_multiple_of", "val": ": typing.Optional[int] = None"}, {"name": "tf_experimental_compile", "val": ": bool = False"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters><paramsdesc>- **tokenizer** ([PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) or [PreTrainedTokenizerFast](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast)) --
  The tokenizer used for encoding the data.
- **mlm** (`bool`, *optional*, defaults to `True`) --
  Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs
  with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked
  tokens and the value to predict for the masked token.
- **mlm_probability** (`float`, *optional*, defaults to 0.15) --
  The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`.
- **pad_to_multiple_of** (`int`, *optional*) --
  If set will pad the sequence to a multiple of the provided value.
- **return_tensors** (`str`) --
  The type of Tensor to return. Allowable values are "np", "pt" and "tf".</paramsdesc><paramgroups>0</paramgroups></docstring>

Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they
are not all of the same length.



<Tip>

For best performance, this data collator should be used with a dataset having items that are dictionaries or
BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) or a
[PreTrainedTokenizerFast](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) with the argument `return_special_tokens_mask=True`.

</Tip>


<div class="docstring">
<docstring><name>numpy\_mask\_tokens</name><anchor>transformers.DataCollatorForLanguageModeling.numpy_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L805</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "special_tokens_mask", "val": ": typing.Optional[typing.Any] = None"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.


</div>
<div class="docstring">
<docstring><name>tf\_mask\_tokens</name><anchor>transformers.DataCollatorForLanguageModeling.tf_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L659</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "vocab_size", "val": ""}, {"name": "mask_token_id", "val": ""}, {"name": "special_tokens_mask", "val": ": typing.Optional[typing.Any] = None"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.


</div>
<div class="docstring">
<docstring><name>torch\_mask\_tokens</name><anchor>transformers.DataCollatorForLanguageModeling.torch_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L748</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "special_tokens_mask", "val": ": typing.Optional[typing.Any] = None"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.


</div></div>

<h2 id="transformers.DataCollatorForWholeWordMask">DataCollatorForWholeWordMask</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorForWholeWordMask</name><anchor>transformers.DataCollatorForWholeWordMask</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L846</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "mlm", "val": ": bool = True"}, {"name": "mlm_probability", "val": ": float = 0.15"}, {"name": "pad_to_multiple_of", "val": ": typing.Optional[int] = None"}, {"name": "tf_experimental_compile", "val": ": bool = False"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters></docstring>

Data collator used for language modeling that masks entire words.

- collates batches of tensors, honoring their tokenizer's pad_token
- preprocesses batches for masked language modeling

<Tip>

This collator relies on details of the implementation of subword tokenization by [BertTokenizer](/docs/transformers/pr_15592/en/model_doc/bert#transformers.BertTokenizer), specifically
that subword tokens are prefixed with *##*. For tokenizers that do not adhere to this scheme, this collator will
produce an output that is roughly equivalent to `.DataCollatorForLanguageModeling`.

</Tip>


<div class="docstring">
<docstring><name>numpy\_mask\_tokens</name><anchor>transformers.DataCollatorForWholeWordMask.numpy_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1072</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "mask_labels", "val": ": typing.Any"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set
'mask_labels' means we use whole word mask (wwm), we directly mask idxs according to it's ref.


</div>
<div class="docstring">
<docstring><name>tf\_mask\_tokens</name><anchor>transformers.DataCollatorForWholeWordMask.tf_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1031</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "mask_labels", "val": ": typing.Any"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set
'mask_labels' means we use whole word mask (wwm), we directly mask idxs according to it's ref.


</div>
<div class="docstring">
<docstring><name>torch\_mask\_tokens</name><anchor>transformers.DataCollatorForWholeWordMask.torch_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L992</source><parameters>[{"name": "inputs", "val": ": typing.Any"}, {"name": "mask_labels", "val": ": typing.Any"}]</parameters></docstring>

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original. Set
'mask_labels' means we use whole word mask (wwm), we directly mask idxs according to it's ref.


</div></div>

<h2 id="transformers.DataCollatorForPermutationLanguageModeling">DataCollatorForPermutationLanguageModeling</h2>

<div class="docstring">

<docstring><name>class transformers.DataCollatorForPermutationLanguageModeling</name><anchor>transformers.DataCollatorForPermutationLanguageModeling</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1197</source><parameters>[{"name": "tokenizer", "val": ": PreTrainedTokenizerBase"}, {"name": "plm_probability", "val": ": float = 0.16666666666666666"}, {"name": "max_span_length", "val": ": int = 5"}, {"name": "return_tensors", "val": ": str = 'pt'"}]</parameters></docstring>

Data collator used for permutation language modeling.

- collates batches of tensors, honoring their tokenizer's pad_token
- preprocesses batches for permutation language modeling with procedures specific to XLNet



<div class="docstring">
<docstring><name>numpy\_mask\_tokens</name><anchor>transformers.DataCollatorForPermutationLanguageModeling.numpy_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1436</source><parameters>[{"name": "inputs", "val": ": typing.Any"}]</parameters></docstring>

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be
   masked
3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -
   span_length]` and mask tokens `start_index:start_index + span_length`
4. Set `cur_len = cur_len + context_length`. If `cur_len &amp;lt; max_len` (i.e. there are tokens remaining in the
   sequence to be processed), repeat from Step 1.


</div>
<div class="docstring">
<docstring><name>tf\_mask\_tokens</name><anchor>transformers.DataCollatorForPermutationLanguageModeling.tf_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1328</source><parameters>[{"name": "inputs", "val": ": typing.Any"}]</parameters></docstring>

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be
   masked
3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -
   span_length]` and mask tokens `start_index:start_index + span_length`
4. Set `cur_len = cur_len + context_length`. If `cur_len &amp;lt; max_len` (i.e. there are tokens remaining in the
   sequence to be processed), repeat from Step 1.


</div>
<div class="docstring">
<docstring><name>torch\_mask\_tokens</name><anchor>transformers.DataCollatorForPermutationLanguageModeling.torch_mask_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/data_collator.py#L1231</source><parameters>[{"name": "inputs", "val": ": typing.Any"}]</parameters></docstring>

The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be
   masked
3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -
   span_length]` and mask tokens `start_index:start_index + span_length`
4. Set `cur_len = cur_len + context_length`. If `cur_len &amp;lt; max_len` (i.e. there are tokens remaining in the
   sequence to be processed), repeat from Step 1.


</div></div>
