---
local: processors
sections:
- local: transformers.ProcessorMixin
  title: Multi-modal processors
- local: transformers.DataProcessor
  title: Deprecated processors
- local: glue
  sections:
  - local: example-usage
    title: Example usage
  title: GLUE
- local: xnli
  title: XNLI
- local: squad
  sections:
  - local: transformers.data.processors.squad.SquadProcessor
    title: Processors
  - local: example-usage
    title: Example usage
  title: SQuAD
title: Processors
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="processors">Processors</h1>

Processors can mean two different things in the Transformers library:
- the objects that pre-process inputs for multi-modal models such as [Wav2Vec2](../model_doc/wav2vec2) (speech and text)
  or [CLIP](../model_doc/clip) (text and vision)
- deprecated objects that were used in older versions of the library to preprocess data for GLUE or SQUAD.

<h2 id="transformers.ProcessorMixin">Multi-modal processors</h2>

Any multi-modal model will require an object to encode or decode the data that groups several modalities (among text,
vision and audio). This is handled by objects called processors, which group tokenizers (for the text modality) and
feature extractors (for vision and audio).

Those processors inherit from the following base class that implements the saving and loading functionality:

<div class="docstring">

<docstring><name>class transformers.ProcessorMixin</name><anchor>transformers.ProcessorMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/processing_utils.py#L44</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

This is a mixin used to provide saving/loading functionality for all processor classes.



<div class="docstring">
<docstring><name>from\_pretrained</name><anchor>transformers.ProcessorMixin.from_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/processing_utils.py#L157</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str` or `os.PathLike`) --
  This can be either:

  - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
    huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or
    namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
  - a path to a *directory* containing a feature extractor file saved using the
    [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) method, e.g., `./my_model_directory/`.
  - a path or url to a saved feature extractor JSON *file*, e.g.,
    `./my_model_directory/preprocessor_config.json`.
**kwargs --
Additional keyword arguments passed along to both
[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained) and
`from_pretrained`.</paramsdesc><paramgroups>0</paramgroups></docstring>

Instantiate a processor associated with a pretrained model.

<Tip>

This class method is simply calling the feature extractor
[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.from_pretrained) and the tokenizer
`from_pretrained` methods. Please refer to the docstrings of the
methods above for more information.

</Tip>




</div>
<div class="docstring">
<docstring><name>push\_to\_hub</name><anchor>transformers.file_utils.PushToHubMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2825</source><parameters>[{"name": "repo_path_or_name", "val": ": typing.Optional[str] = None"}, {"name": "repo_url", "val": ": typing.Optional[str] = None"}, {"name": "use_temp_dir", "val": ": bool = False"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "organization", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "**model_card_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`, *optional*) --
  Can either be a repository name for your processor in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder). If not specified, will default to the name
  given by `repo_url` and a local directory with that name will be created.
- **repo_url** (`str`, *optional*) --
  Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
  repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
- **use_temp_dir** (`bool`, *optional*, defaults to `False`) --
  Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
  current working directory. This will slow things down if you are making changes in an existing repo
  since you will need to clone the repo before every push.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"add processor"`.
- **organization** (`str`, *optional*) --
  Organization in which you want to push your processor (you must be a member of this organization).
- **private** (`bool`, *optional*) --
  Whether or not the repository created should be private (requires a paying subscription).
- **use_auth_token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
  `repo_url` is not specified.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>The url of the commit of your processor in the given repository.</retdesc></docstring>

Upload the processor files to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.







Examples:

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("bert-base-cased")

# Push the processor to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
processor.push_to_hub("my-finetuned-bert")

# Push the processor to your namespace with the name "my-finetuned-bert" with no local clone.
processor.push_to_hub("my-finetuned-bert", use_temp_dir=True)

# Push the processor to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
processor.push_to_hub("my-finetuned-bert", organization="huggingface")

# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
processor.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```


</div>
<div class="docstring">
<docstring><name>register\_for\_auto\_class</name><anchor>transformers.ProcessorMixin.register_for_auto_class</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/processing_utils.py#L190</source><parameters>[{"name": "auto_class", "val": " = 'AutoProcessor'"}]</parameters><paramsdesc>- **auto_class** (`str` or `type`, *optional*, defaults to `"AutoProcessor"`) --
  The auto class to register this new feature extractor with.</paramsdesc><paramgroups>0</paramgroups></docstring>

Register this class with a given auto class. This should only be used for custom feature extractors as the ones
in the library are already mapped with `AutoProcessor`.

<Tip warning=&amp;lcub;true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>




</div>
<div class="docstring">
<docstring><name>save\_pretrained</name><anchor>transformers.ProcessorMixin.save_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/processing_utils.py#L95</source><parameters>[{"name": "save_directory", "val": ""}, {"name": "push_to_hub", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **save_directory** (`str` or `os.PathLike`) --
  Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will
  be created if it does not exist).
- **push_to_hub** (`bool`, *optional*, defaults to `False`) --
  Whether or not to push your processor to the Hugging Face model hub after saving it.

  <Tip warning=&amp;lcub;true}>

  Using `push_to_hub=True` will synchronize the repository you are pushing to with `save_directory`,
  which requires `save_directory` to be a local clone of the repo you are pushing to if it's an existing
  folder. Pass along `temp_dir=True` to use a temporary directory instead.

  </Tip>

  kwargs --
  Additional key word arguments passed along to the [push_to_hub()](/docs/transformers/pr_15592/en/main_classes/model#transformers.file_utils.PushToHubMixin.push_to_hub) method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it
can be reloaded using the [from_pretrained()](/docs/transformers/pr_15592/en/main_classes/processors#transformers.ProcessorMixin.from_pretrained) method.

<Tip>

This class method is simply calling [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/feature_extractor#transformers.FeatureExtractionMixin.save_pretrained) and
`save_pretrained`. Please refer to the docstrings of the methods
above for more information.

</Tip>




</div></div>

<h2 id="transformers.DataProcessor">Deprecated processors</h2>

All processors follow the same architecture which is that of the
[DataProcessor](/docs/transformers/pr_15592/en/main_classes/processors#transformers.DataProcessor). The processor returns a list of
[InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample). These
[InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample) can be converted to
[InputFeatures](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputFeatures) in order to be fed to the model.

<div class="docstring">

<docstring><name>class transformers.DataProcessor</name><anchor>transformers.DataProcessor</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L81</source><parameters>[]</parameters></docstring>
Base class for data converters for sequence classification data sets.


<div class="docstring">
<docstring><name>get\_dev\_examples</name><anchor>transformers.DataProcessor.get_dev_examples</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L98</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample) for the dev set.

</div>
<div class="docstring">
<docstring><name>get\_example\_from\_tensor\_dict</name><anchor>transformers.DataProcessor.get_example_from_tensor_dict</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L84</source><parameters>[{"name": "tensor_dict", "val": ""}]</parameters><paramsdesc>tensor_dict -- Keys and values should match the corresponding Glue
tensorflow_dataset examples.</paramsdesc><paramgroups>0</paramgroups></docstring>

Gets an example from a dict with tensorflow tensors.




</div>
<div class="docstring">
<docstring><name>get\_labels</name><anchor>transformers.DataProcessor.get_labels</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L106</source><parameters>[]</parameters></docstring>
Gets the list of labels for this data set.

</div>
<div class="docstring">
<docstring><name>get\_test\_examples</name><anchor>transformers.DataProcessor.get_test_examples</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L102</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample) for the test set.

</div>
<div class="docstring">
<docstring><name>get\_train\_examples</name><anchor>transformers.DataProcessor.get_train_examples</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L94</source><parameters>[{"name": "data_dir", "val": ""}]</parameters></docstring>
Gets a collection of [InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample) for the train set.

</div>
<div class="docstring">
<docstring><name>tfds\_map</name><anchor>transformers.DataProcessor.tfds_map</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L110</source><parameters>[{"name": "example", "val": ""}]</parameters></docstring>

Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts
examples to the correct format.


</div></div>

<div class="docstring">

<docstring><name>class transformers.InputExample</name><anchor>transformers.InputExample</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L31</source><parameters>[{"name": "guid", "val": ": str"}, {"name": "text_a", "val": ": str"}, {"name": "text_b", "val": ": typing.Optional[str] = None"}, {"name": "label", "val": ": typing.Optional[str] = None"}]</parameters><paramsdesc>guid -- Unique id for the example.
text_a -- string. The untokenized text of the first sequence. For single
sequence tasks, only this sequence must be specified.
text_b -- (Optional) string. The untokenized text of the second sequence.
Only must be specified for sequence pair tasks.
label -- (Optional) string. The label of the example. This should be
specified for train and dev examples, but not for test examples.</paramsdesc><paramgroups>0</paramgroups></docstring>

A single training/test example for simple sequence classification.





<div class="docstring">
<docstring><name>to\_json\_string</name><anchor>transformers.InputExample.to_json_string</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L50</source><parameters>[]</parameters></docstring>
Serializes this instance to a JSON string.

</div></div>

<div class="docstring">

<docstring><name>class transformers.InputFeatures</name><anchor>transformers.InputFeatures</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L56</source><parameters>[{"name": "input_ids", "val": ": typing.List[int]"}, {"name": "attention_mask", "val": ": typing.Optional[typing.List[int]] = None"}, {"name": "token_type_ids", "val": ": typing.Optional[typing.List[int]] = None"}, {"name": "label", "val": ": typing.Union[int, float, NoneType] = None"}]</parameters><paramsdesc>input_ids -- Indices of input sequence tokens in the vocabulary.
attention_mask -- Mask to avoid performing attention on padding token indices.
Mask values selected in `[0, 1]`: Usually `1` for tokens that are NOT MASKED, `0` for MASKED (padded)
tokens.
token_type_ids -- (Optional) Segment token indices to indicate first and second
portions of the inputs. Only some models use them.
label -- (Optional) Label corresponding to the input. Int for classification problems,
float for regression problems.</paramsdesc><paramgroups>0</paramgroups></docstring>

A single set of features of data. Property names are the same names as the corresponding inputs to a model.





<div class="docstring">
<docstring><name>to\_json\_string</name><anchor>transformers.InputFeatures.to_json_string</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/utils.py#L76</source><parameters>[]</parameters></docstring>
Serializes this instance to a JSON string.

</div></div>

<h2 id="glue">GLUE</h2>

[General Language Understanding Evaluation (GLUE)](https://gluebenchmark.com/) is a benchmark that evaluates the
performance of models across a diverse set of existing NLU tasks. It was released together with the paper [GLUE: A
multi-task benchmark and analysis platform for natural language understanding](https://openreview.net/pdf?id=rJ4km2R5t7)

This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched), CoLA, SST2, STSB,
QQP, QNLI, RTE and WNLI.

Those processors are:

- `MrpcProcessor`
- `MnliProcessor`
- `MnliMismatchedProcessor`
- `Sst2Processor`
- `StsbProcessor`
- `QqpProcessor`
- `QnliProcessor`
- `RteProcessor`
- `WnliProcessor`

Additionally, the following method can be used to load values from a data file and convert them to a list of
[InputExample](/docs/transformers/pr_15592/en/main_classes/processors#transformers.InputExample).

automethod,transformers.data.processors.glue.glue_convert_examples_to_features


<h3 id="example-usage">Example usage</h3>

An example using these processors is given in the [run_glue.py](https://github.com/huggingface/transformers/tree/master/examples/legacy/text-classification/run_glue.py) script.


<h2 id="xnli">XNLI</h2>

[The Cross-Lingual NLI Corpus (XNLI)](https://www.nyu.edu/projects/bowman/xnli/) is a benchmark that evaluates the
quality of cross-lingual text representations. XNLI is crowd-sourced dataset based on [*MultiNLI*](http://www.nyu.edu/projects/bowman/multinli/): pairs of text are labeled with textual entailment annotations for 15
different languages (including both high-resource language such as English and low-resource languages such as Swahili).

It was released together with the paper [XNLI: Evaluating Cross-lingual Sentence Representations](https://arxiv.org/abs/1809.05053)

This library hosts the processor to load the XNLI data:

- `XnliProcessor`

Please note that since the gold labels are available on the test set, evaluation is performed on the test set.

An example using these processors is given in the [run_xnli.py](https://github.com/huggingface/transformers/tree/master/examples/legacy/text-classification/run_xnli.py) script.


<h2 id="squad">SQuAD</h2>

[The Stanford Question Answering Dataset (SQuAD)](https://rajpurkar.github.io/SQuAD-explorer//) is a benchmark that
evaluates the performance of models on question answering. Two versions are available, v1.1 and v2.0. The first version
(v1.1) was released together with the paper [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://arxiv.org/abs/1606.05250). The second version (v2.0) was released alongside the paper [Know What You Don't
Know: Unanswerable Questions for SQuAD](https://arxiv.org/abs/1806.03822).

This library hosts a processor for each of the two versions:

<h3 id="transformers.data.processors.squad.SquadProcessor">Processors</h3>

Those processors are:

- `SquadV1Processor`
- `SquadV2Processor`

They both inherit from the abstract class `SquadProcessor`

<div class="docstring">

<docstring><name>class transformers.data.processors.squad.SquadProcessor</name><anchor>transformers.data.processors.squad.SquadProcessor</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/squad.py#L543</source><parameters>[]</parameters></docstring>

Processor for the SQuAD data set. overridden by SquadV1Processor and SquadV2Processor, used by the version 1.1 and
version 2.0 of SQuAD, respectively.



<div class="docstring">
<docstring><name>get\_dev\_examples</name><anchor>transformers.data.processors.squad.SquadProcessor.get_dev_examples</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/squad.py#L631</source><parameters>[{"name": "data_dir", "val": ""}, {"name": "filename", "val": " = None"}]</parameters><paramsdesc>data_dir -- Directory containing the data files used for training and evaluating.
filename -- None by default, specify this if the evaluation file has a different name than the original one
which is `dev-v1.1.json` and `dev-v2.0.json` for squad versions 1.1 and 2.0 respectively.</paramsdesc><paramgroups>0</paramgroups></docstring>

Returns the evaluation example from the data directory.




</div>
<div class="docstring">
<docstring><name>get\_examples\_from\_dataset</name><anchor>transformers.data.processors.squad.SquadProcessor.get_examples_from_dataset</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/squad.py#L576</source><parameters>[{"name": "dataset", "val": ""}, {"name": "evaluate", "val": " = False"}]</parameters><paramsdesc>dataset -- The tfds dataset loaded from *tensorflow_datasets.load("squad")*
evaluate -- Boolean specifying if in evaluation mode or in training mode</paramsdesc><paramgroups>0</paramgroups><retdesc>List of SquadExample</retdesc></docstring>

Creates a list of `SquadExample`using a TFDS dataset.





Examples:

```python
>>> import tensorflow_datasets as tfds

>>> dataset = tfds.load("squad")

>>> training_examples = get_examples_from_dataset(dataset, evaluate=False)
>>> evaluation_examples = get_examples_from_dataset(dataset, evaluate=True)
```

</div>
<div class="docstring">
<docstring><name>get\_train\_examples</name><anchor>transformers.data.processors.squad.SquadProcessor.get_train_examples</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/data/processors/squad.py#L609</source><parameters>[{"name": "data_dir", "val": ""}, {"name": "filename", "val": " = None"}]</parameters><paramsdesc>data_dir -- Directory containing the data files used for training and evaluating.
filename -- None by default, specify this if the training file has a different name than the original one
which is `train-v1.1.json` and `train-v2.0.json` for squad versions 1.1 and 2.0 respectively.</paramsdesc><paramgroups>0</paramgroups></docstring>

Returns the training examples from the data directory.




</div></div>

Additionally, the following method can be used to convert SQuAD examples into
`SquadFeatures` that can be used as model inputs.

automethod,transformers.data.processors.squad.squad_convert_examples_to_features


These processors as well as the aforementionned method can be used with files containing the data as well as with the
*tensorflow_datasets* package. Examples are given below.


<h3 id="example-usage">Example usage</h3>

Here is an example using the processors as well as the conversion method using data files:

```python
# Loading a V2 processor
processor = SquadV2Processor()
examples = processor.get_dev_examples(squad_v2_data_dir)

# Loading a V1 processor
processor = SquadV1Processor()
examples = processor.get_dev_examples(squad_v1_data_dir)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)
```

Using *tensorflow_datasets* is as easy as using a data file:

```python
# tensorflow_datasets only handle Squad V1.
tfds_examples = tfds.load("squad")
examples = SquadV1Processor().get_examples_from_dataset(tfds_examples, evaluate=evaluate)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)
```

Another example using these processors is given in the [run_squad.py](https://github.com/huggingface/transformers/tree/master/examples/legacy/question-answering/run_squad.py) script.
