[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests
Fix #4015 

Edit @thomwolf: I morphed this in a large refactoring of the tokenizer code and test to make it more flexible and have a better API. Here is a summary of the changes.

## Breaking change
There is no breaking change in the user-facing methods (`encode`, `encode_plus`, `batch_encode_plus`, `tokenize`, `convert_XXX`).

There is a breaking change in the internal method`prepare_for_model` which is now a private method `_prepare_for_model` with a simplified signature.

## A new main user-facing method: `__call__` i.e. `model_input = tokenizer(text, **kwargs)`

The extended encoding methods `encode_plus` and `batch_encode_plus` methods had names that could be intimidating for first-time users.

A new main entry point is created as `tokenizer.__call__` which wraps both methods. You can feed `__call__` with single examples, a pair of sentence to encode together or batches of single/pair sentences.

The signature of `__call__` is also a better fit for the 🤗nlp library when it comes to batches of pairs of sequences since the first and second elements in pair of sentences are supplied as separate arguments (see below) instead of a zipped list of pairs like in `batch_encode_plus`.

While all the previously provided methods (`encode`, `encode_plus`, `batch_encode_plus`, `tokenize`, `convert_XXX`) are still supported without breaking changes, `__call__` is now the recommended way to encode all types of inputs when `tokenizer.encode` (which only return the list of input indices for a single sentence) is not enough i.e. for every case beside simple demo purposes.

Here is how you should use this new entry point for encoding text in all the main use-cases:
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

# 1. When you encode "a single sentence"
encoded_input = tokenizer("Hello I'm a single sentence")
# { 'input_ids': [101, 8667, 146, 112, 182, 170, 1423, 5650, 102],
#   'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0],
#   'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}

# 2. When you encode "a pair of sentences in a single input"
encoded_input = tokenizer("How old are you?", "I'm 6 years old")
# { 'input_ids': [101, 1731, 1385, 1132, 1128, 136, 102, 146, 112, 182, 127, 1201, 1385, 102],
#   'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],
#   'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

# 3. When you encode "a batch of single sentences"
batch_sentences = ["Hello I'm a single sentence",
                   "And another sentence",
                   "And the very very last one"]
encoded_input = tokenizer(batch_sentences)
# { 'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102],
#                 [101, 1262, 1330, 5650, 102],
#                 [101, 1262, 1103, 1304, 1304, 1314, 1141, 102]],
#   'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0],
#                      [0, 0, 0, 0, 0],
#                      [0, 0, 0, 0, 0, 0, 0, 0]],
#   'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1],
#                      [1, 1, 1, 1, 1],
#                      [1, 1, 1, 1, 1, 1, 1, 1]]}

# You can batch (to max sequence size) and truncate (to max model length)
# with `padding`and `truncation` (see more details in the next section on padding/truncation)
encoded_input = tokenizer(batch_sentences, padding=True, truncation=True)
# { 'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102],
#                 [101, 1262, 1330, 5650, 102, 0, 0, 0, 0],
#                 [101, 1262, 1103, 1304, 1304, 1314, 1141, 102, 0]],
#   'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0],
#                      [0, 0, 0, 0, 0, 0, 0, 0, 0],
#                      [0, 0, 0, 0, 0, 0, 0, 0, 0]],
#   'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1],
#                      [1, 1, 1, 1, 1, 0, 0, 0, 0],
#                      [1, 1, 1, 1, 1, 1, 1, 1, 0]]}

# 4. When you encode "a batch of pair of sentences"
batch_of_second_sentences = ["I'm a sentence that goes with the first sentence",
                             "And I should be encoded with the second sentence",
                             "And I go with the very last one"]
encoded_input = tokenizer(batch_sentences,
                          batch_of_second_sentences,
                          padding=True,
                          truncation=True)
# { 'input_ids': [[101, 8667, 146, 112, 182, 170, 1423, 5650, 102, 146, 112, 182, 170, 5650, 1115, 2947, 1114, 1103, 1148, 5650, 102],
#                 [101, 1262, 1330, 5650, 102, 1262, 146, 1431, 1129, 12544, 1114, 1103, 1248, 5650, 102, 0, 0, 0, 0, 0, 0],
#                 [101, 1262, 1103, 1304, 1304, 1314, 1141, 102, 1262, 146, 1301, 1114, 1103, 1304, 1314, 1141, 102, 0, 0, 0, 0]],
#   'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
#                      [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],
#                      [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]],
#   'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
#                      [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0],
#                      [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]]}
```

## Padding/truncation

The padding and truncation logic was simplified and improved to cover all the major uses-cases with the simplest possible API.

Here is how to do the two most common use-cases for truncation/padding:
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
batch_sentences = ["Hello I'm a single sentence",
                   "And another sentence",
                   "And the very very last one"]

# 1. No truncation and no padding
encoded_input = tokenizer(batch_sentences)

# 2. Pad to the max sequence length inside the provided batch
# while truncating to the max input length acceptable by the model 
encoded_input = tokenizer(batch_sentences, truncation=True, padding=True)
```

The new API for padding and truncation uses three arguments to the encoding methods: `padding`, `truncation` and `max_length`. This new way to specify padding/truncation is available in all the user-facing encoding methods: `encode`, `encode_plus`, `batch_ecode_plus` and the newly provided `__call__`.

All the previously provided ways to do padding/truncation (`truncation_strategy`, `max_length`, `pad_to_max_length`) are still supported without breaking changes but we recommend to use the new API.

Here are the details of all the possible inputs to `padding`, `truncation` and `max_length`:

- `padding` to control the padding (can be provided with a boolean or a string for finer-grained control). `padding` accepts the following values:

    * `True` or `'longest'`: pad to the longest sequence in the batch (or no padding if only a single sequence if provided),
    * `'max_length'`: pad to a max length specified in `max_length` or to the max acceptable input length for the model if no length is provided (`max_length=None`)
    * `False` or `'do_not_pad'` (default): No padding (i.e. can output batch with sequences of uneven lengths)
- `truncation` to control truncation (can be provided with a boolean or a string for finer-grained control). `truncation` accepts the following values:

    * `True` or `'only_first'`: truncate to a max length specified in `max_length` or to the max acceptable input length for the model if no length is provided (`max_length=None`). This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
    * `'only_second'`: truncate to a max length specified in `max_length` or to the max acceptable input length for the model if no length is provided (`max_length=None`). This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided,
    * `'longest_first'`: truncate to a max length specified in `max_length` or to the max acceptable input length for the model if no length is provided (`max_length=None`). This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided,
    * `False` or `'do_not_truncate'` (default): No truncation (i.e. can output batch with sequences length greater than the model max admissible input size)
- `max_length` to control the length of the padding/truncation (integer or `None`). `max_length` accepts the following values:

    * `None` (default): This will use the predefined model max length if required by one of the truncation/padding parameters. If the model has no specific max input length (e.g. XLNet) truncation/padding to max length is deactivated.
    * `any integer value` (e.g. `42`): Use this specific maximum length value if required by one of the truncation/padding parameters.

Now here is a table summarizing the recommended way to setup `padding` and `truncation` as well as the previously provided way to do it (still supported but not recommended) in all cases.

If you use pair of inputs sequence in any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in `['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation= 'longest_first'` to control how both sequence in the pair are truncated as detailed just before the table. We don't include all these variants for the sake of keeping the table not too long.

| Truncation | Padding | Recommended way | Previously provided (still supported but not recommended)
| --- | --- | --- | --- |
| no truncation | no padding | `tokenizer(batch_sentences)` | `tokenizer.batch_encode_plus(batch_sentences)`
| no truncation | padding to max sequence in batch | `tokenizer(batch_sentences, padding=True)` or `tokenizer(batch_sentences, padding='longest')`| `tokenizer.batch_encode_plus(batch_sentences, pad_to_max_length=True)`
| no truncation | padding to max model input length | `tokenizer(batch_sentences, padding='max_length')` | Not possible
| no truncation | padding to specific length | `tokenizer(batch_sentences, padding='max_length', max_length=42)` | Not possible
|  |  |  |  |
| truncation to max model input length | no padding | `tokenizer(batch_sentences, truncation=True)` or `tokenizer(batch_sentences, truncation=STRATEGY)` | `tokenizer.batch_encode_plus(batch_sentences, max_length=tokenizer.max_len)`
| truncation to max model input length | padding to max sequence in batch | `tokenizer(batch_sentences, padding=True, truncation=True)` or `tokenizer(batch_sentences, padding=True, truncation=STRATEGY)` | Not possible
| truncation to max model input length | padding to max model input length | `tokenizer(batch_sentences, padding='max_length', truncation=True)` or `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)` | `tokenizer.batch_encode_plus(batch_sentences, pad_to_max_length=True, max_length=tokenizer.max_len)`
| truncation to max model input length | padding to specific length | Not possible | Not possible
|  |  |  |  |
| truncation to specific length | no padding | `tokenizer(batch_sentences, truncation=True, max_length=42)` or `tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)` | `tokenizer.batch_encode_plus(batch_sentences, max_length=42)`
| truncation to specific length | padding to max sequence in batch | `tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)` or `tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)` | Not possible
| truncation to specific length | padding to max model input length | Not possible | Not possible
| truncation to specific length | padding to specific length | `tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)` or `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)` | `tokenizer.batch_encode_plus(batch_sentences, pad_to_max_length=True, max_length=42)`

## Pre-tokenized inputs

The tokenizers now accept pre-tokenized inputs, i.e. inputs which are already sliced in words. The main reason for implementing a specific track for this type of inputs is to be able to use the fast mapping methods in `tokenizers` which provide character<=>token<=>words mappings. This can be very handy to easily compute labels and extract predictions for instance for Named-Entity-Recognition (NER) or Part-of-Speech tagging (POS tagging).

If you want to use pre-tokenized inputs, just set `is_pretokenized=True` in any of the encoding methods. Here are some examples:
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
batch_sentences = [["Hello", "I'm", "a", "single", "sentence"],
                   ["And", "another", "sentence"],
                   ["And", "the", "very", "very", "last", "one"]]

encoded_input = tokenizer(batch_sentences, is_pretokenized=True)

# Pre-tokenized inputs can be used in all cases (single/pair/batch of single/batch of pairs) 
batch_of_second_sentences = ["I'm a sentence that goes with the first sentence".split(),
                             "And I should be encoded with the second sentence".split(),
                             "And I go with the very last one".split()]
encoded_input = tokenizer(batch_sentences,
                          batch_of_second_sentences,
                          is_pretokenized=True,
                          padding=True,
                          truncation=True)
```

## Verbose

A new `verbose` argument is provided in all the encoding methods to silence all the warnings related to the length of the input as well as missing special tokens (e.g. missing padding or unknown token).

## Code organization

`tokenization_utils.py` was starting to grow out of control and is now split into three files:
-  `tokenization_utils.py` hosts the code for the `PreTrainedTokenizers`
-  `tokenization_utils_fast.py` hosts the code for the `PreTrainedTokenizersFast`
-  `tokenization_utils_base.py` hosts the common methods for `PreTrainedTokenizers` and `PreTrainedTokenizersFast` (mostly the front API) in a newly created `PretrainedTokenizerBase` as well as all the common logic for special tokens (in `SpecialMixin`) and for the outputs of the encoding (in `BatchEncoding`).

## Full testing of fast tokenizers

The fast tokenizers provided by the [tokenizers](https://github.com/huggingface/tokenizers)  library are now fully tested and follow the same testing pipeline as the python (slow) tokenizers. Additional consistency tests have been added comparing the outputs of the fast and slow tokenizers under various conditions.

## TODO (following PRs)
- Serialization for Fast tokenizers
- Some edge cases for `add_tokens` on Fast tokenizers are not covered (spaces in tokens for byte-level and lower casing of the added tokens).
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=h1) Report
> Merging [#4510](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9931f817b75ecb2c8bb08b6e9d4cbec4b0933935&el=desc) will **increase** coverage by `0.54%`.
> The diff coverage is `92.01%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4510/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4510      +/-   ##
==========================================
+ Coverage   76.89%   77.43%   +0.54%     
==========================================
  Files         128      130       +2     
  Lines       21854    21966     +112     
==========================================
+ Hits        16804    17010     +206     
+ Misses       5050     4956      -94     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `75.33% <ø> (ø)` | |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `94.81% <ø> (+5.11%)` | :arrow_up: |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `91.55% <91.55%> (ø)` | |
| [src/transformers/tokenization\_utils\_fast.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfZmFzdC5weQ==) | `92.59% <92.59%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.14% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `90.45% <100.00%> (-0.80%)` | :arrow_down: |
| [src/transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZ3B0Mi5weQ==) | `97.08% <100.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fb3BlbmFpLnB5) | `83.84% <100.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/tokenization\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdHJhbnNmb194bC5weQ==) | `40.82% <100.00%> (+0.14%)` | :arrow_up: |
| [src/transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtX3JvYmVydGEucHk=) | `95.23% <0.00%> (-2.39%)` | :arrow_down: |
| ... and [11 more](https://codecov.io/gh/huggingface/transformers/pull/4510/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=footer). Last update [9931f81...52a30d6](https://codecov.io/gh/huggingface/transformers/pull/4510?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<|||||>Ok, I morphed this in a large refactoring of the tokenizer code and test to make it more flexible and have a better API.

Here is a summary of the changes:
- there is now a new main user-facing method: `__call__` i.e. model_input = tokenizer(text, **kwargs) which should be the main entry point for converting text in model inputs in the future,
- the padding/truncation logic was refactored to cover more cases and make the most common-case more natural to access
- pre-tokenized inputs (e.g. for NER or POS tagging) are handled a lot better
- the backend code was refactored and split in several files.

There is no breaking change in the user-facing methods (`encode`, `encode_plus`, `batch_encode_plus`, `tokenize`, `convert_XXX`). There is a breaking change in the internal method `prepare_for_model` which is now a private method `_prepare_for_model` with a simplified signature.

All the details are given in the updated description of the PR.

cc @LysandreJik @julien-c @patrickvonplaten @sshleifer @mfuntowicz @yjernite @srush @mariamabarham @lhoestq @VictorSanh @jplu @stefan-it @BramVanroy <|||||>I always love to see changes that improve the usability. I think using __call__ is one that can really make things easier for people to use. I also like pre-tokenized inputs a lot, since most of my data is pre-tokenized anyway. 

The changes are quite big to go over, so just checking: hopefully there are very clear error messages when users choose incompatible options when running the tokenization process. Making the tokenizer easier to use by having a single entry-point is great, but not so much if it can create more user mistakes that are not clear to the user. Clear error messages are key.

A feature request, that I discussed with someone before but I don't remember who, is that it would be nice if the tokenizers could have an optional `device` argument. If we use return_tensors, it should return the tensors immediately on the given devices, e.g.

```python
encoded_on_device = tokenizer(["Hello world.", "Who likes cookies?"], device=torch.device("cuda:0"))
# or
encoded_on_device = tokenizer(["Hello world.", "Who likes cookies?"], device=training_args.device)
```

Might even allow different type of values like device integers or "cuda" or "cpu" strings, and so on.

Great job! Looking forward to using this in practice.<|||||>This is awesome!! Really great work and congratulations with this huge rework of the tokenizers!!!

It is a bit too huge to go through everything but as far as I can see, the way to use the tokenizers now are way more accessible, mostly the pre-tokenizerd part.

> A feature request, that I discussed with someone before but I don't remember who, is that it would be nice if the tokenizers could have an optional device argument. If we use return_tensors, it should return the tensors immediately on the given devices

@BramVanroy I don't think it is the place here because it is not compliant with TF :) I think that the tokenizers should stay as much framework agnostic as possible otherwise if we start to say "if you want to use the tokenizer for PT do that, and for TF do this" it becomes more complicated to maintain. Of course this is only my opinion nothing more :)<|||||>> @BramVanroy I don't think it is the place here because it is not compliant with TF :) I think that the tokenizers should stay as much framework agnostic as possible otherwise if we start to say "if you want to use the tokenizer for PT do that, and for TF do this" it becomes more complicated to maintain. Of course this is only my opinion nothing more :)

But that's what we do for `return_tensors` anyway, right?<|||||>> But that's what we do for return_tensors anyway, right?

Exactly, and I think the same about this parameter, it adds complexity, while this can be easily done afterward.<|||||>> Exactly, and I think the same about this parameter, it adds complexity, while this can be easily done afterward.

It is true that this can be done easily afterwards, but I suppose this is one of those cases: how much ease-of-use do you want your library to have while also taking into account the complexity of the library itself. My main argument is that from a usability perspective it would be awesome to be able to just provide your text to the tokenizer and you immediately get the encoded input back that you can feed to your model without having to do anything else. You then even do this:

```python
out = model(**tokenizer(input_text, return_tensors="pt", device=device))
```

This isn't pretty but it illustrates my point that it makes _usage_ very easy and also _easy to understand_. It removes a lot of booilerplate stuff that as a user you don't want to spend time on. On the other hand I definitely understand your point that this will lead to more complexity on the library's side. I'd be interested to hear other people's opinions about this.<|||||>> how much ease-of-use do you want your library to have while also taking into account the complexity of the library itself.

This is definitely true, I fully agree :) And what you propose makes sense as well. I would be curious to hear other opinions too ^^<|||||>As seen with @thomwolf, will merge this PR as soon as the tests show all green. I'm updating all the library's docstrings to showcase best practices in a second PR.<|||||>Thanks for the update! I was writing my own tokenizer for some special inputs and saw the implementation for the `longest_first` truncation. Is there any reason why tokens are truncated one by one? It seems more efficient to truncate the longer one to the same length as the shorter one, and then truncate the same number of tokens from both of them. In this way, we need only 3 array slices in total, saving a lot of loops. 