---
local: byt5
sections:
- local: overview
  sections:
  - local: example
    title: Example
  title: Overview
- local: transformers.ByT5Tokenizer
  title: ByT5Tokenizer
title: ByT5
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2021 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="byt5">ByT5</h1>

<h2 id="overview">Overview</h2>

The ByT5 model was presented in [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://arxiv.org/abs/2105.13626) by Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir
Kale, Adam Roberts, Colin Raffel.

The abstract from the paper is the following:

*Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Encoding text as a sequence of tokens requires a tokenizer, which is typically created as an independent artifact from
the model. Token-free models that instead operate directly on raw text (bytes or characters) have many benefits: they
can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by
removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token
sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of
operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with
minimal modifications to process byte sequences. We carefully characterize the trade-offs in terms of parameter count,
training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level
counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on
tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of
pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our
experiments.*

This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
found [here](https://github.com/google-research/byt5).

ByT5's architecture is based on the T5v1.1 model, so one can refer to [T5v1.1's documentation page](t5v1.1). They
only differ in how inputs should be prepared for the model, see the code examples below.

Since ByT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.


<h3 id="example">Example</h3>

ByT5 works on raw UTF-8 bytes, so it can be used without a tokenizer:

```python
from transformers import T5ForConditionalGeneration
import torch

model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")

input_ids = torch.tensor([list("Life is like a box of chocolates.".encode("utf-8"))]) + 3  # add 3 for special tokens
labels = (
    torch.tensor([list("La vie est comme une boîte de chocolat.".encode("utf-8"))]) + 3
)  # add 3 for special tokens

loss = model(input_ids, labels=labels).loss  # forward pass
```

For batched inference and training it is however recommended to make use of the tokenizer:

```python
from transformers import T5ForConditionalGeneration, AutoTokenizer

model = T5ForConditionalGeneration.from_pretrained("google/byt5-small")
tokenizer = AutoTokenizer.from_pretrained("google/byt5-small")

model_inputs = tokenizer(
    ["Life is like a box of chocolates.", "Today is Monday."], padding="longest", return_tensors="pt"
)
labels = tokenizer(
    ["La vie est comme une boîte de chocolat.", "Aujourd'hui c'est lundi."], padding="longest", return_tensors="pt"
).input_ids

loss = model(**model_inputs, labels=labels).loss  # forward pass
```

<h2 id="transformers.ByT5Tokenizer">ByT5Tokenizer</h2>

<div class="docstring">

<docstring><name>class transformers.ByT5Tokenizer</name><anchor>transformers.ByT5Tokenizer</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/byt5/tokenization_byt5.py#L28</source><parameters>[{"name": "eos_token", "val": " = '</s>'"}, {"name": "unk_token", "val": " = '<unk>'"}, {"name": "pad_token", "val": " = '<pad>'"}, {"name": "extra_ids", "val": " = 125"}, {"name": "additional_special_tokens", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **eos_token** (`str`, *optional*, defaults to `"&amp;lt;/s>"`) --
  The end of sequence token.

  <Tip>

  When building a sequence using special tokens, this is not the token that is used for the end of sequence.
  The token used is the `sep_token`.

  </Tip>

- **unk_token** (`str`, *optional*, defaults to `"&amp;lt;unk>"`) --
  The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
  token instead.
- **pad_token** (`str`, *optional*, defaults to `"&amp;lt;pad>"`) --
  The token used for padding, for example when batching sequences of different lengths.
- **extra_ids** (`int`, *optional*, defaults to 100) --
  Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
  accessible as "&amp;lt;extra_id_&amp;lcub;%d}>" where "&amp;lcub;%d}" is a number between 0 and extra_ids-1. Extra tokens are
  indexed from the end of the vocabulary up to beginning ("&amp;lt;extra_id_0>" is the last token in the vocabulary
  like in ByT5 preprocessing see
  [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
- **additional_special_tokens** (`List[str]`, *optional*) --
  Additional special tokens used by the tokenizer.</paramsdesc><paramgroups>0</paramgroups></docstring>

Construct a ByT5 tokenizer. ByT5 simply uses raw bytes utf-8 encoding.

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/pr_15592/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.





<div class="docstring">
<docstring><name>build\_inputs\_with\_special\_tokens</name><anchor>transformers.ByT5Tokenizer.build_inputs_with_special_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/byt5/tokenization_byt5.py#L177</source><parameters>[{"name": "token_ids_0", "val": ": typing.List[int]"}, {"name": "token_ids_1", "val": ": typing.Optional[typing.List[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`List[int]`) --
  List of IDs to which the special tokens will be added.
- **token_ids_1** (`List[int]`, *optional*) --
  Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`List[int]`</rettype><retdesc>List of [input IDs](../glossary#input-ids) with the appropriate special tokens.</retdesc></docstring>

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:

- single sequence: `X &amp;lt;/s>`
- pair of sequences: `A &amp;lt;/s> B &amp;lt;/s>`








</div>
<div class="docstring">
<docstring><name>convert\_tokens\_to\_string</name><anchor>transformers.ByT5Tokenizer.convert_tokens_to_string</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/byt5/tokenization_byt5.py#L228</source><parameters>[{"name": "tokens", "val": ""}]</parameters></docstring>
Converts a sequence of tokens (string) in a single string.

</div>
<div class="docstring">
<docstring><name>create\_token\_type\_ids\_from\_sequences</name><anchor>transformers.ByT5Tokenizer.create_token_type_ids_from_sequences</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/byt5/tokenization_byt5.py#L155</source><parameters>[{"name": "token_ids_0", "val": ": typing.List[int]"}, {"name": "token_ids_1", "val": ": typing.Optional[typing.List[int]] = None"}]</parameters><paramsdesc>- **token_ids_0** (`List[int]`) --
  List of IDs.
- **token_ids_1** (`List[int]`, *optional*) --
  Optional second list of IDs for sequence pairs.</paramsdesc><paramgroups>0</paramgroups><rettype>`List[int]`</rettype><retdesc>List of zeros.</retdesc></docstring>

Create a mask from the two sequences passed to be used in a sequence-pair classification task. ByT5 does not
make use of token type ids, therefore a list of zeros is returned.








</div>
<div class="docstring">
<docstring><name>get\_special\_tokens\_mask</name><anchor>transformers.ByT5Tokenizer.get_special_tokens_mask</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/models/byt5/tokenization_byt5.py#L117</source><parameters>[{"name": "token_ids_0", "val": ": typing.List[int]"}, {"name": "token_ids_1", "val": ": typing.Optional[typing.List[int]] = None"}, {"name": "already_has_special_tokens", "val": ": bool = False"}]</parameters><paramsdesc>- **token_ids_0** (`List[int]`) --
  List of IDs.
- **token_ids_1** (`List[int]`, *optional*) --
  Optional second list of IDs for sequence pairs.
- **already_has_special_tokens** (`bool`, *optional*, defaults to `False`) --
  Whether or not the token list is already formatted with special tokens for the model.</paramsdesc><paramgroups>0</paramgroups><rettype>`List[int]`</rettype><retdesc>A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.</retdesc></docstring>

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.








</div></div>

See [ByT5Tokenizer](/docs/transformers/pr_15592/en/model_doc/byt5#transformers.ByT5Tokenizer) for all details.
