---
local: models
sections:
- local: transformers.PreTrainedModel
  sections:
  - local: model-instantiation-dtype
    title: Model Instantiation dtype
  title: PreTrainedModel
- local: transformers.modeling_utils.ModuleUtilsMixin
  title: ModuleUtilsMixin
- local: transformers.TFPreTrainedModel
  title: TFPreTrainedModel
- local: transformers.modeling_tf_utils.TFModelUtilsMixin
  title: TFModelUtilsMixin
- local: transformers.FlaxPreTrainedModel
  title: FlaxPreTrainedModel
- local: transformers.generation_utils.GenerationMixin
  title: Generation
- local: transformers.file_utils.PushToHubMixin
  title: Pushing to the Hub
title: Models
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="models">Models</h1>

The base classes [PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel), [TFPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel), and
[FlaxPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel) implement the common methods for loading/saving a model either from a local
file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace's AWS
S3 repository).

[PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel) and [TFPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel) also implement a few methods which
are common among all the models to:

- resize the input token embeddings when new tokens are added to the vocabulary
- prune the attention heads of the model.

The other methods that are common to each model are defined in [ModuleUtilsMixin](/docs/transformers/pr_15592/en/main_classes/model#transformers.modeling_utils.ModuleUtilsMixin)
(for the PyTorch models) and `TFModuleUtilsMixin` (for the TensorFlow models) or
for text generation, [GenerationMixin](/docs/transformers/pr_15592/en/main_classes/model#transformers.generation_utils.GenerationMixin) (for the PyTorch models),
[TFGenerationMixin](/docs/transformers/pr_15592/en/main_classes/model#transformers.generation_tf_utils.TFGenerationMixin) (for the TensorFlow models) and
[FlaxGenerationMixin](/docs/transformers/pr_15592/en/main_classes/model#transformers.generation_flax_utils.FlaxGenerationMixin) (for the Flax/JAX models).


<h2 id="transformers.PreTrainedModel">PreTrainedModel</h2>

<div class="docstring">

<docstring><name>class transformers.PreTrainedModel</name><anchor>transformers.PreTrainedModel</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L423</source><parameters>[{"name": "config", "val": ": PretrainedConfig"}, {"name": "*inputs", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Base class for all models.

[PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel) takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models as well as a few methods common to all models to:

- resize the input embeddings,
- prune heads in the self-attention heads.

Class attributes (overridden by derived classes):

- **config_class** ([PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig)) -- A subclass of [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) to use as configuration class
  for this model architecture.
- **load_tf_weights** (`Callable`) -- A python *method* for loading a TensorFlow checkpoint in a PyTorch model,
  taking as arguments:

  - **model** ([PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel)) -- An instance of the model on which to load the TensorFlow checkpoint.
  - **config** (`PreTrainedConfig`) -- An instance of the configuration associated to the model.
  - **path** (`str`) -- A path to the TensorFlow checkpoint.

- **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived
  classes of the same architecture adding modules on top of the base model.
- **is_parallelizable** (`bool`) -- A flag indicating whether this model supports model parallelization.
- **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP
  models, `pixel_values` for vision models and `input_values` for speech models).



<div class="docstring">
<docstring><name>push\_to\_hub</name><anchor>transformers.file_utils.PushToHubMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2825</source><parameters>[{"name": "repo_path_or_name", "val": ": typing.Optional[str] = None"}, {"name": "repo_url", "val": ": typing.Optional[str] = None"}, {"name": "use_temp_dir", "val": ": bool = False"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "organization", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "**model_card_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`, *optional*) --
  Can either be a repository name for your model in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder). If not specified, will default to the name
  given by `repo_url` and a local directory with that name will be created.
- **repo_url** (`str`, *optional*) --
  Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
  repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
- **use_temp_dir** (`bool`, *optional*, defaults to `False`) --
  Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
  current working directory. This will slow things down if you are making changes in an existing repo
  since you will need to clone the repo before every push.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"add model"`.
- **organization** (`str`, *optional*) --
  Organization in which you want to push your model (you must be a member of this organization).
- **private** (`bool`, *optional*) --
  Whether or not the repository created should be private (requires a paying subscription).
- **use_auth_token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
  `repo_url` is not specified.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>The url of the commit of your model in the given repository.</retdesc></docstring>

Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.







Examples:

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("bert-base-cased")

# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert")

# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)

# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")

# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```


</div>
<div class="docstring">
<docstring><name>from\_pretrained</name><anchor>transformers.PreTrainedModel.from_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L1090</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ": typing.Union[str, os.PathLike, NoneType]"}, {"name": "*model_args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str` or `os.PathLike`, *optional*) --
  Can be either:

  - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
    Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a
    user or organization name, like `dbmdz/bert-base-german-cased`.
  - A path to a *directory* containing model weights saved using
    [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.save_pretrained), e.g., `./my_model_directory/`.
  - A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In
    this case, `from_tf` should be set to `True` and a configuration object should be provided as
    `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a
    PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards.
  - A path or url to a model folder containing a *flax checkpoint file* in *.msgpack* format (e.g,
    `./flax_model/` containing `flax_model.msgpack`). In this case, `from_flax` should be set to
    `True`.
  - `None` if you are both providing the configuration and state dictionary (resp. with keyword
    arguments `config` and `state_dict`).
- **model_args** (sequence of positional arguments, *optional*) --
  All remaining positional arguments will be passed to the underlying model's `__init__` method.
- **config** (`Union[PretrainedConfig, str, os.PathLike]`, *optional*) --
  Can be either:

  - an instance of a class derived from [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig),
  - a string or path valid as input to [from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained).

Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:

- The model is a model provided by the library (loaded with the *model id* string of a pretrained
  model).
- The model was saved using [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.save_pretrained) and is reloaded by supplying the
  save directory.
- The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a
  configuration JSON file named *config.json* is found in the directory.
- **state_dict** (`Dict[str, torch.Tensor]`, *optional*) --
A state dictionary to use instead of a state dictionary loaded from saved weights file.

This option can be used if you want to create a model from a pretrained configuration but load your own
weights. In this case though, you should check if using [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.save_pretrained) and
[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) is not a simpler option.
  - **cache_dir** (`Union[str, os.PathLike]`, *optional*) --
    Path to a directory in which a downloaded pretrained model configuration should be cached if the
    standard cache should not be used.
- **from_tf** (`bool`, *optional*, defaults to `False`) --
  Load the model weights from a TensorFlow checkpoint save file (see docstring of
  `pretrained_model_name_or_path` argument).
- **from_flax** (`bool`, *optional*, defaults to `False`) --
  Load the model weights from a Flax checkpoint save file (see docstring of
  `pretrained_model_name_or_path` argument).
- **ignore_mismatched_sizes** (`bool`, *optional*, defaults to `False`) --
  Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
  as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
  checkpoint with 3 labels).
- **force_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to force the (re-)download of the model weights and configuration files, overriding the
  cached versions if they exist.
- **resume_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to delete incompletely received files. Will attempt to resume the download if such a
  file exists.
- **proxies** (`Dict[str, str]`, *optional*) --
  A dictionary of proxy servers to use by protocol or endpoint, e.g., `&amp;lcub;'http': 'foo.bar:3128',
  'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
- **output_loading_info(`bool`,** *optional*, defaults to `False`) --
  Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages.
- **local_files_only(`bool`,** *optional*, defaults to `False`) --
  Whether or not to only look at local files (i.e., do not try to download the model).
- **use_auth_token** (`str` or *bool*, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`).
- **revision(`str`,** *optional*, defaults to `"main"`) --
  The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
  git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
  identifier allowed by git.
- **mirror(`str`,** *optional*) --
  Mirror source to accelerate downloads in China. If you are from China and have an accessibility
  problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
  Please refer to the mirror site for more information.
- **_fast_init(`bool`,** *optional*, defaults to ```True`) --
  Whether or not to disable fast initialization.
- **low_cpu_mem_usage(`bool``,** *optional*, defaults to ```False`) --
  Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model.
  This is an experimental feature and a subject to change at any moment.
- **torch_dtype** (`str` or `torch.dtype`, *optional*) --
  Override the default `torch.dtype` and load the model under this dtype. If `"auto"` is passed the dtype
  will be automatically derived from the model's weights.

  <Tip warning=&amp;lcub;true}>

  One should only disable *_fast_init* to ensure backwards compatibility with `transformers.__version__ &amp;lt;
  4.6.0` for seeded model initialization. This argument will be removed at the next major version. See
  [pull request 11471](https://github.com/huggingface/transformers/pull/11471) for more information.

  </Tip>

- **kwargs** (remaining dictionary of keyword arguments, *optional*) --
  Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
  `output_attentions=True`). Behaves differently depending on whether a `config` is provided or
  automatically loaded:

  - If a configuration is provided with `config`, `**kwargs` will be directly passed to the
    underlying model's `__init__` method (we assume all relevant updates to the configuration have
    already been done)
  - If a configuration is not provided, `kwargs` will be first passed to the configuration class
    initialization function ([from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained)). Each key of `kwargs` that
    corresponds to a configuration attribute will be used to override said attribute with the
    supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute
    will be passed to the underlying model's `__init__` function.</paramsdesc><paramgroups>0</paramgroups></docstring>

Instantiate a pretrained pytorch model from a pre-trained model configuration.

The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train
the model, you should first set it back in training mode with `model.train()`.

The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task.

The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
weights are discarded.



<Tip>

Passing `use_auth_token=True`` is required when you want to use a private model.

</Tip>

<Tip>

Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to
use this method in a firewalled environment.

</Tip>

Examples:

```python
>>> from transformers import BertConfig, BertModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = BertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = BertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> model = BertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./tf_model/my_tf_model_config.json")
>>> model = BertModel.from_pretrained("./tf_model/my_tf_checkpoint.ckpt.index", from_tf=True, config=config)
>>> # Loading from a Flax checkpoint file instead of a PyTorch model (slower)
>>> model = BertModel.from_pretrained("bert-base-uncased", from_flax=True)
```

</div>
<div class="docstring">
<docstring><name>get\_input\_embeddings</name><anchor>transformers.PreTrainedModel.get_input_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L575</source><parameters>[]</parameters><rettype>`nn.Module`</rettype><retdesc>A torch module mapping vocabulary to hidden states.</retdesc></docstring>

Returns the model's input embeddings.






</div>
<div class="docstring">
<docstring><name>get\_output\_embeddings</name><anchor>transformers.PreTrainedModel.get_output_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L601</source><parameters>[]</parameters><rettype>`nn.Module`</rettype><retdesc>A torch module mapping hidden states to vocabulary.</retdesc></docstring>

Returns the model's output embeddings.






</div>
<div class="docstring">
<docstring><name>gradient\_checkpointing\_disable</name><anchor>transformers.PreTrainedModel.gradient_checkpointing_disable</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L979</source><parameters>[]</parameters></docstring>

Deactivates gradient checkpointing for the current model.

Note that in other frameworks this feature can be referred to as "activation checkpointing" or "checkpoint
activations".


</div>
<div class="docstring">
<docstring><name>gradient\_checkpointing\_enable</name><anchor>transformers.PreTrainedModel.gradient_checkpointing_enable</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L968</source><parameters>[]</parameters></docstring>

Activates gradient checkpointing for the current model.

Note that in other frameworks this feature can be referred to as "activation checkpointing" or "checkpoint
activations".


</div>
<div class="docstring">
<docstring><name>init\_weights</name><anchor>transformers.PreTrainedModel.init_weights</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L935</source><parameters>[]</parameters></docstring>

If needed prunes and maybe initializes weights.


</div>
<div class="docstring">
<docstring><name>post\_init</name><anchor>transformers.PreTrainedModel.post_init</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L494</source><parameters>[]</parameters></docstring>

A method executed at the end of each Transformer model initialization, to execute code that needs the model's
modules properly initialized (such as weight initialization).


</div>
<div class="docstring">
<docstring><name>prune\_heads</name><anchor>transformers.PreTrainedModel.prune_heads</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L951</source><parameters>[{"name": "heads_to_prune", "val": ": typing.Dict[int, typing.List[int]]"}]</parameters><paramsdesc>- **heads_to_prune** (`Dict[int, List[int]]`) --
  Dictionary with keys being selected layer indices (`int`) and associated values being the list of heads
  to prune in said layer (list of `int`). For instance &amp;lcub;1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on
  layer 1 and heads 2 and 3 on layer 2.</paramsdesc><paramgroups>0</paramgroups></docstring>

Prunes heads of the base model.




</div>
<div class="docstring">
<docstring><name>register\_for\_auto\_class</name><anchor>transformers.PreTrainedModel.register_for_auto_class</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L1812</source><parameters>[{"name": "auto_class", "val": " = 'AutoModel'"}]</parameters><paramsdesc>- **auto_class** (`str` or `type`, *optional*, defaults to `"AutoModel"`) --
  The auto class to register this new model with.</paramsdesc><paramgroups>0</paramgroups></docstring>

Register this class with a given auto class. This should only be used for custom models as the ones in the
library are already mapped with an auto class.

<Tip warning=&amp;lcub;true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>




</div>
<div class="docstring">
<docstring><name>resize\_token\_embeddings</name><anchor>transformers.PreTrainedModel.resize_token_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L729</source><parameters>[{"name": "new_num_tokens", "val": ": typing.Optional[int] = None"}]</parameters><paramsdesc>- **new_num_tokens** (`int`, *optional*) --
  The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
  vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
  returns a pointer to the input tokens `torch.nn.Embedding` module of the model without doing anything.</paramsdesc><paramgroups>0</paramgroups><rettype>`torch.nn.Embedding`</rettype><retdesc>Pointer to the input tokens Embeddings Module of the model.</retdesc></docstring>

Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.








</div>
<div class="docstring">
<docstring><name>save\_pretrained</name><anchor>transformers.PreTrainedModel.save_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L999</source><parameters>[{"name": "save_directory", "val": ": typing.Union[str, os.PathLike]"}, {"name": "save_config", "val": ": bool = True"}, {"name": "state_dict", "val": ": typing.Optional[dict] = None"}, {"name": "save_function", "val": ": typing.Callable = <function save at 0x7f116a072d30>"}, {"name": "push_to_hub", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **save_directory** (`str` or `os.PathLike`) --
  Directory to which to save. Will be created if it doesn't exist.
- **save_config** (`bool`, *optional*, defaults to `True`) --
  Whether or not to save the config of the model. Useful when in distributed training like TPUs and need
  to call this function on all processes. In this case, set `save_config=True` only on the main process
  to avoid race conditions.
- **state_dict** (nested dictionary of `torch.Tensor`) --
  The state dictionary of the model to save. Will default to `self.state_dict()`, but can be used to only
  save parts of the model or if special precautions need to be taken when recovering the state dictionary
  of a model (like when using model parallelism).
- **save_function** (`Callable`) --
  The function to use to save the state dictionary. Useful on distributed training like TPUs when one
  need to replace `torch.save` by another method.
- **push_to_hub** (`bool`, *optional*, defaults to `False`) --
  Whether or not to push your model to the Hugging Face model hub after saving it.

  <Tip warning=&amp;lcub;true}>

  Using `push_to_hub=True` will synchronize the repository you are pushing to with `save_directory`,
  which requires `save_directory` to be a local clone of the repo you are pushing to if it's an existing
  folder. Pass along `temp_dir=True` to use a temporary directory instead.

  </Tip>

  kwargs --
  Additional key word arguments passed along to the [push_to_hub()](/docs/transformers/pr_15592/en/main_classes/model#transformers.file_utils.PushToHubMixin.push_to_hub) method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Save a model and its configuration file to a directory, so that it can be re-loaded using the
`[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.from_pretrained)` class method.




</div>
<div class="docstring">
<docstring><name>set\_input\_embeddings</name><anchor>transformers.PreTrainedModel.set_input_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L588</source><parameters>[{"name": "value", "val": ": Module"}]</parameters><paramsdesc>- **value** (`nn.Module`) -- A module mapping vocabulary to hidden states.</paramsdesc><paramgroups>0</paramgroups></docstring>

Set model's input embeddings.




</div>
<div class="docstring">
<docstring><name>tie\_weights</name><anchor>transformers.PreTrainedModel.tie_weights</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L616</source><parameters>[]</parameters></docstring>

Tie the weights between the input embeddings and the output embeddings.

If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the
weights instead.


</div></div>

<a id='from_pretrained-torch-dtype'></a>

<h3 id="model-instantiation-dtype">Model Instantiation dtype</h3>

Under Pytorch a model normally gets instantiated with `torch.float32` format. This can be an issue if one tries to
load a model whose weights are in fp16, since it'd require twice as much memory. To overcome this limitation, you can
either explicitly pass the desired `dtype` using `torch_dtype` argument:

```python
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype=torch.float16)
```

or, if you want the model to always load in the most optimal memory pattern, you can use the special value `"auto"`,
and then `dtype` will be automatically derived from the model's weights:

```python
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype="auto")
```

Models instantiated from scratch can also be told which `dtype` to use with:

```python
config = T5Config.from_pretrained("t5")
model = AutoModel.from_config(config)
```

Due to Pytorch design, this functionality is only available for floating dtypes.



<h2 id="transformers.modeling_utils.ModuleUtilsMixin">ModuleUtilsMixin</h2>

<div class="docstring">

<docstring><name>class transformers.modeling\_utils.ModuleUtilsMixin</name><anchor>transformers.modeling_utils.ModuleUtilsMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L151</source><parameters>[]</parameters></docstring>

A few utilities for `torch.nn.Modules`, to be used as a mixin.



<div class="docstring">
<docstring><name>add\_memory\_hooks</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.add_memory_hooks</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L182</source><parameters>[]</parameters></docstring>

Add a memory hook before and after each sub-module forward pass to record increase in memory consumption.

Increase in memory consumption is stored in a `mem_rss_diff` attribute for each module and can be reset to zero
with `model.reset_memory_hooks_state()`.


</div>
<div class="docstring">
<docstring><name>estimate\_tokens</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.estimate_tokens</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L378</source><parameters>[{"name": "input_dict", "val": ": typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]"}]</parameters><paramsdesc>- **inputs** (`dict`) -- The model inputs.</paramsdesc><paramgroups>0</paramgroups><rettype>`int`</rettype><retdesc>The total number of tokens.</retdesc></docstring>

Helper function to estimate the total number of tokens from the model inputs.








</div>
<div class="docstring">
<docstring><name>floating\_point\_ops</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.floating_point_ops</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L396</source><parameters>[{"name": "input_dict", "val": ": typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]"}, {"name": "exclude_embeddings", "val": ": bool = True"}]</parameters><paramsdesc>- **batch_size** (`int`) --
  The batch size for the forward pass.

- **sequence_length** (`int`) --
  The number of tokens in each line of the batch.

- **exclude_embeddings** (`bool`, *optional*, defaults to `True`) --
  Whether or not to count embedding and softmax operations.</paramsdesc><paramgroups>0</paramgroups><rettype>`int`</rettype><retdesc>The number of floating-point operations.</retdesc></docstring>

Get number of (optionally, non-embeddings) floating-point operations for the forward and backward passes of a
batch with this transformer model. Default approximation neglects the quadratic dependency on the number of
tokens (valid if `12 * d_model << sequence_length`) as laid out in [this
paper](https://arxiv.org/pdf/2001.08361.pdf) section 2.1. Should be overridden for transformers with parameter
re-use e.g. Albert or Universal Transformers, or if doing long-range modeling with very high sequence lengths.








</div>
<div class="docstring">
<docstring><name>get\_extended\_attention\_mask</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.get_extended_attention_mask</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L271</source><parameters>[{"name": "attention_mask", "val": ": Tensor"}, {"name": "input_shape", "val": ": typing.Tuple[int]"}, {"name": "device", "val": ": <property object at 0x7f1101d340e0>"}]</parameters><paramsdesc>- **attention_mask** (`torch.Tensor`) --
  Mask with ones indicating tokens to attend to, zeros for tokens to ignore.
- **input_shape** (`Tuple[int]`) --
  The shape of the input to the model.
  device -- (`torch.device`):
  The device of the input to the model.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`.</retdesc></docstring>

Makes broadcastable attention and causal masks so that future and masked tokens are ignored.






</div>
<div class="docstring">
<docstring><name>get\_head\_mask</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.get_head_mask</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L314</source><parameters>[{"name": "head_mask", "val": ": typing.Optional[torch.Tensor]"}, {"name": "num_hidden_layers", "val": ": int"}, {"name": "is_attention_chunked", "val": ": bool = False"}]</parameters><paramsdesc>- **head_mask** (`torch.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_heads]`, *optional*) --
  The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard).
- **num_hidden_layers** (`int`) --
  The number of hidden layers in the model.
  is_attention_chunked -- (`bool`, *optional*, defaults to `False`):
  Whether or not the attentions scores are computed by chunks or not.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.Tensor` with shape `[num_hidden_layers x batch x num_heads x seq_length x seq_length]` or list with
`[None]` for each layer.</retdesc></docstring>

Prepare the head mask if needed.






</div>
<div class="docstring">
<docstring><name>invert\_attention\_mask</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.invert_attention_mask</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L218</source><parameters>[{"name": "encoder_attention_mask", "val": ": Tensor"}]</parameters><paramsdesc>- **encoder_attention_mask** (`torch.Tensor`) -- An attention mask.</paramsdesc><paramgroups>0</paramgroups><rettype>`torch.Tensor`</rettype><retdesc>The inverted attention mask.</retdesc></docstring>

Invert an attention mask (e.g., switches 0. and 1.).








</div>
<div class="docstring">
<docstring><name>num\_parameters</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.num_parameters</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L352</source><parameters>[{"name": "only_trainable", "val": ": bool = False"}, {"name": "exclude_embeddings", "val": ": bool = False"}]</parameters><paramsdesc>- **only_trainable** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return only the number of trainable parameters

- **exclude_embeddings** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return only the number of non-embeddings parameters</paramsdesc><paramgroups>0</paramgroups><rettype>`int`</rettype><retdesc>The number of parameters.</retdesc></docstring>

Get number of (optionally, trainable or non-embeddings) parameters in the module.








</div>
<div class="docstring">
<docstring><name>reset\_memory\_hooks\_state</name><anchor>transformers.modeling_utils.ModuleUtilsMixin.reset_memory_hooks_state</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_utils.py#L194</source><parameters>[]</parameters></docstring>

Reset the `mem_rss_diff` attribute of each module (see [add_memory_hooks()](/docs/transformers/pr_15592/en/main_classes/model#transformers.modeling_utils.ModuleUtilsMixin.add_memory_hooks)).


</div></div>

<h2 id="transformers.TFPreTrainedModel">TFPreTrainedModel</h2>

<div class="docstring">

<docstring><name>class transformers.TFPreTrainedModel</name><anchor>transformers.TFPreTrainedModel</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L645</source><parameters>[{"name": "*args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Base class for all TF models.

[TFPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel) takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models as well as a few methods common to all models to:

- resize the input embeddings,
- prune heads in the self-attention heads.

Class attributes (overridden by derived classes):

- **config_class** ([PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig)) -- A subclass of [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) to use as configuration class
  for this model architecture.
- **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived
  classes of the same architecture adding modules on top of the base model.
- **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP
  models, `pixel_values` for vision models and `input_values` for speech models).



<div class="docstring">
<docstring><name>push\_to\_hub</name><anchor>transformers.file_utils.PushToHubMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2825</source><parameters>[{"name": "repo_path_or_name", "val": ": typing.Optional[str] = None"}, {"name": "repo_url", "val": ": typing.Optional[str] = None"}, {"name": "use_temp_dir", "val": ": bool = False"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "organization", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "**model_card_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`, *optional*) --
  Can either be a repository name for your model in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder). If not specified, will default to the name
  given by `repo_url` and a local directory with that name will be created.
- **repo_url** (`str`, *optional*) --
  Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
  repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
- **use_temp_dir** (`bool`, *optional*, defaults to `False`) --
  Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
  current working directory. This will slow things down if you are making changes in an existing repo
  since you will need to clone the repo before every push.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"add model"`.
- **organization** (`str`, *optional*) --
  Organization in which you want to push your model (you must be a member of this organization).
- **private** (`bool`, *optional*) --
  Whether or not the repository created should be private (requires a paying subscription).
- **use_auth_token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
  `repo_url` is not specified.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>The url of the commit of your model in the given repository.</retdesc></docstring>

Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.







Examples:

```python
from transformers import TFAutoModel

model = TFAutoModel.from_pretrained("bert-base-cased")

# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert")

# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)

# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")

# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```


</div>
<div class="docstring">
<docstring><name>compile</name><anchor>transformers.TFPreTrainedModel.compile</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L836</source><parameters>[{"name": "optimizer", "val": " = 'rmsprop'"}, {"name": "loss", "val": " = 'passthrough'"}, {"name": "metrics", "val": " = None"}, {"name": "loss_weights", "val": " = None"}, {"name": "weighted_metrics", "val": " = None"}, {"name": "run_eagerly", "val": " = None"}, {"name": "steps_per_execution", "val": " = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

This is a thin wrapper that sets the model's loss output head as the loss if the user does not specify a loss
function themselves.


</div>
<div class="docstring">
<docstring><name>from\_pretrained</name><anchor>transformers.TFPreTrainedModel.from_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1388</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ""}, {"name": "*model_args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str`, *optional*) --
  Can be either:

  - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
    Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a
    user or organization name, like `dbmdz/bert-base-german-cased`.
  - A path to a *directory* containing model weights saved using
    [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel.save_pretrained), e.g., `./my_model_directory/`.
  - A path or url to a *PyTorch state_dict save file* (e.g, `./pt_model/pytorch_model.bin`). In this
    case, `from_pt` should be set to `True` and a configuration object should be provided as `config`
    argument. This loading path is slower than converting the PyTorch model in a TensorFlow model
    using the provided conversion scripts and loading the TensorFlow model afterwards.
  - `None` if you are both providing the configuration and state dictionary (resp. with keyword
    arguments `config` and `state_dict`).
- **model_args** (sequence of positional arguments, *optional*) --
  All remaining positional arguments will be passed to the underlying model's `__init__` method.
- **config** (`Union[PretrainedConfig, str]`, *optional*) --
  Can be either:

  - an instance of a class derived from [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig),
  - a string valid as input to [from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained).

Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:

- The model is a model provided by the library (loaded with the *model id* string of a pretrained
  model).
- The model was saved using [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel.save_pretrained) and is reloaded by supplying the
  save directory.
- The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a
  configuration JSON file named *config.json* is found in the directory.
from_pt -- (`bool`, *optional*, defaults to `False`):
Load the model weights from a PyTorch state_dict save file (see docstring of
`pretrained_model_name_or_path` argument).
- **ignore_mismatched_sizes** (`bool`, *optional*, defaults to `False`) --
  Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
  as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
  checkpoint with 3 labels).
- **cache_dir** (`str`, *optional*) --
  Path to a directory in which a downloaded pretrained model configuration should be cached if the
  standard cache should not be used.
- **force_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to force the (re-)download of the model weights and configuration files, overriding the
  cached versions if they exist.
- **resume_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to delete incompletely received files. Will attempt to resume the download if such a
  file exists.
  proxies --
  (`Dict[str, str], `optional`): A dictionary of proxy servers to use by protocol or endpoint, e.g.,
  `&amp;lcub;'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
  output_loading_info(`bool`, *optional*, defaults to `False`): Whether ot not to also return a
  dictionary containing missing keys, unexpected keys and error messages.
- **local_files_only(`bool`,** *optional*, defaults to `False`) --
  Whether or not to only look at local files (e.g., not try doanloading the model).
- **use_auth_token** (`str` or *bool*, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`).
- **revision(`str`,** *optional*, defaults to `"main"`) --
  The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
  git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
  identifier allowed by git.
- **mirror(`str`,** *optional*) --
  Mirror source to accelerate downloads in China. If you are from China and have an accessibility
  problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.
  Please refer to the mirror site for more information.
- **kwargs** (remaining dictionary of keyword arguments, *optional*) --
  Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
  `output_attentions=True`). Behaves differently depending on whether a `config` is provided or
  automatically loaded:

  - If a configuration is provided with `config`, `**kwargs` will be directly passed to the
    underlying model's `__init__` method (we assume all relevant updates to the configuration have
    already been done)
  - If a configuration is not provided, `kwargs` will be first passed to the configuration class
    initialization function ([from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained)). Each key of `kwargs` that
    corresponds to a configuration attribute will be used to override said attribute with the
    supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute
    will be passed to the underlying model's `__init__` function.</paramsdesc><paramgroups>0</paramgroups></docstring>

Instantiate a pretrained TF 2.0 model from a pre-trained model configuration.

The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task.

The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
weights are discarded.



<Tip>

Passing `use_auth_token=True` is required when you want to use a private model.

</Tip>

Examples:

```python
>>> from transformers import BertConfig, TFBertModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFBertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = TFBertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> model = TFBertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/my_pt_model_config.json")
>>> model = TFBertModel.from_pretrained("./pt_model/my_pytorch_model.bin", from_pt=True, config=config)
```

</div>
<div class="docstring">
<docstring><name>get\_bias</name><anchor>transformers.TFPreTrainedModel.get_bias</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1072</source><parameters>[]</parameters><rettype>`tf.Variable`</rettype><retdesc>The weights representing the bias, None if not an LM model.</retdesc></docstring>

Dict of bias attached to an LM head. The key represents the name of the bias attribute.






</div>
<div class="docstring">
<docstring><name>get\_input\_embeddings</name><anchor>transformers.TFPreTrainedModel.get_input_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L755</source><parameters>[]</parameters><rettype>`tf.Variable`</rettype><retdesc>The embeddings layer mapping vocabulary to hidden states.</retdesc></docstring>

Returns the model's input embeddings layer.






</div>
<div class="docstring">
<docstring><name>get\_lm\_head</name><anchor>transformers.TFPreTrainedModel.get_lm_head</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1105</source><parameters>[]</parameters><rettype>`tf.keras.layers.Layer`</rettype><retdesc>The LM head layer if the model has one, None if not.</retdesc></docstring>

The LM Head layer. This method must be overwritten by all the models that have a lm head.






</div>
<div class="docstring">
<docstring><name>get\_output\_embeddings</name><anchor>transformers.TFPreTrainedModel.get_output_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1012</source><parameters>[]</parameters><rettype>`tf.Variable`</rettype><retdesc>The new weights mapping vocabulary to hidden states.</retdesc></docstring>

Returns the model's output embeddings






</div>
<div class="docstring">
<docstring><name>get\_output\_layer\_with\_bias</name><anchor>transformers.TFPreTrainedModel.get_output_layer_with_bias</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1049</source><parameters>[]</parameters><rettype>`tf.keras.layers.Layer`</rettype><retdesc>The layer that handles the bias, None if not an LM model.</retdesc></docstring>

Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the
embeddings






</div>
<div class="docstring">
<docstring><name>get\_prefix\_bias\_name</name><anchor>transformers.TFPreTrainedModel.get_prefix_bias_name</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1062</source><parameters>[]</parameters><rettype>`str`</rettype><retdesc>The _prefix name of the bias.</retdesc></docstring>

Get the concatenated _prefix name of the bias from the model name to the parent layer






</div>
<div class="docstring">
<docstring><name>load\_repo\_checkpoint</name><anchor>transformers.TFPreTrainedModel.load_repo_checkpoint</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L782</source><parameters>[{"name": "repo_path_or_name", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`) --
  Can either be a repository name for your &amp;lcub;object} in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder).</paramsdesc><paramgroups>0</paramgroups><rettype>`dict`</rettype><retdesc>A dictionary of extra metadata from the checkpoint, most commonly an "epoch" count.</retdesc></docstring>

Loads a saved checkpoint (model weights and optimizer state) from a repo. Returns the current epoch count when
the checkpoint was made.








</div>
<div class="docstring">
<docstring><name>prune\_heads</name><anchor>transformers.TFPreTrainedModel.prune_heads</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1314</source><parameters>[{"name": "heads_to_prune", "val": ""}]</parameters><paramsdesc>- **heads_to_prune** (`Dict[int, List[int]]`) --
  Dictionary with keys being selected layer indices (`int`) and associated values being the list of heads
  to prune in said layer (list of `int`). For instance &amp;lcub;1: [0, 2], 2: [2, 3]} will prune heads 0 and 2 on
  layer 1 and heads 2 and 3 on layer 2.</paramsdesc><paramgroups>0</paramgroups></docstring>

Prunes heads of the base model.




</div>
<div class="docstring">
<docstring><name>resize\_token\_embeddings</name><anchor>transformers.TFPreTrainedModel.resize_token_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1114</source><parameters>[{"name": "new_num_tokens", "val": " = None"}]</parameters><paramsdesc>- **new_num_tokens** (`int`, *optional*) --
  The number of new tokens in the embedding matrix. Increasing the size will add newly initialized
  vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just
  returns a pointer to the input tokens `tf.Variable` module of the model without doing anything.</paramsdesc><paramgroups>0</paramgroups><rettype>`tf.Variable`</rettype><retdesc>Pointer to the input tokens Embeddings Module of the model.</retdesc></docstring>

Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`.

Takes care of tying weights embeddings afterwards if the model class has a `tie_weights()` method.








</div>
<div class="docstring">
<docstring><name>save\_pretrained</name><anchor>transformers.TFPreTrainedModel.save_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1326</source><parameters>[{"name": "save_directory", "val": ""}, {"name": "saved_model", "val": " = False"}, {"name": "version", "val": " = 1"}, {"name": "push_to_hub", "val": " = False"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **save_directory** (`str`) --
  Directory to which to save. Will be created if it doesn't exist.
- **saved_model** (`bool`, *optional*, defaults to `False`) --
  If the model has to be saved in saved model format as well or not.
- **version** (`int`, *optional*, defaults to 1) --
  The version of the saved model. A saved model needs to be versioned in order to be properly loaded by
  TensorFlow Serving as detailed in the official documentation
  https://www.tensorflow.org/tfx/serving/serving_basic
- **push_to_hub** (`bool`, *optional*, defaults to `False`) --
  Whether or not to push your model to the Hugging Face model hub after saving it.

  <Tip warning=&amp;lcub;true}>

  Using `push_to_hub=True` will synchronize the repository you are pushing to with `save_directory`,
  which requires `save_directory` to be a local clone of the repo you are pushing to if it's an existing
  folder. Pass along `temp_dir=True` to use a temporary directory instead.

  </Tip>

  kwargs --
  Additional key word arguments passed along to the [push_to_hub()](/docs/transformers/pr_15592/en/main_classes/model#transformers.file_utils.PushToHubMixin.push_to_hub) method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel.from_pretrained) class method.




</div>
<div class="docstring">
<docstring><name>serving</name><anchor>transformers.TFPreTrainedModel.serving</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L724</source><parameters>[{"name": "inputs", "val": ""}]</parameters><paramsdesc>- **inputs** (`Dict[str, tf.Tensor]`) --
  The input of the saved model as a dictionary of tensors.</paramsdesc><paramgroups>0</paramgroups></docstring>

Method used for serving the model.




</div>
<div class="docstring">
<docstring><name>serving\_output</name><anchor>transformers.TFPreTrainedModel.serving_output</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L745</source><parameters>[{"name": "output", "val": ""}]</parameters><paramsdesc>- **output** (`TFBaseModelOutput`) --
  The output returned by the model.</paramsdesc><paramgroups>0</paramgroups></docstring>

Prepare the output of the saved model. Each model must implement this function.




</div>
<div class="docstring">
<docstring><name>set\_bias</name><anchor>transformers.TFPreTrainedModel.set_bias</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1089</source><parameters>[{"name": "value", "val": ""}]</parameters><paramsdesc>- **value** (`Dict[tf.Variable]`) --
  All the new bias attached to an LM head.</paramsdesc><paramgroups>0</paramgroups></docstring>

Set all the bias in the LM head.




</div>
<div class="docstring">
<docstring><name>set\_input\_embeddings</name><anchor>transformers.TFPreTrainedModel.set_input_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L992</source><parameters>[{"name": "value", "val": ""}]</parameters><paramsdesc>- **value** (`tf.Variable`) --
  The new weights mapping hidden states to vocabulary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Set model's input embeddings




</div>
<div class="docstring">
<docstring><name>set\_output\_embeddings</name><anchor>transformers.TFPreTrainedModel.set_output_embeddings</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L1032</source><parameters>[{"name": "value", "val": ""}]</parameters><paramsdesc>- **value** (`tf.Variable`) --
  The new weights mapping hidden states to vocabulary.</paramsdesc><paramgroups>0</paramgroups></docstring>

Set model's output embeddings




</div>
<div class="docstring">
<docstring><name>test\_step</name><anchor>transformers.TFPreTrainedModel.test_step</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L926</source><parameters>[{"name": "data", "val": ""}]</parameters></docstring>

A modification of Keras's default test_step that cleans up the printed metrics when we use a dummy loss.


</div>
<div class="docstring">
<docstring><name>train\_step</name><anchor>transformers.TFPreTrainedModel.train_step</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L885</source><parameters>[{"name": "data", "val": ""}]</parameters></docstring>

A modification of Keras's default train_step that cleans up the printed metrics when we use a dummy loss.


</div></div>

<h2 id="transformers.modeling_tf_utils.TFModelUtilsMixin">TFModelUtilsMixin</h2>

<div class="docstring">

<docstring><name>class transformers.modeling\_tf\_utils.TFModelUtilsMixin</name><anchor>transformers.modeling_tf_utils.TFModelUtilsMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L83</source><parameters>[]</parameters></docstring>

A few utilities for `tf.keras.Model`, to be used as a mixin.



<div class="docstring">
<docstring><name>num\_parameters</name><anchor>transformers.modeling_tf_utils.TFModelUtilsMixin.num_parameters</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_tf_utils.py#L88</source><parameters>[{"name": "only_trainable", "val": ": bool = False"}]</parameters><paramsdesc>- **only_trainable** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return only the number of trainable parameters</paramsdesc><paramgroups>0</paramgroups><rettype>`int`</rettype><retdesc>The number of parameters.</retdesc></docstring>

Get the number of (optionally, trainable) parameters in the model.








</div></div>

<h2 id="transformers.FlaxPreTrainedModel">FlaxPreTrainedModel</h2>

<div class="docstring">

<docstring><name>class transformers.FlaxPreTrainedModel</name><anchor>transformers.FlaxPreTrainedModel</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L72</source><parameters>[{"name": "config", "val": ": PretrainedConfig"}, {"name": "module", "val": ": Module"}, {"name": "input_shape", "val": ": typing.Tuple = (1, 1)"}, {"name": "seed", "val": ": int = 0"}, {"name": "dtype", "val": ": dtype = <class 'jax._src.numpy.lax_numpy.float32'>"}]</parameters></docstring>

Base class for all models.

[FlaxPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel) takes care of storing the configuration of the models and handles methods for loading,
downloading and saving models.

Class attributes (overridden by derived classes):

- **config_class** ([PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig)) -- A subclass of [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) to use as configuration class
  for this model architecture.
- **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived
  classes of the same architecture adding modules on top of the base model.
- **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP
  models, `pixel_values` for vision models and `input_values` for speech models).



<div class="docstring">
<docstring><name>push\_to\_hub</name><anchor>transformers.file_utils.PushToHubMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2825</source><parameters>[{"name": "repo_path_or_name", "val": ": typing.Optional[str] = None"}, {"name": "repo_url", "val": ": typing.Optional[str] = None"}, {"name": "use_temp_dir", "val": ": bool = False"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "organization", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "**model_card_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`, *optional*) --
  Can either be a repository name for your model in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder). If not specified, will default to the name
  given by `repo_url` and a local directory with that name will be created.
- **repo_url** (`str`, *optional*) --
  Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
  repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
- **use_temp_dir** (`bool`, *optional*, defaults to `False`) --
  Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
  current working directory. This will slow things down if you are making changes in an existing repo
  since you will need to clone the repo before every push.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"add model"`.
- **organization** (`str`, *optional*) --
  Organization in which you want to push your model (you must be a member of this organization).
- **private** (`bool`, *optional*) --
  Whether or not the repository created should be private (requires a paying subscription).
- **use_auth_token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
  `repo_url` is not specified.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>The url of the commit of your model in the given repository.</retdesc></docstring>

Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.







Examples:

```python
from transformers import FlaxAutoModel

model = FlaxAutoModel.from_pretrained("bert-base-cased")

# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert")

# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)

# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")

# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```


</div>
<div class="docstring">
<docstring><name>from\_pretrained</name><anchor>transformers.FlaxPreTrainedModel.from_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L296</source><parameters>[{"name": "pretrained_model_name_or_path", "val": ": typing.Union[str, os.PathLike]"}, {"name": "dtype", "val": ": dtype = <class 'jax._src.numpy.lax_numpy.float32'>"}, {"name": "*model_args", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **pretrained_model_name_or_path** (`str` or `os.PathLike`) --
  Can be either:

  - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.
    Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a
    user or organization name, like `dbmdz/bert-base-german-cased`.
  - A path to a *directory* containing model weights saved using
    [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel.save_pretrained), e.g., `./my_model_directory/`.
  - A path or url to a *pt index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case,
    `from_pt` should be set to `True`.
- **dtype** (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`) --
  The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and
  `jax.numpy.bfloat16` (on TPUs).

  This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If
  specified all the computation will be performed with the given `dtype`.

  **Note that this only specifies the dtype of the computation and does not influence the dtype of model
  parameters.**

  If you wish to change the dtype of the model parameters, see [to_fp16()](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel.to_fp16) and
  [to_bf16()](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel.to_bf16).
- **model_args** (sequence of positional arguments, *optional*) --
  All remaining positional arguments will be passed to the underlying model's `__init__` method.
- **config** (`Union[PretrainedConfig, str, os.PathLike]`, *optional*) --
  Can be either:

  - an instance of a class derived from [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig),
  - a string or path valid as input to [from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained).

Configuration for the model to use instead of an automatically loaded configuration. Configuration can
be automatically loaded when:

- The model is a model provided by the library (loaded with the *model id* string of a pretrained
  model).
- The model was saved using [save_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel.save_pretrained) and is reloaded by supplying the
  save directory.
- The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a
  configuration JSON file named *config.json* is found in the directory.
- **cache_dir** (`Union[str, os.PathLike]`, *optional*) --
Path to a directory in which a downloaded pretrained model configuration should be cached if the
standard cache should not be used.
  - **from_pt** (`bool`, *optional*, defaults to `False`) --
    Load the model weights from a PyTorch checkpoint save file (see docstring of
    `pretrained_model_name_or_path` argument).
- **ignore_mismatched_sizes** (`bool`, *optional*, defaults to `False`) --
  Whether or not to raise an error if some of the weights from the checkpoint do not have the same size
  as the weights of the model (if for instance, you are instantiating a model with 10 labels from a
  checkpoint with 3 labels).
- **force_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to force the (re-)download of the model weights and configuration files, overriding the
  cached versions if they exist.
- **resume_download** (`bool`, *optional*, defaults to `False`) --
  Whether or not to delete incompletely received files. Will attempt to resume the download if such a
  file exists.
- **proxies** (`Dict[str, str]`, *optional*) --
  A dictionary of proxy servers to use by protocol or endpoint, e.g., `&amp;lcub;'http': 'foo.bar:3128',
  'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
- **local_files_only(`bool`,** *optional*, defaults to `False`) --
  Whether or not to only look at local files (i.e., do not try to download the model).
- **revision(`str`,** *optional*, defaults to `"main"`) --
  The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
  git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
  identifier allowed by git.
- **kwargs** (remaining dictionary of keyword arguments, *optional*) --
  Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
  `output_attentions=True`). Behaves differently depending on whether a `config` is provided or
  automatically loaded:

  - If a configuration is provided with `config`, `**kwargs` will be directly passed to the
    underlying model's `__init__` method (we assume all relevant updates to the configuration have
    already been done)
  - If a configuration is not provided, `kwargs` will be first passed to the configuration class
    initialization function ([from_pretrained()](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained)). Each key of `kwargs` that
    corresponds to a configuration attribute will be used to override said attribute with the
    supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute
    will be passed to the underlying model's `__init__` function.</paramsdesc><paramgroups>0</paramgroups></docstring>

Instantiate a pretrained flax model from a pre-trained model configuration.

The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come
pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning
task.

The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those
weights are discarded.



Examples:

```python
>>> from transformers import BertConfig, FlaxBertModel

>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = FlaxBertModel.from_pretrained("./test/saved_model/")
>>> # Loading from a PyTorch checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/config.json")
>>> model = FlaxBertModel.from_pretrained("./pt_model/pytorch_model.bin", from_pt=True, config=config)
```

</div>
<div class="docstring">
<docstring><name>register\_for\_auto\_class</name><anchor>transformers.FlaxPreTrainedModel.register_for_auto_class</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L716</source><parameters>[{"name": "auto_class", "val": " = 'FlaxAutoModel'"}]</parameters><paramsdesc>- **auto_class** (`str` or `type`, *optional*, defaults to `"FlaxAutoModel"`) --
  The auto class to register this new model with.</paramsdesc><paramgroups>0</paramgroups></docstring>

Register this class with a given auto class. This should only be used for custom models as the ones in the
library are already mapped with an auto class.

<Tip warning=&amp;lcub;true}>

This API is experimental and may have some slight breaking changes in the next releases.

</Tip>




</div>
<div class="docstring">
<docstring><name>save\_pretrained</name><anchor>transformers.FlaxPreTrainedModel.save_pretrained</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L659</source><parameters>[{"name": "save_directory", "val": ": typing.Union[str, os.PathLike]"}, {"name": "params", "val": " = None"}, {"name": "push_to_hub", "val": " = False"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **save_directory** (`str` or `os.PathLike`) --
  Directory to which to save. Will be created if it doesn't exist.
- **push_to_hub** (`bool`, *optional*, defaults to `False`) --
  Whether or not to push your model to the Hugging Face model hub after saving it.

  <Tip warning=&amp;lcub;true}>

  Using `push_to_hub=True` will synchronize the repository you are pushing to with `save_directory`,
  which requires `save_directory` to be a local clone of the repo you are pushing to if it's an existing
  folder. Pass along `temp_dir=True` to use a temporary directory instead.

  </Tip>

  kwargs --
  Additional key word arguments passed along to the [push_to_hub()](/docs/transformers/pr_15592/en/main_classes/model#transformers.file_utils.PushToHubMixin.push_to_hub) method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Save a model and its configuration file to a directory, so that it can be re-loaded using the
`[from_pretrained()](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel.from_pretrained)` class method




</div>
<div class="docstring">
<docstring><name>to\_bf16</name><anchor>transformers.FlaxPreTrainedModel.to_bf16</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L191</source><parameters>[{"name": "params", "val": ": typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict]"}, {"name": "mask", "val": ": typing.Any = None"}]</parameters><paramsdesc>- **params** (`Union[Dict, FrozenDict]`) --
  A `PyTree` of model parameters.
- **mask** (`Union[Dict, FrozenDict]`) --
  A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params
  you want to cast, and should be `False` for those you want to skip.</paramsdesc><paramgroups>0</paramgroups></docstring>

Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast
the `params` in place.

This method can be used on TPU to explicitly convert the model parameters to bfloat16 precision to do full
half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed.



Examples:

```python
>>> from transformers import FlaxBertModel

>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision
>>> model.params = model.to_bf16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util

>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = &amp;lcub;
...     path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
...     for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_bf16(model.params, mask)
```

</div>
<div class="docstring">
<docstring><name>to\_fp16</name><anchor>transformers.FlaxPreTrainedModel.to_fp16</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L257</source><parameters>[{"name": "params", "val": ": typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict]"}, {"name": "mask", "val": ": typing.Any = None"}]</parameters><paramsdesc>- **params** (`Union[Dict, FrozenDict]`) --
  A `PyTree` of model parameters.
- **mask** (`Union[Dict, FrozenDict]`) --
  A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params
  you want to cast, and should be `False` for those you want to skip</paramsdesc><paramgroups>0</paramgroups></docstring>

Cast the floating-point `parmas` to `jax.numpy.float16`. This returns a new `params` tree and does not cast the
`params` in place.

This method can be used on GPU to explicitly convert the model parameters to float16 precision to do full
half-precision training or to save weights in float16 for inference in order to save memory and improve speed.



Examples:

```python
>>> from transformers import FlaxBertModel

>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to cast these to float16
>>> model.params = model.to_fp16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util

>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = &amp;lcub;
...     path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
...     for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_fp16(model.params, mask)
```

</div>
<div class="docstring">
<docstring><name>to\_fp32</name><anchor>transformers.FlaxPreTrainedModel.to_fp32</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/modeling_flax_utils.py#L230</source><parameters>[{"name": "params", "val": ": typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict]"}, {"name": "mask", "val": ": typing.Any = None"}]</parameters><paramsdesc>- **params** (`Union[Dict, FrozenDict]`) --
  A `PyTree` of model parameters.
- **mask** (`Union[Dict, FrozenDict]`) --
  A `PyTree` with same structure as the `params` tree. The leaves should be booleans, `True` for params
  you want to cast, and should be `False` for those you want to skip</paramsdesc><paramgroups>0</paramgroups></docstring>

Cast the floating-point `parmas` to `jax.numpy.float32`. This method can be used to explicitly convert the
model parameters to fp32 precision. This returns a new `params` tree and does not cast the `params` in place.



Examples:

```python
>>> from transformers import FlaxBertModel

>>> # Download model and configuration from huggingface.co
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to illustrate the use of this method,
>>> # we'll first cast to fp16 and back to fp32
>>> model.params = model.to_f16(model.params)
>>> # now cast back to fp32
>>> model.params = model.to_fp32(model.params)
```

</div></div>

<h2 id="transformers.generation_utils.GenerationMixin">Generation</h2>

<div class="docstring">

<docstring><name>class transformers.generation\_utils.GenerationMixin</name><anchor>transformers.generation_utils.GenerationMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L378</source><parameters>[]</parameters></docstring>

A class containing all of the functions supporting generation, to be used as a mixin in [PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel).



<div class="docstring">
<docstring><name>adjust\_logits\_during\_generation</name><anchor>transformers.generation_utils.GenerationMixin.adjust_logits_during_generation</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L457</source><parameters>[{"name": "logits", "val": ": FloatTensor"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Implement in subclasses of [PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel) for custom behavior to adjust the logits in the generate method.


</div>
<div class="docstring">
<docstring><name>beam\_sample</name><anchor>transformers.generation_utils.GenerationMixin.beam_sample</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L2165</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "beam_scorer", "val": ": BeamScorer"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "logits_warper", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **beam_scorer** (`BeamScorer`) --
  A derived instance of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) that defines how beam hypotheses are constructed, stored and
  sorted during generation. For more information, the documentation of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) should be read.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.
- **logits_warper** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsWarper](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsWarper) used
  to warp the prediction score distribution of the language modeling head applied before multinomial
  sampling at each generation step.
- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is
  an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>[BeamSampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleDecoderOnlyOutput), [BeamSampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleEncoderDecoderOutput) or
`torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[BeamSampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleDecoderOnlyOutput) if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [BeamSampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleEncoderDecoderOutput) if
`model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using beam search with multinomial sampling.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     TopKLogitsWarper,
...     TemperatureLogitsWarper,
...     BeamSearchScorer,
... )
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")

>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids

>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id

>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = &amp;lcub;
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }

>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
...     batch_size=1,
...     max_length=model.config.max_length,
...     num_beams=num_beams,
...     device=model.device,
... )

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id)]
... )
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList(
...     [
...         TopKLogitsWarper(50),
...         TemperatureLogitsWarper(0.7),
...     ]
... )

>>> outputs = model.beam_sample(
...     input_ids, beam_scorer, logits_processor=logits_processor, logits_warper=logits_warper, **model_kwargs
... )

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>beam\_search</name><anchor>transformers.generation_utils.GenerationMixin.beam_search</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L1855</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "beam_scorer", "val": ": BeamScorer"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **beam_scorer** (`BeamScorer`) --
  An derived instance of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) that defines how beam hypotheses are constructed, stored and
  sorted during generation. For more information, the documentation of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) should be read.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.
- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is
  an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>`generation_utilsBeamSearchDecoderOnlyOutput`, [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) or
`torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) if
`model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using beam search decoding.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     BeamSearchScorer,
... )
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")

>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids


>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id

>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = &amp;lcub;
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }

>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
...     batch_size=1,
...     num_beams=num_beams,
...     device=model.device,
... )

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
...     ]
... )

>>> outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>compute\_transition\_beam\_scores</name><anchor>transformers.generation_utils.GenerationMixin.compute_transition_beam_scores</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L767</source><parameters>[{"name": "sequences", "val": ": Tensor"}, {"name": "scores", "val": ": typing.Tuple[torch.Tensor]"}, {"name": "beam_indices", "val": ": Tensor"}, {"name": "eos_token_id", "val": ": int = None"}]</parameters></docstring>
compute the transition probabilities of sequences given generation
scores and beam indices

</div>
<div class="docstring">
<docstring><name>constrained\_beam\_search</name><anchor>transformers.generation_utils.GenerationMixin.constrained_beam_search</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L2848</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "constrained_beam_scorer", "val": ": ConstrainedBeamSearchScorer"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = None"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **constrained_beam_scorer** (`ConstrainedBeamSearchScorer`) --
  A derived instance of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) that defines how beam hypotheses are constructed, stored and
  sorted during generation, while satisfying a list of positive constraints. For more information, the
  documentation of [ConstrainedBeamSearchScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.ConstrainedBeamSearchScorer) should be read.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.
- **logits_warper** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsWarper](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsWarper) used
  to warp the prediction score distribution of the language modeling head applied before multinomial
  sampling at each generation step.
- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is
  an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>`generation_utilsBeamSearchDecoderOnlyOutput`, [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) or
`torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) if
`model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using beam search decoding.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     ConstrainedBeamSearchScorer,
...     PhrasalConstraint,
... )
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")

>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids


>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id

>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = &amp;lcub;
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }

>>> constraint_str = "sind"
>>> constraint_token_ids = tokenizer.encode(constraint_str)[:-1]  # slice to remove eos token
>>> constraints = [PhrasalConstraint(token_ids=constraint_token_ids)]


>>> # instantiate beam scorer
>>> beam_scorer = ConstrainedBeamSearchScorer(
...     batch_size=1, num_beams=num_beams, device=model.device, constraints=constraints
... )

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
...     ]
... )

>>> outputs = model.constrained_beam_search(
...     input_ids, beam_scorer, constraints=constraints, logits_processor=logits_processor, **model_kwargs
... )

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
# => ['Wie alter sind Sie?']
```

</div>
<div class="docstring">
<docstring><name>generate</name><anchor>transformers.generation_utils.GenerationMixin.generate</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L806</source><parameters>[{"name": "inputs", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "min_length", "val": ": typing.Optional[int] = None"}, {"name": "do_sample", "val": ": typing.Optional[bool] = None"}, {"name": "early_stopping", "val": ": typing.Optional[bool] = None"}, {"name": "num_beams", "val": ": typing.Optional[int] = None"}, {"name": "temperature", "val": ": typing.Optional[float] = None"}, {"name": "top_k", "val": ": typing.Optional[int] = None"}, {"name": "top_p", "val": ": typing.Optional[float] = None"}, {"name": "typical_p", "val": ": typing.Optional[float] = None"}, {"name": "repetition_penalty", "val": ": typing.Optional[float] = None"}, {"name": "bad_words_ids", "val": ": typing.Optional[typing.Iterable[int]] = None"}, {"name": "bos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "length_penalty", "val": ": typing.Optional[float] = None"}, {"name": "no_repeat_ngram_size", "val": ": typing.Optional[int] = None"}, {"name": "encoder_no_repeat_ngram_size", "val": ": typing.Optional[int] = None"}, {"name": "num_return_sequences", "val": ": typing.Optional[int] = None"}, {"name": "max_time", "val": ": typing.Optional[float] = None"}, {"name": "max_new_tokens", "val": ": typing.Optional[int] = None"}, {"name": "decoder_start_token_id", "val": ": typing.Optional[int] = None"}, {"name": "use_cache", "val": ": typing.Optional[bool] = None"}, {"name": "num_beam_groups", "val": ": typing.Optional[int] = None"}, {"name": "diversity_penalty", "val": ": typing.Optional[float] = None"}, {"name": "prefix_allowed_tokens_fn", "val": ": typing.Union[typing.Callable[[int, torch.Tensor], typing.List[int]], NoneType] = None"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = []"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = []"}, {"name": "constraints", "val": ": typing.Optional[typing.List[transformers.generation_beam_constraints.Constraint]] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "forced_bos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "forced_eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "remove_invalid_values", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **inputs** (`torch.Tensor` of shape `(batch_size, sequence_length)`, `(batch_size, sequence_length, --
- **feature_dim)`** or `(batch_size, num_channels, height, width)`, *optional*) --
  The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
  method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
  should of in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of
  `input_ids`, `input_values`, `input_features`, or `pixel_values`.
- **max_length** (`int`, *optional*, defaults to `model.config.max_length`) --
  The maximum length of the sequence to be generated.
- **max_new_tokens** (`int`, *optional*, defaults to None) --
  The maximum numbers of tokens to generate, ignore the current number of tokens. Use either
  `max_new_tokens` or `max_length` but not both, they serve the same purpose.
- **min_length** (`int`, *optional*, defaults to 10) --
  The minimum length of the sequence to be generated.
- **do_sample** (`bool`, *optional*, defaults to `False`) --
  Whether or not to use sampling ; use greedy decoding otherwise.
- **early_stopping** (`bool`, *optional*, defaults to `False`) --
  Whether to stop the beam search when at least `num_beams` sentences are finished per batch or not.
- **num_beams** (`int`, *optional*, defaults to 1) --
  Number of beams for beam search. 1 means no beam search.
- **temperature** (`float`, *optional*, defaults to 1.0) --
  The value used to module the next token probabilities.
- **top_k** (`int`, *optional*, defaults to 50) --
  The number of highest probability vocabulary tokens to keep for top-k-filtering.
- **top_p** (`float`, *optional*, defaults to 1.0) --
  If set to float &amp;lt; 1, only the most probable tokens with probabilities that add up to `top_p` or higher
  are kept for generation.
- **repetition_penalty** (`float`, *optional*, defaults to 1.0) --
  The parameter for repetition penalty. 1.0 means no penalty. See [this
  paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **bos_token_id** (`int`, *optional*) --
  The id of the *beginning-of-sequence* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **length_penalty** (`float`, *optional*, defaults to 1.0) --
  Exponential penalty to the length. 1.0 means no penalty. Set to values &amp;lt; 1.0 in order to encourage the
  model to generate shorter sequences, to a value > 1.0 in order to encourage the model to produce longer
  sequences.
- **no_repeat_ngram_size** (`int`, *optional*, defaults to 0) --
  If set to int > 0, all ngrams of that size can only occur once.
- **encoder_no_repeat_ngram_size** (`int`, *optional*, defaults to 0) --
  If set to int > 0, all ngrams of that size that occur in the `encoder_input_ids` cannot occur in the
  `decoder_input_ids`.
- **bad_words_ids(`List[List[int]]`,** *optional*) --
  List of token ids that are not allowed to be generated. In order to get the token ids of the words that
  should not appear in the generated text, use `tokenizer(bad_words, add_prefix_space=True,
  add_special_tokens=False).input_ids`.
- **num_return_sequences(`int`,** *optional*, defaults to 1) --
  The number of independently computed returned sequences for each element in the batch.
- **max_time(`float`,** *optional*, defaults to None) --
  The maximum amount of time you allow the computation to run for in seconds. generation will still
  finish the current pass after allocated time has been passed.
- **attention_mask** (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*) --
  Mask to avoid performing attention on padding token indices. Mask values are in `[0, 1]`, 1 for tokens
  that are not masked, and 0 for masked tokens. If not provided, will default to a tensor the same shape
  as `input_ids` that masks the pad token. [What are attention masks?](../glossary#attention-mask)
- **decoder_start_token_id** (`int`, *optional*) --
  If an encoder-decoder model starts decoding with a different token than *bos*, the id of that token.
  use_cache -- (`bool`, *optional*, defaults to `True`):
  Whether or not the model should use the past last key/values attentions (if applicable to the model) to
  speed up decoding.
- **num_beam_groups** (`int`, *optional*, defaults to 1) --
  Number of groups to divide `num_beams` into in order to ensure diversity among different groups of
  beams. [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details.
- **diversity_penalty** (`float`, *optional*, defaults to 0.0) --
  This value is subtracted from a beam's score if it generates a token same as any beam from other group
  at a particular time. Note that `diversity_penalty` is only effective if `group beam search` is
  enabled.
  prefix_allowed_tokens_fn -- (`Callable[[int, torch.Tensor], List[int]]`, *optional*):
  If provided, this function constraints the beam search to allowed tokens only at each step. If not
  provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and
  `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned
  on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful
  for constrained generation conditioned on the prefix, as described in [Autoregressive Entity
  Retrieval](https://arxiv.org/abs/2010.00904).
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  Custom logits processors that complement the default logits processors built from arguments and a
  model's config. If a logit processor is passed that is already created with the arguments or a model's
  config an error is thrown. This feature is intended for advanced users.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  Custom stopping criteria that complement the default stopping criteria built from arguments and a
  model's config. If a stopping criteria is passed that is already created with the arguments or a
  model's config an error is thrown. This feature is intended for advanced users.
- **constraints** (`List[Constraint]`, *optional*) --
  Custom constraints that can be added to the generation to ensure that the output will contain the use
  of certain tokens as defined by `Constraint` objects, in the most sensible way possible.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **forced_bos_token_id** (`int`, *optional*) --
  The id of the token to force as the first generated token after the `decoder_start_token_id`. Useful
  for multilingual models like [mBART](../model_doc/mbart) where the first generated token needs to be
  the target language token.
- **forced_eos_token_id** (`int`, *optional*) --
  The id of the token to force as the last generated token when `max_length` is reached.
- **remove_invalid_values** (`bool`, *optional*) --
  Whether to remove possible *nan* and *inf* outputs of the model to prevent the generation method to
  crash. Note that using `remove_invalid_values` can slow down generation.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If the model
  is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs
  should be prefixed with *decoder_*.</paramsdesc><paramgroups>0</paramgroups><rettype>[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) or `torch.LongTensor`</rettype><retdesc>A [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) (if
`return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor`.

If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) types are:

- [GreedySearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchDecoderOnlyOutput),
- [SampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleDecoderOnlyOutput),
- [BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput),
- [BeamSampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleDecoderOnlyOutput)

If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) types are:

- [GreedySearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchEncoderDecoderOutput),
- [SampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleEncoderDecoderOutput),
- [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput),
- [BeamSampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSampleEncoderDecoderOutput)</retdesc></docstring>

Generates sequences for models with a language modeling head. The method currently supports greedy decoding,
multinomial sampling, beam-search decoding, and beam-search multinomial sampling.

Apart from `inputs`, all the arguments below will default to the value of the attribute of the same name inside
the [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) of the model. The default values indicated are the default values of those config.

Most of these parameters are explained in more detail in [this blog
post](https://huggingface.co/blog/how-to-generate).







Examples:

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM

>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> # do greedy decoding without providing a prompt
>>> outputs = model.generate(max_length=40)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))

>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> document = (
...     "at least two people were killed in a suspected bomb attack on a passenger bus "
...     "in the strife-torn southern philippines on monday , the military said."
... )
>>> # encode input context
>>> input_ids = tokenizer(document, return_tensors="pt").input_ids
>>> # generate 3 independent sequences using beam search decoding (5 beams)
>>> # with T5 encoder-decoder model conditioned on short news article.
>>> outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> input_context = "The dog"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> # generate 3 candidates using sampling
>>> outputs = model.generate(input_ids=input_ids, max_length=20, num_return_sequences=3, do_sample=True)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))

>>> tokenizer = AutoTokenizer.from_pretrained("ctrl")
>>> model = AutoModelForCausalLM.from_pretrained("ctrl")
>>> # "Legal" is one of the control codes for ctrl
>>> input_context = "Legal My neighbor is"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> outputs = model.generate(input_ids=input_ids, max_length=20, repetition_penalty=1.2)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))

>>> tokenizer = AutoTokenizer.from_pretrained("gpt2", use_fast=False)
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> input_context = "My cute dog"
>>> # get tokens of words that should not be generated
>>> bad_words_ids = tokenizer(
...     ["idiot", "stupid", "shut up"], add_prefix_space=True, add_special_tokens=False
>>> ).input_ids
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> # generate sequences without allowing bad_words to be generated
>>> outputs = model.generate(input_ids=input_ids, max_length=20, do_sample=True, bad_words_ids=bad_words_ids)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>greedy\_search</name><anchor>transformers.generation_utils.GenerationMixin.greedy_search</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L1387</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.

- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific keyword arguments will be forwarded to the `forward` function of the model.
  If model is an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>[GreedySearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchDecoderOnlyOutput), [GreedySearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchEncoderDecoderOutput)
or `torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[GreedySearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchDecoderOnlyOutput) if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [GreedySearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.GreedySearchEncoderDecoderOutput) if
`model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using greedy decoding.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForCausalLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
... )

>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")

>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id

>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         MinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
...     ]
... )

>>> outputs = model.greedy_search(input_ids, logits_processor=logits_processor)

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>group\_beam\_search</name><anchor>transformers.generation_utils.GenerationMixin.group_beam_search</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L2485</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "beam_scorer", "val": ": BeamScorer"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **beam_scorer** (`BeamScorer`) --
  An derived instance of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) that defines how beam hypotheses are constructed, stored and
  sorted during generation. For more information, the documentation of [BeamScorer](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.BeamScorer) should be read.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.
- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)

  model_kwargs --
  Additional model specific kwargs that will be forwarded to the `forward` function of the model. If
  model is an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>[BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput), [BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) or
`torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) if [BeamSearchDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) if
`model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a
[BeamSearchEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.BeamSearchEncoderDecoderOutput) if `model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using beam search decoding.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForSeq2SeqLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     HammingDiversityLogitsProcessor,
...     BeamSearchScorer,
... )
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")

>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids


>>> # lets run diverse beam search using 6 beams
>>> num_beams = 6
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id

>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = &amp;lcub;
...     "encoder_outputs": model.get_encoder()(
...         encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
...     )
... }

>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
...     batch_size=1,
...     max_length=model.config.max_length,
...     num_beams=num_beams,
...     device=model.device,
...     num_beam_groups=3,
... )

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         HammingDiversityLogitsProcessor(5.5, num_beams=6, num_beam_groups=3),
...         MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
...     ]
... )

>>> outputs = model.group_beam_search(
...     input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs
... )

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>prepare\_inputs\_for\_generation</name><anchor>transformers.generation_utils.GenerationMixin.prepare_inputs_for_generation</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L451</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Implement in subclasses of [PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel) for custom behavior to prepare inputs in the generate method.


</div>
<div class="docstring">
<docstring><name>sample</name><anchor>transformers.generation_utils.GenerationMixin.sample</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_utils.py#L1612</source><parameters>[{"name": "input_ids", "val": ": LongTensor"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "stopping_criteria", "val": ": typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None"}, {"name": "logits_warper", "val": ": typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "synced_gpus", "val": ": typing.Optional[bool] = False"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`torch.LongTensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **logits_processor** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **stopping_criteria** (`StoppingCriteriaList`, *optional*) --
  An instance of [StoppingCriteriaList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteriaList). List of instances of class derived from [StoppingCriteria](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.StoppingCriteria)
  used to tell if the generation loop should stop.
- **logits_warper** (`LogitsProcessorList`, *optional*) --
  An instance of [LogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsProcessorList). List of instances of class derived from [LogitsWarper](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.LogitsWarper) used
  to warp the prediction score distribution of the language modeling head applied before multinomial
  sampling at each generation step.
- **max_length** (`int`, *optional*, defaults to 20) --
  **DEPRECATED**. Use `logits_processor` or `stopping_criteria` directly to cap the number of generated
  tokens. The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **synced_gpus** (`bool`, *optional*, defaults to `False`) --
  Whether to continue running the while loop until max_length (needed for ZeRO stage 3)
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If model is
  an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>[SampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleDecoderOnlyOutput), [SampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleEncoderDecoderOutput) or
`torch.LongTensor`: A `torch.LongTensor` containing the generated tokens (default behaviour) or a
[SampleDecoderOnlyOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleDecoderOnlyOutput) if `model.config.is_encoder_decoder=False` and
`return_dict_in_generate=True` or a [SampleEncoderDecoderOutput](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.generation_utils.SampleEncoderDecoderOutput) if
`model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using multinomial sampling.





Examples:

```python
>>> from transformers import (
...     AutoTokenizer,
...     AutoModelForCausalLM,
...     LogitsProcessorList,
...     MinLengthLogitsProcessor,
...     TopKLogitsWarper,
...     TemperatureLogitsWarper,
... )

>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")

>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id

>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids

>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
...     [
...         MinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
...     ]
... )
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList(
...     [
...         TopKLogitsWarper(50),
...         TemperatureLogitsWarper(0.7),
...     ]
... )

>>> outputs = model.sample(input_ids, logits_processor=logits_processor, logits_warper=logits_warper)

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div></div>

<div class="docstring">

<docstring><name>class transformers.generation\_tf\_utils.TFGenerationMixin</name><anchor>transformers.generation_tf_utils.TFGenerationMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_tf_utils.py#L339</source><parameters>[]</parameters></docstring>

A class containing all of the functions supporting generation, to be used as a mixin in [TFPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel).



<div class="docstring">
<docstring><name>adjust\_logits\_during\_generation</name><anchor>transformers.generation_tf_utils.TFGenerationMixin.adjust_logits_during_generation</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_tf_utils.py#L1507</source><parameters>[{"name": "logits", "val": ""}, {"name": "cur_len", "val": ""}, {"name": "max_length", "val": ""}, {"name": "forced_bos_token_id", "val": ""}, {"name": "forced_eos_token_id", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Implement in subclasses of [PreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.PreTrainedModel) for custom behavior to adjust the logits in the generate method.


</div>
<div class="docstring">
<docstring><name>generate</name><anchor>transformers.generation_tf_utils.TFGenerationMixin.generate</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_tf_utils.py#L359</source><parameters>[{"name": "input_ids", "val": " = None"}, {"name": "max_length", "val": " = None"}, {"name": "min_length", "val": " = None"}, {"name": "do_sample", "val": " = None"}, {"name": "early_stopping", "val": " = None"}, {"name": "num_beams", "val": " = None"}, {"name": "temperature", "val": " = None"}, {"name": "top_k", "val": " = None"}, {"name": "top_p", "val": " = None"}, {"name": "repetition_penalty", "val": " = None"}, {"name": "bad_words_ids", "val": " = None"}, {"name": "bos_token_id", "val": " = None"}, {"name": "pad_token_id", "val": " = None"}, {"name": "eos_token_id", "val": " = None"}, {"name": "length_penalty", "val": " = None"}, {"name": "no_repeat_ngram_size", "val": " = None"}, {"name": "num_return_sequences", "val": " = None"}, {"name": "attention_mask", "val": " = None"}, {"name": "decoder_start_token_id", "val": " = None"}, {"name": "use_cache", "val": " = None"}, {"name": "output_scores", "val": " = None"}, {"name": "output_attentions", "val": " = None"}, {"name": "output_hidden_states", "val": " = None"}, {"name": "return_dict_in_generate", "val": " = None"}, {"name": "forced_bos_token_id", "val": " = None"}, {"name": "forced_eos_token_id", "val": " = None"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`tf.Tensor` of shape `(batch_size, sequence_length)`, `(batch_size, sequence_length, --
- **feature_dim)`** or `(batch_size, num_channels, height, width)`, *optional*) --
  The sequence used as a prompt for the generation or as model inputs to the encoder. If `None` the
  method initializes it with `bos_token_id` and a batch size of 1. For decoder-only models `inputs`
  should of in the format of `input_ids`. For encoder-decoder models *inputs* can represent any of
  `input_ids`, `input_values`, `input_features`, or `pixel_values`.
- **max_length** (`int`, *optional*, defaults to 20) --
  The maximum length of the sequence to be generated.
- **min_length** (`int`, *optional*, defaults to 10) --
  The minimum length of the sequence to be generated.
- **do_sample** (`bool`, *optional*, defaults to `False`) --
  Whether or not to use sampling ; use greedy decoding otherwise.
- **early_stopping** (`bool`, *optional*, defaults to `False`) --
  Whether to stop the beam search when at least `num_beams` sentences are finished per batch or not.
- **num_beams** (`int`, *optional*, defaults to 1) --
  Number of beams for beam search. 1 means no beam search.
- **temperature** (`float`, *optional*, defaults to 1.0) --
  The value used to module the next token probabilities.
- **top_k** (`int`, *optional*, defaults to 50) --
  The number of highest probability vocabulary tokens to keep for top-k-filtering.
- **top_p** (`float`, *optional*, defaults to 1.0) --
  If set to float &amp;lt; 1, only the most probable tokens with probabilities that add up to `top_p` or higher
  are kept for generation.
- **repetition_penalty** (`float`, *optional*, defaults to 1.0) --
  The parameter for repetition penalty. 1.0 means no penalty. See [this
  paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **bos_token_id** (`int`, *optional*) --
  The id of the *beginning-of-sequence* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **length_penalty** (`float`, *optional*, defaults to 1.0) --
  Exponential penalty to the length. 1.0 means no penalty.

  Set to values &amp;lt; 1.0 in order to encourage the model to generate shorter sequences, to a value > 1.0 in
  order to encourage the model to produce longer sequences.
- **no_repeat_ngram_size** (`int`, *optional*, defaults to 0) --
  If set to int > 0, all ngrams of that size can only occur once.
- **bad_words_ids(`List[int]`,** *optional*) --
  List of token ids that are not allowed to be generated. In order to get the tokens of the words that
  should not appear in the generated text, use `tokenizer.encode(bad_word, add_prefix_space=True)`.
- **num_return_sequences(`int`,** *optional*, defaults to 1) --
  The number of independently computed returned sequences for each element in the batch.
- **attention_mask** (`tf.Tensor` of `dtype=tf.int32` and shape `(batch_size, sequence_length)`, *optional*) --
  Mask to avoid performing attention on padding token indices. Mask values are in `[0, 1]`, 1 for tokens
  that are not masked, and 0 for masked tokens.

  If not provided, will default to a tensor the same shape as `input_ids` that masks the pad token.

  [What are attention masks?](../glossary#attention-mask)
- **decoder_start_token_id** (`int`, *optional*) --
  If an encoder-decoder model starts decoding with a different token than *bos*, the id of that token.
  use_cache -- (`bool`, *optional*, defaults to `True`):
  Whether or not the model should use the past last key/values attentions (if applicable to the model) to
  speed up decoding.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
- **forced_bos_token_id** (`int`, *optional*) --
  The id of the token to force as the first generated token after the `decoder_start_token_id`. Useful
  for multilingual models like [mBART](../model_doc/mbart) where the first generated token needs to be
  the target language token.
- **forced_eos_token_id** (`int`, *optional*) --
  The id of the token to force as the last generated token when `max_length` is reached.
  model_specific_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model.</paramsdesc><paramgroups>0</paramgroups><rettype>[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) or `tf.Tensor`</rettype><retdesc>A [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) (if
`return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `tf.Tensor`.

If the model is *not* an encoder-decoder model (`model.config.is_encoder_decoder=False`), the possible
[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) types are:

- `TFGreedySearchDecoderOnlyOutput`
- `TFSampleDecoderOnlyOutput`
- `TFBeamSearchDecoderOnlyOutput`
- `TFBeamSampleDecoderOnlyOutput`
If the model is an encoder-decoder model (`model.config.is_encoder_decoder=True`), the possible
[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) types are:

- `TFGreedySearchEncoderDecoderOutput`
- `TFSampleEncoderDecoderOutput`
- `TFBeamSearchEncoderDecoderOutput`
- `TFBeamSampleEncoderDecoderOutput`/retdesc></docstring>

Generates sequences for models with a language modeling head. The method currently supports greedy decoding,
beam-search decoding, sampling with temperature, sampling with top-k or nucleus sampling.

Adapted in part from [Facebook's XLM beam search
code](https://github.com/facebookresearch/XLM/blob/9e6f6814d17be4fe5b15f2e6c43eb2b2d76daeb4/src/model/transformer.py#L529).

Apart from `input_ids` and `attention_mask`, all the arguments below will default to the value of the attribute
of the same name inside the [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) of the model. The default values indicated are the default
values of those config.

Most of these parameters are explained in more detail in [this blog
post](https://huggingface.co/blog/how-to-generate).







Examples:

```python
tokenizer = AutoTokenizer.from_pretrained("distilgpt2")  # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained(
    "distilgpt2"
)  # Download model and configuration from huggingface.co and cache.
outputs = model.generate(max_length=40)  # do greedy decoding
print(f"Generated: &amp;lcub;tokenizer.decode(outputs[0], skip_special_tokens=True)}")

tokenizer = AutoTokenizer.from_pretrained("openai-gpt")  # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained(
    "openai-gpt"
)  # Download model and configuration from huggingface.co and cache.
input_context = "The dog"
input_ids = tokenizer.encode(input_context, return_tensors="tf")  # encode input context
outputs = model.generate(
    input_ids=input_ids, num_beams=5, num_return_sequences=3, temperature=1.5
)  # generate 3 independent sequences using beam search decoding (5 beams) with sampling from initial context 'The dog'
for i in range(3):  #  3 output sequences were generated
    print(f"Generated &amp;lcub;i}: &amp;lcub;tokenizer.decode(outputs[i], skip_special_tokens=True)}")

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")  # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained(
    "distilgpt2"
)  # Download model and configuration from huggingface.co and cache.
input_context = "The dog"
input_ids = tokenizer.encode(input_context, return_tensors="tf")  # encode input context
outputs = model.generate(
    input_ids=input_ids, max_length=40, temperature=0.7, num_return_sequences=3, do_sample=True
)  # generate 3 candidates using sampling
for i in range(3):  #  3 output sequences were generated
    print(f"Generated &amp;lcub;i}: &amp;lcub;tokenizer.decode(outputs[i], skip_special_tokens=True)}")

tokenizer = AutoTokenizer.from_pretrained("ctrl")  # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained(
    "ctrl"
)  # Download model and configuration from huggingface.co and cache.
input_context = "Legal My neighbor is"  # "Legal" is one of the control codes for ctrl
input_ids = tokenizer.encode(input_context, return_tensors="tf")  # encode input context
outputs = model.generate(
    input_ids=input_ids, max_length=50, temperature=0.7, repetition_penalty=1.2
)  # generate sequences
print(f"Generated: &amp;lcub;tokenizer.decode(outputs[0], skip_special_tokens=True)}")

tokenizer = AutoTokenizer.from_pretrained("gpt2")  # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained(
    "gpt2"
)  # Download model and configuration from huggingface.co and cache.
input_context = "My cute dog"
bad_words_ids = [
    tokenizer.encode(bad_word, add_prefix_space=True) for bad_word in ["idiot", "stupid", "shut up"]
]
input_ids = tokenizer.encode(input_context, return_tensors="tf")  # encode input context
outputs = model.generate(
    input_ids=input_ids, max_length=100, do_sample=True, bad_words_ids=bad_words_ids
)  # generate sequences without allowing bad_words to be generated
```

</div>
<div class="docstring">
<docstring><name>greedy\_search</name><anchor>transformers.generation_tf_utils.TFGenerationMixin.greedy_search</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_tf_utils.py#L1992</source><parameters>[{"name": "input_ids", "val": ": Tensor"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "logits_processor", "val": ": typing.Optional[transformers.generation_tf_logits_process.TFLogitsProcessorList] = None"}, {"name": "output_attentions", "val": ": typing.Optional[bool] = None"}, {"name": "output_hidden_states", "val": ": typing.Optional[bool] = None"}, {"name": "output_scores", "val": ": typing.Optional[bool] = None"}, {"name": "return_dict_in_generate", "val": ": typing.Optional[bool] = None"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`tf.Tensor` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **logits_processor** (`TFLogitsProcessorList`, *optional*) --
  An instance of [TFLogitsProcessorList](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.TFLogitsProcessorList). List of instances of class derived from [TFLogitsProcessor](/docs/transformers/pr_15592/en/internal/generation_utils#transformers.TFLogitsProcessor)
  used to modify the prediction scores of the language modeling head applied at each generation step.
- **max_length** (`int`, *optional*, defaults to 20) --
  The maximum length of the sequence to be generated.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **output_attentions** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the attentions tensors of all attention layers. See `attentions` under
  returned tensors for more details.
- **output_hidden_states** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
  for more details.
- **output_scores** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return the prediction scores. See `scores` under returned tensors for more details.
- **return_dict_in_generate** (`bool`, *optional*, defaults to `False`) --
  Whether or not to return a [ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput) instead of a plain tuple.
  model_kwargs --
  Additional model specific keyword arguments will be forwarded to the `forward` function of the model.
  If model is an encoder-decoder model the kwargs should include `encoder_outputs`.</paramsdesc><paramgroups>0</paramgroups><retdesc>`TFGreedySearchDecoderOnlyOutput`
`TFGreedySearchEncoderDecoderOutput`or `tf.Tensor`: A `tf.Tensor` containing the
generated tokens (default behaviour) or a `TFGreedySearchDecoderOnlyOutput`if
`model.config.is_encoder_decoder=False` and `return_dict_in_generate=True` or a
`TFGreedySearchEncoderDecoderOutput`if `model.config.is_encoder_decoder=True`.</retdesc></docstring>

Generates sequences for models with a language modeling head using greedy decoding.





Examples:

```python
>>> from transformers import (
...     TFAutoTokenizer,
...     TFAutoModelForCausalLM,
...     TFLogitsProcessorList,
...     TFMinLengthLogitsProcessor,
... )

>>> tokenizer = TFAutoTokenizer.from_pretrained("gpt2")
>>> model = TFAutoModelForCausalLM.from_pretrained("gpt2")

>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id

>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="tf").input_ids

>>> # instantiate logits processors
>>> logits_processor = TFLogitsProcessorList(
...     [
...         TFMinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
...     ]
... )

>>> outputs = model.greedy_search(input_ids, logits_processor=logits_processor)

>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div>
<div class="docstring">
<docstring><name>prepare\_inputs\_for\_generation</name><anchor>transformers.generation_tf_utils.TFGenerationMixin.prepare_inputs_for_generation</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_tf_utils.py#L344</source><parameters>[{"name": "inputs", "val": ""}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

Implement in subclasses of [TFPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.TFPreTrainedModel) for custom behavior to prepare inputs in the generate method.


</div></div>

<div class="docstring">

<docstring><name>class transformers.generation\_flax\_utils.FlaxGenerationMixin</name><anchor>transformers.generation_flax_utils.FlaxGenerationMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_flax_utils.py#L119</source><parameters>[]</parameters></docstring>

A class containing all of the functions supporting generation, to be used as a mixin in [FlaxPreTrainedModel](/docs/transformers/pr_15592/en/main_classes/model#transformers.FlaxPreTrainedModel).



<div class="docstring">
<docstring><name>generate</name><anchor>transformers.generation_flax_utils.FlaxGenerationMixin.generate</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/generation_flax_utils.py#L154</source><parameters>[{"name": "input_ids", "val": ": ndarray"}, {"name": "max_length", "val": ": typing.Optional[int] = None"}, {"name": "pad_token_id", "val": ": typing.Optional[int] = None"}, {"name": "bos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "decoder_start_token_id", "val": ": typing.Optional[int] = None"}, {"name": "do_sample", "val": ": typing.Optional[bool] = None"}, {"name": "prng_key", "val": ": typing.Optional[jax._src.numpy.lax_numpy.ndarray] = None"}, {"name": "top_k", "val": ": typing.Optional[int] = None"}, {"name": "top_p", "val": ": typing.Optional[float] = None"}, {"name": "temperature", "val": ": typing.Optional[float] = None"}, {"name": "num_beams", "val": ": typing.Optional[int] = None"}, {"name": "no_repeat_ngram_size", "val": ": typing.Optional[int] = None"}, {"name": "min_length", "val": ": typing.Optional[int] = None"}, {"name": "forced_bos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "forced_eos_token_id", "val": ": typing.Optional[int] = None"}, {"name": "length_penalty", "val": ": typing.Optional[float] = None"}, {"name": "early_stopping", "val": ": typing.Optional[bool] = None"}, {"name": "trace", "val": ": bool = True"}, {"name": "params", "val": ": typing.Union[typing.Dict[str, jax._src.numpy.lax_numpy.ndarray], NoneType] = None"}, {"name": "**model_kwargs", "val": ""}]</parameters><paramsdesc>- **input_ids** (`jnp.ndarray` of shape `(batch_size, sequence_length)`) --
  The sequence used as a prompt for the generation.
- **max_length** (`int`, *optional*, defaults to 20) --
  The maximum length of the sequence to be generated.
- **do_sample** (`bool`, *optional*, defaults to `False`) --
  Whether or not to use sampling ; use greedy decoding otherwise.
- **temperature** (`float`, *optional*, defaults to 1.0) --
  The value used to module the next token probabilities.
- **top_k** (`int`, *optional*, defaults to 50) --
  The number of highest probability vocabulary tokens to keep for top-k-filtering.
- **top_p** (`float`, *optional*, defaults to 1.0) --
  If set to float &amp;lt; 1, only the most probable tokens with probabilities that add up to `top_p` or higher
  are kept for generation.
- **pad_token_id** (`int`, *optional*) --
  The id of the *padding* token.
- **bos_token_id** (`int`, *optional*) --
  The id of the *beginning-of-sequence* token.
- **eos_token_id** (`int`, *optional*) --
  The id of the *end-of-sequence* token.
- **num_beams** (`int`, *optional*, defaults to 1) --
  Number of beams for beam search. 1 means no beam search.
- **decoder_start_token_id** (`int`, *optional*) --
  If an encoder-decoder model starts decoding with a different token than *bos*, the id of that token.
- **trace** (`bool`, *optional*, defaults to `True`) --
  Whether to trace generation. Setting `trace=False` should only be used for debugging and will lead to a
  considerably slower runtime.
- **params** (`Dict[str, jnp.ndarray]`, *optional*) --
  Optionally the model parameters can be passed. Can be useful for parallelized generation.
  model_kwargs --
  Additional model specific kwargs will be forwarded to the `forward` function of the model. If the model
  is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs
  should be prefixed with *decoder_*. Also accepts `encoder_outputs` to skip encoder part.</paramsdesc><paramgroups>0</paramgroups><retdesc>[ModelOutput](/docs/transformers/pr_15592/en/main_classes/output#transformers.file_utils.ModelOutput).</retdesc></docstring>

Generates sequences for models with a language modeling head. The method currently supports greedy decoding,
and, multinomial sampling.

Apart from `input_ids`, all the arguments below will default to the value of the attribute of the same name
inside the [PretrainedConfig](/docs/transformers/pr_15592/en/main_classes/configuration#transformers.PretrainedConfig) of the model. The default values indicated are the default values of those
config.

Most of these parameters are explained in more detail in [this blog
post](https://huggingface.co/blog/how-to-generate).





Examples:

```python
>>> from transformers import AutoTokenizer, FlaxAutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = FlaxAutoModelForCausalLM.from_pretrained("distilgpt2")
>>> input_context = "The dog"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="np").input_ids
>>> # generate candidates using sampling
>>> outputs = model.generate(input_ids=input_ids, max_length=20, top_k=30, do_sample=True)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
```

</div></div>

<h2 id="transformers.file_utils.PushToHubMixin">Pushing to the Hub</h2>

<div class="docstring">

<docstring><name>class transformers.file\_utils.PushToHubMixin</name><anchor>transformers.file_utils.PushToHubMixin</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2820</source><parameters>[]</parameters></docstring>

A Mixin containing the functionality to push a model or tokenizer to the hub.



<div class="docstring">
<docstring><name>push\_to\_hub</name><anchor>transformers.file_utils.PushToHubMixin.push_to_hub</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/file_utils.py#L2825</source><parameters>[{"name": "repo_path_or_name", "val": ": typing.Optional[str] = None"}, {"name": "repo_url", "val": ": typing.Optional[str] = None"}, {"name": "use_temp_dir", "val": ": bool = False"}, {"name": "commit_message", "val": ": typing.Optional[str] = None"}, {"name": "organization", "val": ": typing.Optional[str] = None"}, {"name": "private", "val": ": typing.Optional[bool] = None"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "**model_card_kwargs", "val": ""}]</parameters><paramsdesc>- **repo_path_or_name** (`str`, *optional*) --
  Can either be a repository name for your &amp;lcub;object} in the Hub or a path to a local folder (in which case
  the repository will have the name of that local folder). If not specified, will default to the name
  given by `repo_url` and a local directory with that name will be created.
- **repo_url** (`str`, *optional*) --
  Specify this in case you want to push to an existing repository in the hub. If unspecified, a new
  repository will be created in your namespace (unless you specify an `organization`) with `repo_name`.
- **use_temp_dir** (`bool`, *optional*, defaults to `False`) --
  Whether or not to clone the distant repo in a temporary directory or in `repo_path_or_name` inside the
  current working directory. This will slow things down if you are making changes in an existing repo
  since you will need to clone the repo before every push.
- **commit_message** (`str`, *optional*) --
  Message to commit while pushing. Will default to `"add &amp;lcub;object}"`.
- **organization** (`str`, *optional*) --
  Organization in which you want to push your &amp;lcub;object} (you must be a member of this organization).
- **private** (`bool`, *optional*) --
  Whether or not the repository created should be private (requires a paying subscription).
- **use_auth_token** (`bool` or `str`, *optional*) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `transformers-cli login` (stored in `~/.huggingface`). Will default to `True` if
  `repo_url` is not specified.</paramsdesc><paramgroups>0</paramgroups><rettype>`str`</rettype><retdesc>The url of the commit of your &amp;lcub;object} in the given repository.</retdesc></docstring>

Upload the &amp;lcub;object_files} to the 🤗 Model Hub while synchronizing a local clone of the repo in
`repo_path_or_name`.







Examples:

```python
from transformers import &amp;lcub;object_class}

&amp;lcub;object} = &amp;lcub;object_class}.from_pretrained("bert-base-cased")

# Push the &amp;lcub;object} to your namespace with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
&amp;lcub;object}.push_to_hub("my-finetuned-bert")

# Push the &amp;lcub;object} to your namespace with the name "my-finetuned-bert" with no local clone.
&amp;lcub;object}.push_to_hub("my-finetuned-bert", use_temp_dir=True)

# Push the &amp;lcub;object} to an organization with the name "my-finetuned-bert" and have a local clone in the
# *my-finetuned-bert* folder.
&amp;lcub;object}.push_to_hub("my-finetuned-bert", organization="huggingface")

# Make a change to an existing repo that has been cloned locally in *my-finetuned-bert*.
&amp;lcub;object}.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")
```


</div></div>
