---
local: optimization
sections:
- local: transformers.AdamW
  title: AdamW (PyTorch)
- local: transformers.Adafactor
  title: AdaFactor (PyTorch)
- local: transformers.AdamWeightDecay
  title: AdamWeightDecay (TensorFlow)
- local: schedules
  sections:
  - local: transformers.SchedulerType
    title: Learning Rate Schedules (Pytorch)
  - local: transformers.WarmUp
    title: Warmup (TensorFlow)
  title: Schedules
- local: gradient-strategies
  sections:
  - local: transformers.GradientAccumulator
    title: GradientAccumulator (TensorFlow)
  title: Gradient Strategies
title: Optimization
---
<script>
import Tip from "./Tip.svelte";
import Youtube from "./Youtube.svelte";
import Docstring from "./Docstring.svelte";
import CodeBlock from "./CodeBlock.svelte";
import CodeBlockFw from "./CodeBlockFw.svelte";
import DocNotebookDropdown from "./DocNotebookDropdown.svelte";
import IconCopyLink from "./IconCopyLink.svelte";
export let fw: "pt" | "tf"
</script>
<!--Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.
-->

<h1 id="optimization">Optimization</h1>

The `.optimization` module provides:

- an optimizer with weight decay fixed that can be used to fine-tuned models, and
- several schedules in the form of schedule objects that inherit from `_LRSchedule`:
- a gradient accumulation class to accumulate the gradients of multiple batches

<h2 id="transformers.AdamW">AdamW (PyTorch)</h2>

<div class="docstring">

<docstring><name>class transformers.AdamW</name><anchor>transformers.AdamW</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L273</source><parameters>[{"name": "params", "val": ": typing.Iterable[torch.nn.parameter.Parameter]"}, {"name": "lr", "val": ": float = 0.001"}, {"name": "betas", "val": ": typing.Tuple[float, float] = (0.9, 0.999)"}, {"name": "eps", "val": ": float = 1e-06"}, {"name": "weight_decay", "val": ": float = 0.0"}, {"name": "correct_bias", "val": ": bool = True"}, {"name": "no_deprecation_warning", "val": ": bool = False"}]</parameters><paramsdesc>- **params** (`Iterable[nn.parameter.Parameter]`) --
  Iterable of parameters to optimize or dictionaries defining parameter groups.
- **lr** (`float`, *optional*, defaults to 1e-3) --
  The learning rate to use.
- **betas** (`Tuple[float,float]`, *optional*, defaults to (0.9, 0.999)) --
  Adam's betas parameters (b1, b2).
- **eps** (`float`, *optional*, defaults to 1e-6) --
  Adam's epsilon for numerical stability.
- **weight_decay** (`float`, *optional*, defaults to 0) --
  Decoupled weight decay to apply.
- **correct_bias** (`bool`, *optional*, defaults to `True`) --
  Whether or not to correct bias in Adam (for instance, in Bert TF repository they use `False`).
- **no_deprecation_warning** (`bool`, *optional*, defaults to `False`) --
  A flag used to disable the deprecation warning (set to `True` to disable the warning).</paramsdesc><paramgroups>0</paramgroups></docstring>

Implements Adam algorithm with weight decay fix as introduced in [Decoupled Weight Decay
Regularization](https://arxiv.org/abs/1711.05101).





<div class="docstring">
<docstring><name>step</name><anchor>transformers.AdamW.step</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L323</source><parameters>[{"name": "closure", "val": ": typing.Callable = None"}]</parameters><paramsdesc>- **closure** (`Callable`, *optional*) -- A closure that reevaluates the model and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>

Performs a single optimization step.




</div></div>

<h2 id="transformers.Adafactor">AdaFactor (PyTorch)</h2>

<div class="docstring">

<docstring><name>class transformers.Adafactor</name><anchor>transformers.Adafactor</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L385</source><parameters>[{"name": "params", "val": ""}, {"name": "lr", "val": " = None"}, {"name": "eps", "val": " = (1e-30, 0.001)"}, {"name": "clip_threshold", "val": " = 1.0"}, {"name": "decay_rate", "val": " = -0.8"}, {"name": "beta1", "val": " = None"}, {"name": "weight_decay", "val": " = 0.0"}, {"name": "scale_parameter", "val": " = True"}, {"name": "relative_step", "val": " = True"}, {"name": "warmup_init", "val": " = False"}]</parameters><paramsdesc>- **params** (`Iterable[nn.parameter.Parameter]`) --
  Iterable of parameters to optimize or dictionaries defining parameter groups.
- **lr** (`float`, *optional*) --
  The external learning rate.
- **eps** (`Tuple[float, float]`, *optional*, defaults to (1e-30, 1e-3)) --
  Regularization constants for square gradient and parameter scale respectively
- **clip_threshold** (`float`, *optional*, defaults 1.0) --
  Threshold of root mean square of final gradient update
- **decay_rate** (`float`, *optional*, defaults to -0.8) --
  Coefficient used to compute running averages of square
- **beta1** (`float`, *optional*) --
  Coefficient used for computing running averages of gradient
- **weight_decay** (`float`, *optional*, defaults to 0) --
  Weight decay (L2 penalty)
- **scale_parameter** (`bool`, *optional*, defaults to `True`) --
  If True, learning rate is scaled by root mean square
- **relative_step** (`bool`, *optional*, defaults to `True`) --
  If True, time-dependent learning rate is computed instead of external learning rate
- **warmup_init** (`bool`, *optional*, defaults to `False`) --
  Time-dependent learning rate computation depends on whether warm-up initialization is being used</paramsdesc><paramgroups>0</paramgroups></docstring>

AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code:
https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py

Paper: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost* https://arxiv.org/abs/1804.04235 Note that
this optimizer internally adjusts the learning rate depending on the `scale_parameter`, `relative_step` and
`warmup_init` options. To use a manual (external) learning rate schedule you should set `scale_parameter=False` and
`relative_step=False`.



This implementation handles low-precision (FP16, bfloat) values, but we have not thoroughly tested.

Recommended T5 finetuning settings (https://discuss.huggingface.co/t/t5-finetuning-tips/684/3):

- Training without LR warmup or clip_threshold is not recommended.

  - use scheduled LR warm-up to fixed LR
  - use clip_threshold=1.0 (https://arxiv.org/abs/1804.04235)
- Disable relative updates
- Use scale_parameter=False
- Additional optimizer operations like gradient clipping should not be used alongside Adafactor

Example:

```python
Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3)
```

Others reported the following combination to work well:

```python
Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
```

When using `lr=None` with [Trainer](/docs/transformers/pr_15592/en/main_classes/trainer#transformers.Trainer) you will most likely need to use `AdafactorSchedule`scheduler as following:

```python
from transformers.optimization import Adafactor, AdafactorSchedule

optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
lr_scheduler = AdafactorSchedule(optimizer)
trainer = Trainer(..., optimizers=(optimizer, lr_scheduler))
```

Usage:

```python
# replace AdamW with Adafactor
optimizer = Adafactor(
    model.parameters(),
    lr=1e-3,
    eps=(1e-30, 1e-3),
    clip_threshold=1.0,
    decay_rate=-0.8,
    beta1=None,
    weight_decay=0.0,
    relative_step=False,
    scale_parameter=False,
    warmup_init=False,
)
```


<div class="docstring">
<docstring><name>step</name><anchor>transformers.Adafactor.step</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L531</source><parameters>[{"name": "closure", "val": " = None"}]</parameters><paramsdesc>- **closure** (callable, optional) -- A closure that reevaluates the model
  and returns the loss.</paramsdesc><paramgroups>0</paramgroups></docstring>

Performs a single optimization step




</div></div>

<h2 id="transformers.AdamWeightDecay">AdamWeightDecay (TensorFlow)</h2>

<div class="docstring">

<docstring><name>class transformers.AdamWeightDecay</name><anchor>transformers.AdamWeightDecay</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L152</source><parameters>[{"name": "learning_rate", "val": ": typing.Union[float, keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule] = 0.001"}, {"name": "beta_1", "val": ": float = 0.9"}, {"name": "beta_2", "val": ": float = 0.999"}, {"name": "epsilon", "val": ": float = 1e-07"}, {"name": "amsgrad", "val": ": bool = False"}, {"name": "weight_decay_rate", "val": ": float = 0.0"}, {"name": "include_in_weight_decay", "val": ": typing.Optional[typing.List[str]] = None"}, {"name": "exclude_from_weight_decay", "val": ": typing.Optional[typing.List[str]] = None"}, {"name": "name", "val": ": str = 'AdamWeightDecay'"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **learning_rate** (`Union[float, tf.keras.optimizers.schedules.LearningRateSchedule]`, *optional*, defaults to 1e-3) --
  The learning rate to use or a schedule.
- **beta_1** (`float`, *optional*, defaults to 0.9) --
  The beta1 parameter in Adam, which is the exponential decay rate for the 1st momentum estimates.
- **beta_2** (`float`, *optional*, defaults to 0.999) --
  The beta2 parameter in Adam, which is the exponential decay rate for the 2nd momentum estimates.
- **epsilon** (`float`, *optional*, defaults to 1e-7) --
  The epsilon parameter in Adam, which is a small constant for numerical stability.
- **amsgrad** (`bool`, *optional*, default to `False`) --
  Whether to apply AMSGrad variant of this algorithm or not, see [On the Convergence of Adam and
  Beyond](https://arxiv.org/abs/1904.09237).
- **weight_decay_rate** (`float`, *optional*, defaults to 0) --
  The weight decay to apply.
- **include_in_weight_decay** (`List[str]`, *optional*) --
  List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is
  applied to all parameters by default (unless they are in `exclude_from_weight_decay`).
- **exclude_from_weight_decay** (`List[str]`, *optional*) --
  List of the parameter names (or re patterns) to exclude from applying weight decay to. If a
  `include_in_weight_decay` is passed, the names in it will supersede this list.
- **name** (`str`, *optional*, defaults to 'AdamWeightDecay') --
  Optional name for the operations created when applying gradients.
  kwargs --
  Keyword arguments. Allowed to be &amp;lcub;`clipnorm`, `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by
  norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time
  inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use
  `learning_rate` instead.</paramsdesc><paramgroups>0</paramgroups></docstring>

Adam enables L2 weight decay and clip_by_global_norm on gradients. Just adding the square of the weights to the
loss function is *not* the correct way of using L2 regularization/weight decay with Adam, since that will interact
with the m and v parameters in strange ways as shown in [Decoupled Weight Decay
Regularization](https://arxiv.org/abs/1711.05101).

Instead we want ot decay the weights in a manner that doesn't interact with the m/v parameters. This is equivalent
to adding the square of the weights to the loss with plain (non-momentum) SGD.





<div class="docstring">
<docstring><name>from\_config</name><anchor>transformers.AdamWeightDecay.from_config</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L209</source><parameters>[{"name": "config", "val": ""}]</parameters></docstring>
Creates an optimizer from its config with WarmUp custom object.

</div></div>

<div class="docstring">

<docstring><name>transformers.create\_optimizer</name><anchor>transformers.create_optimizer</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L82</source><parameters>[{"name": "init_lr", "val": ": float"}, {"name": "num_train_steps", "val": ": int"}, {"name": "num_warmup_steps", "val": ": int"}, {"name": "min_lr_ratio", "val": ": float = 0.0"}, {"name": "adam_beta1", "val": ": float = 0.9"}, {"name": "adam_beta2", "val": ": float = 0.999"}, {"name": "adam_epsilon", "val": ": float = 1e-08"}, {"name": "weight_decay_rate", "val": ": float = 0.0"}, {"name": "power", "val": ": float = 1.0"}, {"name": "include_in_weight_decay", "val": ": typing.Optional[typing.List[str]] = None"}]</parameters><paramsdesc>- **init_lr** (`float`) --
  The desired learning rate at the end of the warmup phase.
- **num_train_steps** (`int`) --
  The total number of training steps.
- **num_warmup_steps** (`int`) --
  The number of warmup steps.
- **min_lr_ratio** (`float`, *optional*, defaults to 0) --
  The final learning rate at the end of the linear decay will be `init_lr * min_lr_ratio`.
- **adam_beta1** (`float`, *optional*, defaults to 0.9) --
  The beta1 to use in Adam.
- **adam_beta2** (`float`, *optional*, defaults to 0.999) --
  The beta2 to use in Adam.
- **adam_epsilon** (`float`, *optional*, defaults to 1e-8) --
  The epsilon to use in Adam.
- **weight_decay_rate** (`float`, *optional*, defaults to 0) --
  The weight decay to use.
- **power** (`float`, *optional*, defaults to 1.0) --
  The power to use for PolynomialDecay.
- **include_in_weight_decay** (`List[str]`, *optional*) --
  List of the parameter names (or re patterns) to apply weight decay to. If none is passed, weight decay is
  applied to all parameters except bias and layer norm parameters.</paramsdesc><paramgroups>0</paramgroups></docstring>

Creates an optimizer with a learning rate schedule using a warmup phase followed by a linear decay.




</div>

<h2 id="schedules">Schedules</h2>

<h3 id="transformers.SchedulerType">Learning Rate Schedules (Pytorch)</h3>

<div class="docstring">

<docstring><name>class transformers.SchedulerType</name><anchor>transformers.SchedulerType</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/trainer_utils.py#L301</source><parameters>[{"name": "value", "val": ""}, {"name": "names", "val": " = None"}, {"name": "module", "val": " = None"}, {"name": "qualname", "val": " = None"}, {"name": "type", "val": " = None"}, {"name": "start", "val": " = 1"}]</parameters></docstring>
An enumeration.

</div>

<div class="docstring">

<docstring><name>transformers.get\_scheduler</name><anchor>transformers.get_scheduler</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L233</source><parameters>[{"name": "name", "val": ": typing.Union[str, transformers.trainer_utils.SchedulerType]"}, {"name": "optimizer", "val": ": Optimizer"}, {"name": "num_warmup_steps", "val": ": typing.Optional[int] = None"}, {"name": "num_training_steps", "val": ": typing.Optional[int] = None"}]</parameters><paramsdesc>- **name** (`str` or `SchedulerType`) --
  The name of the scheduler to use.
- **optimizer** (`torch.optim.Optimizer`) --
  The optimizer that will be used during training.
- **num_warmup_steps** (`int`, *optional*) --
  The number of warmup steps to do. This is not required by all schedulers (hence the argument being
  optional), the function will raise an error if it's unset and the scheduler type requires it.
- **num_training_steps** (`int``, *optional*) --
  The number of training steps to do. This is not required by all schedulers (hence the argument being
  optional), the function will raise an error if it's unset and the scheduler type requires it.</paramsdesc><paramgroups>0</paramgroups></docstring>

Unified API to get any scheduler from its name.




</div>

<div class="docstring">

<docstring><name>transformers.get\_constant\_schedule</name><anchor>transformers.get_constant_schedule</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L34</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "last_epoch", "val": ": int = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a constant learning rate, using the learning rate set in optimizer.






</div>

<div class="docstring">

<docstring><name>transformers.get\_constant\_schedule\_with\_warmup</name><anchor>transformers.get_constant_schedule_with_warmup</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L50</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "num_warmup_steps", "val": ": int"}, {"name": "last_epoch", "val": ": int = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **num_warmup_steps** (`int`) --
  The number of steps for the warmup phase.
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate
increases linearly between 0 and the initial lr set in the optimizer.






</div>

<img alt="" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/warmup_constant_schedule.png"/>

<div class="docstring">

<docstring><name>transformers.get\_cosine\_schedule\_with\_warmup</name><anchor>transformers.get_cosine_schedule_with_warmup</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L104</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "num_warmup_steps", "val": ": int"}, {"name": "num_training_steps", "val": ": int"}, {"name": "num_cycles", "val": ": float = 0.5"}, {"name": "last_epoch", "val": ": int = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **num_warmup_steps** (`int`) --
  The number of steps for the warmup phase.
- **num_training_steps** (`int`) --
  The total number of training steps.
- **num_cycles** (`float`, *optional*, defaults to 0.5) --
  The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0
  following a half-cosine).
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a learning rate that decreases following the values of the cosine function between the
initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the
initial lr set in the optimizer.






</div>

<img alt="" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/warmup_cosine_schedule.png"/>

<div class="docstring">

<docstring><name>transformers.get\_cosine\_with\_hard\_restarts\_schedule\_with\_warmup</name><anchor>transformers.get_cosine_with_hard_restarts_schedule_with_warmup</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L138</source><parameters>[{"name": "optimizer", "val": ": Optimizer"}, {"name": "num_warmup_steps", "val": ": int"}, {"name": "num_training_steps", "val": ": int"}, {"name": "num_cycles", "val": ": int = 1"}, {"name": "last_epoch", "val": ": int = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **num_warmup_steps** (`int`) --
  The number of steps for the warmup phase.
- **num_training_steps** (`int`) --
  The total number of training steps.
- **num_cycles** (`int`, *optional*, defaults to 1) --
  The number of hard restarts to use.
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a learning rate that decreases following the values of the cosine function between the
initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases
linearly between 0 and the initial lr set in the optimizer.






</div>

<img alt="" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/warmup_cosine_hard_restarts_schedule.png"/>

<div class="docstring">

<docstring><name>transformers.get\_linear\_schedule\_with\_warmup</name><anchor>transformers.get_linear_schedule_with_warmup</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L75</source><parameters>[{"name": "optimizer", "val": ""}, {"name": "num_warmup_steps", "val": ""}, {"name": "num_training_steps", "val": ""}, {"name": "last_epoch", "val": " = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **num_warmup_steps** (`int`) --
  The number of steps for the warmup phase.
- **num_training_steps** (`int`) --
  The total number of training steps.
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after
a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.






</div>

<img alt="" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/warmup_linear_schedule.png"/>

<div class="docstring">

<docstring><name>transformers.get\_polynomial\_decay\_schedule\_with\_warmup</name><anchor>transformers.get_polynomial_decay_schedule_with_warmup</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization.py#L173</source><parameters>[{"name": "optimizer", "val": ""}, {"name": "num_warmup_steps", "val": ""}, {"name": "num_training_steps", "val": ""}, {"name": "lr_end", "val": " = 1e-07"}, {"name": "power", "val": " = 1.0"}, {"name": "last_epoch", "val": " = -1"}]</parameters><paramsdesc>- **optimizer** (`Optimizer`) --
  The optimizer for which to schedule the learning rate.
- **num_warmup_steps** (`int`) --
  The number of steps for the warmup phase.
- **num_training_steps** (`int`) --
  The total number of training steps.
- **lr_end** (`float`, *optional*, defaults to 1e-7) --
  The end LR.
- **power** (`float`, *optional*, defaults to 1.0) --
  Power factor.
- **last_epoch** (`int`, *optional*, defaults to -1) --
  The index of the last epoch when resuming training.</paramsdesc><paramgroups>0</paramgroups><retdesc>`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.</retdesc></docstring>

Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the
optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the
initial lr set in the optimizer.



Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT
implementation at
https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37




</div>

<h3 id="transformers.WarmUp">Warmup (TensorFlow)</h3>

<div class="docstring">

<docstring><name>class transformers.WarmUp</name><anchor>transformers.WarmUp</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L24</source><parameters>[{"name": "initial_learning_rate", "val": ": float"}, {"name": "decay_schedule_fn", "val": ": typing.Callable"}, {"name": "warmup_steps", "val": ": int"}, {"name": "power", "val": ": float = 1.0"}, {"name": "name", "val": ": str = None"}]</parameters><paramsdesc>- **initial_learning_rate** (`float`) --
  The initial learning rate for the schedule after the warmup (so this will be the learning rate at the end
  of the warmup).
- **decay_schedule_fn** (`Callable`) --
  The schedule function to apply after the warmup for the rest of training.
- **warmup_steps** (`int`) --
  The number of steps for the warmup part of training.
- **power** (`float`, *optional*, defaults to 1) --
  The power to use for the polynomial warmup (defaults is a linear warmup).
- **name** (`str`, *optional*) --
  Optional name prefix for the returned tensors during the schedule.</paramsdesc><paramgroups>0</paramgroups></docstring>

Applies a warmup schedule on a given learning rate decay schedule.




</div>

<h2 id="gradient-strategies">Gradient Strategies</h2>

<h3 id="transformers.GradientAccumulator">GradientAccumulator (TensorFlow)</h3>

<div class="docstring">

<docstring><name>class transformers.GradientAccumulator</name><anchor>transformers.GradientAccumulator</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L282</source><parameters>[]</parameters></docstring>

Gradient accumulation utility. When used with a distribution strategy, the accumulator should be called in a
replica context. Gradients will be accumulated locally on each replica and without synchronization. Users should
then call `.gradients`, scale the gradients if required, and pass the result to `apply_gradients`.



<div class="docstring">
<docstring><name>reset</name><anchor>transformers.GradientAccumulator.reset</anchor><source>https://github.com/huggingface/transformers/blob/pr_15592/src/transformers/optimization_tf.py#L344</source><parameters>[]</parameters></docstring>
Resets the accumulated gradients on the current replica.

</div></div>
