Error when using Adafactor without learn rate
Hi,
I get these strange errors when I use the Adafactor.  This code will result in this (expected) error:
```
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=1e-4)
```

> ValueError: Cannot combine manual `lr` and `relative_step=True` options

however, if I do not set a manual learn rate I get a different error. Btw: This code is recommended in the [documentation](https://huggingface.co/transformers/main_classes/optimizer_schedules.html?highlight=others%20reported%20following%20combination%20work%20well).
```
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None)
# same for 
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True)
```
will return this error
> TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Linux
- Python version: 3.7.1
- PyTorch version (GPU?): 1.8.0+cu111 and 1.8.1+cu111
- Tensorflow version (GPU?):  -
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

Trainer: @sgugger 
This was added by @jsrozner and @stas00 in #10526, so pinging them here.<|||||>Thank you @sgugger for the feedback. I install the latest transformers version from source using: 
```pip install git+https://github.com/huggingface/transformers```

and set the recommended parameters from the patch:

optimizer = Adafactor(model.parameters(), scale_parameter=False, relative_step=True, warmup_init=True, lr=None)

> TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

However, the error message remains the same. Can you give me a hint where I can address this issue?

For reference this is the code that I am using:

```python

model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=len(label_2_id))

args = TrainingArguments(
    output_dir=f"models/{run_name}/checkpoints",
    run_name=run_name,    
    evaluation_strategy = "epoch",    
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    gradient_accumulation_steps=1,
    num_train_epochs=2,
    report_to=["tensorboard"],
    logging_dir='runs/'+run_name,
    logging_first_step=True,
    logging_steps=100,
    save_steps= 10000,
    save_total_limit=10,
    seed=16,     
    fp16=True
)

optimizer = Adafactor(model.parameters(), scale_parameter=False, relative_step=True, warmup_init=True, lr=None)
lrs = get_constant_schedule_with_warmup(optimizer,100)

data_collator = DataCollatorForTokenClassification(tokenizer)

trainer = Trainer(
    model,
    args,    
    train_dataset=tokenized_dataset_train,
    eval_dataset=tokenized_dataset_val,
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics_sklearn, 
    optimizers=(optimizer,lrs)
)

```


<|||||>@oliverguhr, please always post a full traceback for errors. It's impossible otherwise to know where the error came from, please refer to https://github.com/huggingface/transformers/blob/master/ISSUES.md#the-github-issues item (3). 

The actual recommendation is:
```
Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3)
```
The alternative one I saved because others said it worked well for them.

Once you post the full traceback then we can see why it fails.

Thank you!

p.s. colab notebook reproducing the problem is even better <|||||>Thanks for looking at this @stas00 

Here is a traceback and this [is a colab notebook to reproduce the issue](https://colab.research.google.com/drive/1DFsmXObv8JVvGRbX8uc6_sfPfbAnKKMx?usp=sharing).

Hint: Depending on setting

```
lrs = get_constant_schedule_with_warmup(optimizer,100)
```
or 

```
lrs = none
```
`
get_constant_schedule_with_warmup` fails directly or `trainer.train()`.

```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-18-031302865887> in <module>()
     12 
     13 optimizer = Adafactor(model.parameters(), scale_parameter=False, relative_step=True, warmup_init=True, lr=None)
---> 14 lrs = get_constant_schedule_with_warmup(optimizer,100)
     15 
     16 training_args = TrainingArguments(

5 frames

/usr/local/lib/python3.7/dist-packages/transformers/optimization.py in get_constant_schedule_with_warmup(optimizer, num_warmup_steps, last_epoch)
     67         return 1.0
     68 
---> 69     return LambdaLR(optimizer, lr_lambda, last_epoch=last_epoch)
     70 
     71 

/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, lr_lambda, last_epoch, verbose)
    201                     len(optimizer.param_groups), len(lr_lambda)))
    202             self.lr_lambdas = list(lr_lambda)
--> 203         super(LambdaLR, self).__init__(optimizer, last_epoch, verbose)
    204 
    205     def state_dict(self):

/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, last_epoch, verbose)
     75         self.verbose = verbose
     76 
---> 77         self.step()
     78 
     79     def state_dict(self):

/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in step(self, epoch)
    150             if epoch is None:
    151                 self.last_epoch += 1
--> 152                 values = self.get_lr()
    153             else:
    154                 warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)

/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in get_lr(self)
    249 
    250         return [base_lr * lmbda(self.last_epoch)
--> 251                 for lmbda, base_lr in zip(self.lr_lambdas, self.base_lrs)]
    252 
    253 

/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in <listcomp>(.0)
    249 
    250         return [base_lr * lmbda(self.last_epoch)
--> 251                 for lmbda, base_lr in zip(self.lr_lambdas, self.base_lrs)]
    252 
    253 

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

```<|||||>Thank you for creating the reproducible colab notebook, @oliverguhr - that's very helpful.

So when you use `Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=None)` the learning rate scheduling is performed internally by the optimizer and so there is no need for a scheduler. 

But I see that barebones HF Trainer doesn't support training w/o a scheduler. So we aren't quite supporting this option then and perhaps we should. 

Regardless of the outcome we should document the conclusion of this thread in the Adafactor docstring.

So here are a few ideas meanwhile:

1. Create a dummy scheduler that always returns a fixed lr, example:
```
from torch.optim.lr_scheduler import LambdaLR
class DummyLR(LambdaLR):
    def __init__(self, optimizer, lr=0):
        for group in optimizer.param_groups:
            group['initial_lr'] = lr
        lr_lambda = lambda x: lr
        super().__init__(optimizer, lr_lambda)
        for group in optimizer.param_groups:
            del group['initial_lr']

def get_dummy_schedule(optimizer):
    return DummyLR(optimizer)

lrs = get_dummy_schedule(optimizer) 
``` 

Let me know if this unblocks you a bit.

2. Alternatively, if you want to be able to access lr outside of optimizer, here a proxy scheduler that pulls the LR out of the optimizer at run time, rather than feeding the optimizer.

```
from torch.optim.lr_scheduler import LambdaLR
class AdafactorSchedule(LambdaLR):
    def __init__(self, optimizer, initial_lr=0):
        for group in optimizer.param_groups:
            group['initial_lr'] = initial_lr
        lr_lambda = lambda x: initial_lr
        super().__init__(optimizer, lr_lambda)
        for group in optimizer.param_groups:
            del group['initial_lr']

    def get_lr(self):
        opt = self.optimizer
        lrs = [opt._get_lr(group, opt.state[group["params"][0]]) for group in opt.param_groups if group["params"][0].grad is not None]
        if len(lrs) == 0:
            lrs = self.base_lrs # if called before stepping
        # print(f"lr={lrs}")
        return lrs

def get_adafactor_schedule(optimizer):
    return AdafactorSchedule(optimizer)


optimizer = Adafactor(model.parameters(), scale_parameter=False, relative_step=True, warmup_init=True, lr=None)
scheduler = get_adafactor_schedule(optimizer) 
```

clearly this is a quick hack, but it seems to work. it returns `initial_lr` during startup and the actual `lr` during stepping (disable the debug print to see).

As you can see I had to hack `initial_lr` into it since optimizer doesn't have any lr until it starts stepping.

If this is desired than we could add `Adafactor.get_scheduler()` which would return the above. Perhaps it needs to assert if `lr != None`  I haven't looked that close.

If you like the 2nd solution feel free to clean it up, and making a PR, perhaps getting rid of `LambdaLR` to not need the `group['initial_lr']` hack, going straight for the ` _LRScheduler` super class.

3. Make HF `Trainer` support `scheduler=None` - that would be hard for the loggers and other places that expect being able to get the value for lr. I think a clean version of the 2nd solution is probably more suitable.
<|||||>So @sgugger suggests the 3rd option. For that will have to track down all the cases where the scheduler is used and condition those on `scheduler != None`

Not sure about back-compat though since we auto-create a scheduler if it's not passed:

https://github.com/huggingface/transformers/blob/33fd83bc01e781633dad58a9de6c91591e2fc786/src/transformers/trainer.py#L817-L829
<|||||>Another proposition from @sgugger is:
>  this can be handled with the `lr_scheduler_type` argument: we could add an acceptable value "no"  that would leave the scheduler at `None`.<|||||>@stas00 Sorry for the late reply and thanks for your feedback. The DummyLR worked for me, but this parameter combination did not improve my results, maybe these parameter settings are kind of an edge case. 

Regarding the 3rd option: Would it possible to check if ```lr_scheduler is None``` and ```optimizer is Adafactor``` and then auto-create an instance of the "AdafactorScheduler"? This could eliminate the need to check all the other parts of the code that rely on the LR value from the opimzier. <|||||>> Regarding the 3rd option: Would it possible to check if lr_scheduler is None and optimizer is Adafactor and then auto-create an instance of the "AdafactorScheduler"? This could eliminate the need to check all the other parts of the code that rely on the LR value from the opimzier.

@sgugger, what is your take - `AdafactorScheduler` is the hack I posted here: https://github.com/huggingface/transformers/issues/11612#issuecomment-833888170

I'm happy with either way, but let's resolve it one way or another.<|||||>Mmm, the `lr_scheduler` is always None by default. Can we add a value `"adafactor"` that would use that `AdafactorScheduler`?<|||||>@oliverguhr, we went with the `AdafactorSchedule` - please check that it works for you

https://github.com/huggingface/transformers/pull/12123