XLNet - Finetuning - Layer-wise LR decay
## ❓ Questions & Help
I'm trying to finetuning a XLNet using run_glue.py, but i haven't seen any references about **Layer-wise lr decay**, that were commented by the authors in the paper. 

- Where can I set this parameter on finetuning optimizer?
- The *linear learning rate decay* commented in the paper is related to Warmup Scheduler ?(considering that after warmup_steps is reached, the lr rate begins to decay) 

References:
(https://arxiv.org/pdf/1906.08237.pdf - page 16)

No this means the layer rate is smaller deeper in the network, what is called "discriminative learning" in ULMFiT.
Check our NAACL Tutorial on Transfer Learning for more details, in particular, Hands-on n°5 slide 163 here: https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/edit?ts=5c8d09e7#slide=id.g5888218f39_54_89 <|||||>Thanks @thomwolf for the answer.