Roberta’s pretraining is described below

BERT is optimized with Adam (Kingma and Ba,
2015) using the following parameters: β1 = 0.9,
β2 = 0.999, ǫ = 1e-6 and L2 weight decay of 0.01. The learning rate is warmed up
over the first 10,000 steps to a peak value of
1e-4, and then linearly decayed. BERT trains
with a dropout of 0.1 on all layers and attention weights, and a GELU activation function (Hendrycks and Gimpel, 2016). Models are
pretrained for S = 1,000,000 updates, with minibatches containing B = 256 sequences of maximum length T = 512 tokens.

I’m trying to figure out how to replicate this optimizer schedule. I see that in the trainer.py code there’s AdamW and get_linear_schedule_with_warmup

  
      github.com
  
  
    huggingface/transformers/blob/master/src/transformers/trainer.py#L510 17

    optimizer_grouped_parameters = [        {            "params": [p for n, p in self.model.named_parameters() if not any(nd in n for nd in no_decay)],            "weight_decay": self.args.weight_decay,        },        {            "params": [p for n, p in self.model.named_parameters() if any(nd in n for nd in no_decay)],            "weight_decay": 0.0,        },    ]    self.optimizer = AdamW(        optimizer_grouped_parameters,        lr=self.args.learning_rate,        betas=(self.args.adam_beta1, self.args.adam_beta2),        eps=self.args.adam_epsilon,    )if self.lr_scheduler is None:    self.lr_scheduler = get_linear_schedule_with_warmup(        self.optimizer, num_warmup_steps=self.args.warmup_steps, num_training_steps=num_training_steps    )



  
  
    
    
  
  


But I’m not sure how to replicate Roberta’s learning rate schedule from these classes

  
      
      huggingface.co
  
  
    

Optimization — transformers 3.5.0 documentation 7



  
  
    
    
  
  


It seems that AdamW already has the decay rate, so using AdamW with get_linear_schedule_with_warmup will result in two types of decay. So to me it makes more sense to use AdamW with get_constant_schedule_with_warmup.
I am also wondering how to set the schedule based on 1) a starting learning rate 2) warm it up to a particular maximum value 3) from the maximum value, decay using a particular decay rate.
The classes on the main optimization class seem to be based on warming up/decaying to/from zero.