From further looking into the code for Roberta (https://github.com/pytorch/fairseq/blob/dd52ed0f3896639b3c04aa67c44775f689faf1a5/fairseq/optim/lr_scheduler/polynomial_decay_schedule.py 16) and also Bert (https://github.com/google-research/bert/blob/master/optimization.py#L36 8)
It seems that the learning rate starts what is specified in the optimizer, increased to a particular LR, and then linearly decreased to zero. It seems that get_linear_schedule_with_warmup could work, but would need to be altered for a different learning rate.
It seems that it uses torch.optim.lr_scheduler.LambdaLR

  
      github.com
  
  
    huggingface/transformers/blob/08f534d2da47875a4b7eb1c125cfa7f0f3b79642/src/transformers/optimization.py#L90 4

            The number of steps for the warmup phase.        num_training_steps (:obj:`int`):            The total number of training steps.        last_epoch (:obj:`int`, `optional`, defaults to -1):            The index of the last epoch when resuming training.    Return:        :obj:`torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.    """    def lr_lambda(current_step: int):        if current_step < num_warmup_steps:            return float(current_step) / float(max(1, num_warmup_steps))        return max(            0.0, float(num_training_steps - current_step) / float(max(1, num_training_steps - num_warmup_steps))        )    return LambdaLR(optimizer, lr_lambda, last_epoch)def get_cosine_schedule_with_warmup(



  
  
    
    
  
  


So I’m thinking of creating a custom function directly to use that method

  
      pytorch.org
  
  
    

torch.optim — PyTorch 1.7.0 documentation 4