TORCH.OPTIM.LR_SCHEDULER.CosineAnnealingWarmRestarts is a scheduler for SGDR.

  
      github.com
  
  
    pytorch/pytorch/blob/66e6571eb871bafe8d03419c17be260a9b06c32f/torch/optim/lr_scheduler.py#L656 3

                    momentum = max_momentum - base_height * self.scale_fn(cycle)
                else:
                    momentum = max_momentum - base_height * self.scale_fn(self.last_epoch)
                momentums.append(momentum)
            for param_group, momentum in zip(self.optimizer.param_groups, momentums):
                param_group['momentum'] = momentum


        return lrs




class CosineAnnealingWarmRestarts(_LRScheduler):
    r"""Set the learning rate of each parameter group using a cosine annealing
    schedule, where :math:`\eta_{max}` is set to the initial lr, :math:`T_{cur}`
    is the number of epochs since the last restart and :math:`T_{i}` is the number
    of epochs between two warm restarts in SGDR:


    .. math::
        \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 +
        \cos(\frac{T_{cur}}{T_{i}}\pi))


    When :math:`T_{cur}=T_{i}`, set :math:`\eta_t = \eta_{min}`.