Hello,
I was wondering the same thing with respect to huggingface transformers’ scheduler. I found that one of this library’s examples 22 addresses this in the scheduler constructor by dividing the “pre-accumulation” number of steps by gradient_accumulation_steps:
t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs
scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
)

They then call scheduler.step() immediately after each optimizer.step().
In your case with CyclicLR, I think you’ll want to divide your original step_size_up and step_size_down by gradient_accumulation_steps.