You could try to manipulate the scheduler.base_lrs using this code:
model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)

scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
    optimizer, T_0=10)

for epoch in range(40):
    print('scheduler ', scheduler.get_last_lr())
    print('optimizer ', optimizer.param_groups[0]['lr'])
    if (epoch+1) % 10 == 0:
        print('decrease')
        scheduler.base_lrs[0] = scheduler.base_lrs[0] * 0.5
        
    scheduler.step()

A quick test shows, that the optimizer’s learning rate for the default parameter group is also changed.
However, I haven’t tested the code, so you should double check, if the training is working as expected.