You could freeze the backbone parameters by setting their .requires_grad attributes to False during the warmup iterations, and then to True during the complete training.
This would make sure that these parameters won’t get any gradients and the optimizer won’t update them in its step() operation.