I would indeed try to use separate optimizers here.
I think you could also replace for p in model.parameters(): p.grad = None to see if that skips the momentum update, but this is less obvious to the user so the next change might break it again.
Best regards
Thomas