I see, thanks @ptrblck ! So the only way to implement this then would be to define 4 disjoint groups: 1) biases/LayerNorm in model.base, 2) non biases/LayerNorm in model.base, 3) biases/LayerNorm in model.classifier, 4) non biases/LayerNorm in model.classifier.
Any suggestions on how to use a scheduler to change the learning rate for groups 1-2 together, and 3-4 together?