Hi,
I’d like to train only several rows of a linear layer (the linear layer is a classification head on top of BERT for multi-label classification, and I’d like to only train the head further for 10 labels). I tried the following:
model.classifier.weight[other_labels_indices,:].requires_grad = False model.classifier.bias[other_labels_indices].requires_grad = False
In other words, I’d like to freeze the parameters of the classifier for the “other labels”, which already have good performance, and only train the parameters corresponding to the labels that are currently underperforming.
However, this gave me the following warning:
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
I then tried the following:
model.classifier.weight[other_labels_indices,:].detach_()
model.classifier.bias[other_labels_indices].detach_()

However, it seems that these parameters are still being updated. Am I doing something wrong?