You could define separate parts of the self.classifier parameter and only pass the parts, which should be optimized, to the optimizer. In the forward method you would then have to recreate the “full” parameter using torch.cat and/or torch.stack and apply it in the matmul.
Another approach would be to zero out the gradients of specific parts of the parameter, e.g. via register_hook, but this approach might be a bit tricky especially if your optimizer uses internal running stats.