how to freeze bert model and just train a classifier?
How to freeze all layers of bert and just train task based classifier?
Hi the BERT models are regular PyTorch models, you can just use the usual way we freeze layers in PyTorch. For example you can have a look at the [Transfer Learning tutorial of PyTorch](https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor).

In our case freezing the pretrained part of a `BertForSequenceClassification` model would look like this
```python
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False
```
Then only the classification layer should be have `requires_grad=True`.<|||||>Wouldn't this solution freeze all layers, including the classifier layer?<|||||>@nuno-carneiro  @thomwolf 
I think, this  will freeze all the layers including the classifier layer. (Correct me, if I'm wrong)  
```
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False
```

**I think the below code will freeze only the BERT layers (Correct me, if I'm wrong)**

```
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.bert.parameters():
    param.requires_grad = False
```<|||||>You are right @kalyanks0611 <|||||>AttributeError: 'BertModel' object has no attribute 'bert'

I got the above error when doing 

for param in model.bert.bert.parameters():
    param.requires_grad = False

Any clue? Thanks :)<|||||>@yhl48 Hi, please open a new issue with the problem you're facing. Thank you!<|||||>does anyone know how to do this for the TF 2.0 version too?<|||||>> AttributeError: 'BertModel' object has no attribute 'bert'
> 
> I got the above error when doing
> 
> for param in model.bert.bert.parameters():
> param.requires_grad = False
> 
> Any clue? Thanks :)

I got the same error. This is my workaround following the one in #1431 :

	for name, param in model.named_parameters():
		if 'classifier' not in name: # classifier layer
			param.requires_grad = False<|||||>You're probably instantiating your model as a `BertModel` whereas @kalyanks0611's example uses `BertForSequenceClassification`.<|||||>> does anyone know how to do this for the TF 2.0 version too?

This is the real question<|||||>> does anyone know how to do this for the TF 2.0 version too?

Actually in tf2 framework, pre-trained is saved in `model.bert.weights`, which is a list. So you need to do:
```
for w in model.bert.weights():
        w._trainable= False
```
<|||||>> > does anyone know how to do this for the TF 2.0 version too?
> 
> This is the real question

see [my post](https://github.com/huggingface/transformers/issues/400#issuecomment-571354368)<|||||>> does anyone know how to do this for the TF 2.0 version too?

model.bert.trainable = False
Is this ok?
But I found that the training speed has not accelerated<|||||>> > does anyone know how to do this for the TF 2.0 version too?
> 
> Actually in tf2 framework, pre-trained is saved in `model.bert.weights`, which is a list. So you need to do:
> 
> ```
> for w in model.bert.weights():
>         w._trainable= False
> ```

Thanks! I needed a very slight amendment for it to work.

```
for w in model.get_layer('tf_distil_bert_model').weights:
    w._trainable = False
```

Works for me. And training time is halved from 9 hours to 4 hours.<|||||>I am trying to let bert model layer 10 and layer 11 are trainable, like below:
```shell
bert_model = TFBertModel.from_pretrained("bert-base-uncased")
for w in bert_model.bert.weights:
  if w.name.find('layer_._10') == -1 and w.name.find('layer_._11') == -1:
    print(w.name)
    w._trainable = False
```
But the result is as below:
```shell
Total params: 109,483,776
Trainable params: 109,483,776
Non-trainable params: 0
```
How can i change the specific layers to be trainable?

Thanks<|||||>> 
> 
> I am trying to let bert model layer 10 and layer 11 are trainable, like below:
> 
> ```shell
> bert_model = TFBertModel.from_pretrained("bert-base-uncased")
> for w in bert_model.bert.weights:
>   if w.name.find('layer_._10') == -1 and w.name.find('layer_._11') == -1:
>     print(w.name)
>     w._trainable = False
> ```
> 
> But the result is as below:
> 
> ```shell
> Total params: 109,483,776
> Trainable params: 109,483,776
> Non-trainable params: 0
> ```
> 
> How can i change the specific layers to be trainable?
> 
> Thanks

The codes below would successfully freezes layers in a model:
```
from transformers import TFXLNetModel as transformerModel
transformer_model_name = 'xlnet-base-cased'
transformer_model = transformerModel.from_pretrained(transformer_model_name)
for layer in transformer_model.layers:
    layer.trainable = False
```<|||||>> @nuno-carneiro @thomwolf
> I think, this will freeze all the layers including the classifier layer. (Correct me, if I'm wrong)
> 
> ```
> model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
> 
> for param in model.bert.parameters():
>     param.requires_grad = False
> ```
> 
> **I think the below code will freeze only the BERT layers (Correct me, if I'm wrong)**
> 
> ```
> model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
> 
> for param in model.bert.bert.parameters():
>     param.requires_grad = False
> ```

Hi, I have 

> @nuno-carneiro @thomwolf
> I think, this will freeze all the layers including the classifier layer. (Correct me, if I'm wrong)
> 
> ```
> model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
> 
> for param in model.bert.parameters():
>     param.requires_grad = False
> ```
> 
> **I think the below code will freeze only the BERT layers (Correct me, if I'm wrong)**
> 
> ```
> model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
> 
> for param in model.bert.bert.parameters():
>     param.requires_grad = False
> ```

I think there are 2 elements in BertForSequenceClassification. One is bert and another one is classifier？or? So I think we dont need model.bert.bert.parameter(). We just need model.bert.parameters()? or?<|||||>`model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`

When you run `model` you will get the following architecture (I skipped many intermediate modules)
```
BertForSequenceClassification(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      ....
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          ...
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          ...
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=2, bias=True)
)
```

So basically  `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).


Therefore, the following code
```
for param in model.bert.bert.parameters():
    param.requires_grad = False
```
should be

```
for param in model.bert.parameters():
    param.requires_grad = False
```<|||||>> `model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> 
> When you run `model` you will get the following architecture (I skipped many intermediate modules)
> 
> ```
> BertForSequenceClassification(
>   (bert): BertModel(
>     (embeddings): BertEmbeddings(
>       ....
>     (encoder): BertEncoder(
>       (layer): ModuleList(
>         (0): BertLayer(
>           ...
>             (output): BertSelfOutput(
>               (dense): Linear(in_features=768, out_features=768, bias=True)
>               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
>               (dropout): Dropout(p=0.1, inplace=False)
>             )
>           )
>           ...
>     (pooler): BertPooler(
>       (dense): Linear(in_features=768, out_features=768, bias=True)
>       (activation): Tanh()
>     )
>   )
>   (dropout): Dropout(p=0.1, inplace=False)
>   (classifier): Linear(in_features=768, out_features=2, bias=True)
> )
> ```
> 
> So basically `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).
> 
> Therefore, the following code
> 
> ```
> for param in model.bert.bert.parameters():
>     param.requires_grad = False
> ```
> 
> should be
> 
> ```
> for param in model.bert.parameters():
>     param.requires_grad = False
> ```

I think you are right. And I think there are no model.bert.bert? Or?<|||||>> > `model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> > When you run `model` you will get the following architecture (I skipped many intermediate modules)
> > ```
> > BertForSequenceClassification(
> >   (bert): BertModel(
> >     (embeddings): BertEmbeddings(
> >       ....
> >     (encoder): BertEncoder(
> >       (layer): ModuleList(
> >         (0): BertLayer(
> >           ...
> >             (output): BertSelfOutput(
> >               (dense): Linear(in_features=768, out_features=768, bias=True)
> >               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
> >               (dropout): Dropout(p=0.1, inplace=False)
> >             )
> >           )
> >           ...
> >     (pooler): BertPooler(
> >       (dense): Linear(in_features=768, out_features=768, bias=True)
> >       (activation): Tanh()
> >     )
> >   )
> >   (dropout): Dropout(p=0.1, inplace=False)
> >   (classifier): Linear(in_features=768, out_features=2, bias=True)
> > )
> > ```
> > 
> > 
> > So basically `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).
> > Therefore, the following code
> > ```
> > for param in model.bert.bert.parameters():
> >     param.requires_grad = False
> > ```
> > 
> > 
> > should be
> > ```
> > for param in model.bert.parameters():
> >     param.requires_grad = False
> > ```
> 
> I think you are right. And I think there are no model.bert.bert? Or?

I think so, if you want to freeze bert part in bert model, u just say 
 for param in model.bert.parameters():
 param.requires_grad = False
but how are about optizmer ? 

is look like in this way ? 
    optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)<|||||>> > > `model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> > > When you run `model` you will get the following architecture (I skipped many intermediate modules)
> > > ```
> > > BertForSequenceClassification(
> > >   (bert): BertModel(
> > >     (embeddings): BertEmbeddings(
> > >       ....
> > >     (encoder): BertEncoder(
> > >       (layer): ModuleList(
> > >         (0): BertLayer(
> > >           ...
> > >             (output): BertSelfOutput(
> > >               (dense): Linear(in_features=768, out_features=768, bias=True)
> > >               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
> > >               (dropout): Dropout(p=0.1, inplace=False)
> > >             )
> > >           )
> > >           ...
> > >     (pooler): BertPooler(
> > >       (dense): Linear(in_features=768, out_features=768, bias=True)
> > >       (activation): Tanh()
> > >     )
> > >   )
> > >   (dropout): Dropout(p=0.1, inplace=False)
> > >   (classifier): Linear(in_features=768, out_features=2, bias=True)
> > > )
> > > ```
> > > 
> > > 
> > > So basically `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).
> > > Therefore, the following code
> > > ```
> > > for param in model.bert.bert.parameters():
> > >     param.requires_grad = False
> > > ```
> > > 
> > > 
> > > should be
> > > ```
> > > for param in model.bert.parameters():
> > >     param.requires_grad = False
> > > ```
> > 
> > 
> > I think you are right. And I think there are no model.bert.bert? Or?
> 
> I think so, if you want to freeze bert part in bert model, u just say
> for param in model.bert.parameters():
> param.requires_grad = False
> but how are about optizmer ?
> 
> is look like in this way ?
> optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)

I think you have written right code. But we should write usually 2 parts together. I mean:
for param in model.bert.parameters():
param.requires_grad = False
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)
Cuz we need to know which parameters are frozen. <|||||>> > > > `model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> > > > When you run `model` you will get the following architecture (I skipped many intermediate modules)
> > > > ```
> > > > BertForSequenceClassification(
> > > >   (bert): BertModel(
> > > >     (embeddings): BertEmbeddings(
> > > >       ....
> > > >     (encoder): BertEncoder(
> > > >       (layer): ModuleList(
> > > >         (0): BertLayer(
> > > >           ...
> > > >             (output): BertSelfOutput(
> > > >               (dense): Linear(in_features=768, out_features=768, bias=True)
> > > >               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
> > > >               (dropout): Dropout(p=0.1, inplace=False)
> > > >             )
> > > >           )
> > > >           ...
> > > >     (pooler): BertPooler(
> > > >       (dense): Linear(in_features=768, out_features=768, bias=True)
> > > >       (activation): Tanh()
> > > >     )
> > > >   )
> > > >   (dropout): Dropout(p=0.1, inplace=False)
> > > >   (classifier): Linear(in_features=768, out_features=2, bias=True)
> > > > )
> > > > ```
> > > > 
> > > > 
> > > > So basically `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).
> > > > Therefore, the following code
> > > > ```
> > > > for param in model.bert.bert.parameters():
> > > >     param.requires_grad = False
> > > > ```
> > > > 
> > > > 
> > > > should be
> > > > ```
> > > > for param in model.bert.parameters():
> > > >     param.requires_grad = False
> > > > ```
> > > 
> > > 
> > > I think you are right. And I think there are no model.bert.bert? Or?
> > 
> > 
> > I think so, if you want to freeze bert part in bert model, u just say
> > for param in model.bert.parameters():
> > param.requires_grad = False
> > but how are about optizmer ?
> > is look like in this way ?
> > optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)
> 
> I think you have written right code. But we should write usually 2 parts together. I mean:
> for param in model.bert.parameters():
> param.requires_grad = False
> optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)
> Cuz we need to know which parameters are frozen.

sorry, I do not know what do you mean?
so let say, if we just want to freeze the input part ( input embedding). how can I write the code? 
bert = BertModel.from_pretrained('bert-base-uncased')

for name, param in bert.named_parameters():                
    if name.startswith('embeddings'):
        param.requires_grad = False
this above three line, is tell model do not train the embedding right?

but how to tell the optimizer to do not change the embedding?

<|||||>> > > > > `model = BertForSequenceClassification.from_pretrained('bert-base-uncased')`
> > > > > When you run `model` you will get the following architecture (I skipped many intermediate modules)
> > > > > ```
> > > > > BertForSequenceClassification(
> > > > >   (bert): BertModel(
> > > > >     (embeddings): BertEmbeddings(
> > > > >       ....
> > > > >     (encoder): BertEncoder(
> > > > >       (layer): ModuleList(
> > > > >         (0): BertLayer(
> > > > >           ...
> > > > >             (output): BertSelfOutput(
> > > > >               (dense): Linear(in_features=768, out_features=768, bias=True)
> > > > >               (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
> > > > >               (dropout): Dropout(p=0.1, inplace=False)
> > > > >             )
> > > > >           )
> > > > >           ...
> > > > >     (pooler): BertPooler(
> > > > >       (dense): Linear(in_features=768, out_features=768, bias=True)
> > > > >       (activation): Tanh()
> > > > >     )
> > > > >   )
> > > > >   (dropout): Dropout(p=0.1, inplace=False)
> > > > >   (classifier): Linear(in_features=768, out_features=2, bias=True)
> > > > > )
> > > > > ```
> > > > > 
> > > > > 
> > > > > So basically `model` has 3 main submodules `bert`, `dropout`, and `classifier` (you can see this from the indentation as well.). Try running `model.bert`, `model.classifier`. When you call `model.bert `and freeze all the params, it will freeze entire encoder blocks(12 of them).
> > > > > Therefore, the following code
> > > > > ```
> > > > > for param in model.bert.bert.parameters():
> > > > >     param.requires_grad = False
> > > > > ```
> > > > > 
> > > > > 
> > > > > should be
> > > > > ```
> > > > > for param in model.bert.parameters():
> > > > >     param.requires_grad = False
> > > > > ```
> > > > 
> > > > 
> > > > I think you are right. And I think there are no model.bert.bert? Or?
> > > 
> > > 
> > > I think so, if you want to freeze bert part in bert model, u just say
> > > for param in model.bert.parameters():
> > > param.requires_grad = False
> > > but how are about optizmer ?
> > > is look like in this way ?
> > > optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)
> > 
> > 
> > I think you have written right code. But we should write usually 2 parts together. I mean:
> > for param in model.bert.parameters():
> > param.requires_grad = False
> > optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)
> > Cuz we need to know which parameters are frozen.
> 
> sorry, I do not know what do you mean?
> so let say, if we just want to freeze the input part ( input embedding). how can I write the code?
> bert = BertModel.from_pretrained('bert-base-uncased')
> 
> for name, param in bert.named_parameters():
> if name.startswith('embeddings'):
> param.requires_grad = False
> this above three line, is tell model do not train the embedding right?
> 
> but how to tell the optimizer to do not change the embedding?

these three lines have set False in 'Embedding' right? Then we need just to write what you have written:
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)<|||||>Are any of the layers in a pre-trained BERT model originally frozen? If not are there any supporting articles or research papers that I can look to? <|||||>> Are any of the layers in a pre-trained BERT model originally frozen? If not are there any supporting articles or research papers that I can look to?

I dont think there are any of layers originally frozen. You can read some Papers about Bert firstly, then in Pytoch forum or Tensorflow you could learn how to freeze the layers.<|||||>In tensorflow, other than the classifier layers, to freeze some layers in a underlying base transformer model something like this maybe needed(corrections are welcome):

```
bertmodel = TFAutoModelForSequenceClassification.from_pretrained(bertModelName, num_labels=output_num_classes, from_pt=True)

for i,v in enumerate(bertmodel.layers[0].variables):
  if i>=19:
    bertmodel.layers[0].variables[i] = tf.stop_gradient(v)
    print("do freeze {} {}".format(i, v.name))
  else:
    print("not freeze {} {}".format(i, v.name))
```<|||||>If I want to freeze the entire model, I can use model.trainable = False and vice vs, however, if I want to freeze part of it, I am setting that up in the loop. It's still not updating the trainable flag. I am using TensorFlow. Any thoughts? 

```
# Adjust the trainable layer weights based on retrain_layer_count
    # If retrain_layer_count is 0, then base model is frozen.
    # If retrain_layer_count is 12, then the entire base model is trainable.
    # And that implies that all the pretrained weights are lost and it relearns
    # from the input data.
    # If retrain_layer_count is between 1 and 11, then the last n layers of
    # the pretrained model retrained.
    if retrain_layer_count == 0:
        # The pretained model is frozen
        model.trainable = False           

    elif retrain_layer_count == 12:  
        # The pretrained model is retrained thru all layers.       
        model.trainable = True     

    else:    
        # Restrict training to the num_train_layers outer transformer layers
        retrain_layer_list = []
        #model.trainable = False  
        for retrain_layer_number in range(retrain_layer_count):

            layer_code = '_' + str(11 - retrain_layer_number)
            retrain_layer_list.append(layer_code)
        
        print('Retrain layers: \n', retrain_layer_list)
        print("After adjusting...")
        for layer in model.weights:
            layer._trainable = False
            print("***", layer.name, layer._trainable)
            if 'layer_' in layer.name and layer.name.split(".")[1].split("/")[0] in retrain_layer_list:
                layer._trainable = True
                print("$$$", layer.name, layer._trainable)
            elif 'layer_' not in layer.name :
                layer._trainable = True
                print("###", layer.name, layer._trainable)
        
        #for weight_details in model.weights:
        #    print(weight_details.name, weight_details._trainable)
        print(f"Number of trainable parameters : {count_params(model.trainable_weights)}")
        print(f"Number of non-trainable parameters : {count_params(model.non_trainable_variables)}")
```
    
    <|||||>> If I want to freeze the entire model, I can use model.trainable = False and vice vs, however, if I want to freeze part of it, I am setting that up in the loop. It's still not updating the trainable flag. I am using TensorFlow. Any thoughts?
> 
> ```
> # Adjust the trainable layer weights based on retrain_layer_count
>     # If retrain_layer_count is 0, then base model is frozen.
>     # If retrain_layer_count is 12, then the entire base model is trainable.
>     # And that implies that all the pretrained weights are lost and it relearns
>     # from the input data.
>     # If retrain_layer_count is between 1 and 11, then the last n layers of
>     # the pretrained model retrained.
>     if retrain_layer_count == 0:
>         # The pretained model is frozen
>         model.trainable = False           
> 
>     elif retrain_layer_count == 12:  
>         # The pretrained model is retrained thru all layers.       
>         model.trainable = True     
> 
>     else:    
>         # Restrict training to the num_train_layers outer transformer layers
>         retrain_layer_list = []
>         #model.trainable = False  
>         for retrain_layer_number in range(retrain_layer_count):
> 
>             layer_code = '_' + str(11 - retrain_layer_number)
>             retrain_layer_list.append(layer_code)
>         
>         print('Retrain layers: \n', retrain_layer_list)
>         print("After adjusting...")
>         for layer in model.weights:
>             layer._trainable = False
>             print("***", layer.name, layer._trainable)
>             if 'layer_' in layer.name and layer.name.split(".")[1].split("/")[0] in retrain_layer_list:
>                 layer._trainable = True
>                 print("$$$", layer.name, layer._trainable)
>             elif 'layer_' not in layer.name :
>                 layer._trainable = True
>                 print("###", layer.name, layer._trainable)
>         
>         #for weight_details in model.weights:
>         #    print(weight_details.name, weight_details._trainable)
>         print(f"Number of trainable parameters : {count_params(model.trainable_weights)}")
>         print(f"Number of non-trainable parameters : {count_params(model.non_trainable_variables)}")
> ```

I ran into this issue too, where if I first freeze all layers, and then selectively make some trainable, it does not work. Do it the other way round: set everything trainable and then freeze the layers you want!