Does run_lm_finetuning.py finetune the entire BERT / Xlnet architecture
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1. When finetuning on data without a task, i.e. **unsupervised finetuning** by running the **run_lm_finetuning.py** script, does the code finetune all the weight layers of the model or just finetunes by adding an extra layer over the top of the architecture? 

2. The correct way to do this is to first finetune the top n layers for some epochs and then finetune all the layers. Is this how the finetuning takes place through run_lm_finetuning.py ? 
I think answer is no. 

I went through the run_lm_finetuning.py code and it seems that the entire model is getting finetuned from the start.

Still having doubt on this as I have little experience with pytorch.
1) Yes, the entire model is fine-tuned.

2) We follow the fine-tuning that takes place in the BERT paper:
    > All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label.<|||||>Thanks.
<|||||>Why would _The correct way to do this is to first finetune the top n layers for some epochs and then finetune all the layers._ be the "one and only correct way to do this"? I've seen this being done to finetune VGG19 (IR), for different tasks, but I haven't see papers showing that one technique will result in better performance than others. Do you have a paper reference? As @LysandreJik indicates, the BERT paper indicates what the authors thing is the most efficient approach.<|||||>Follow up on this thread: in practice, if we have a small training set, would it make sense to only finetune the top n layers?
In that case, do we want to consider adding an option that controls the number of trainable layers?<|||||>You can also consider replacing the classification head first if you find a pre-trained model suiting your task..<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
