How does arg --vocab_transform help in extract_distilbert.py?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi everyone, I'm new to experiment with bert model distillation. When running extract_distilbert.py on my fine tuned bert model, I came across the argusment vocat_transform.
```
if args.vocab_transform:
        for w in ['weight', 'bias']:
            compressed_sd[f'vocab_transform.{w}'] = state_dict[f'cls.predictions.transform.dense.{w}']
            compressed_sd[f'vocab_layer_norm.{w}'] = state_dict[f'cls.predictions.transform.LayerNorm.{w}']
```
When should we use this argument in running extract_distilbert? Any scenario we could benefit doing so?
Thanks!
Hello @evehsu,

BERT uses an additional non-linearity before the vocabulary projection (see [here](https://github.com/huggingface/transformers/blob/master/transformers/modeling_bert.py#L381)).
It's a design choice, as far as I know, XLM doesn't a non-linearity right before the vocab projection (the language modeling head).

I left this option because I experimented with it, but if you want to keep the BERT architecture as unchanged as possible, you should use the `--vocab_transform` to ensure you also extract the pre-trained weights for this non-linearity.

Victor<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
