Does Pre-Trained Weights Work Internally in pytorch? 
I am using bert’s pretrained model in from_pretrained and coming across it’s fine tuning code we can save the new model weights and other hyper params in save_pretrained.
My doubt is in there modeling_bert code there is no explicit code that takes the pre-trained weights in acount and then trains as it generally takes attention matrices and puts it in a feed forward network in the class `BertPredictionHeadTransform`
```
class BertPredictionHeadTransform(nn.Module):
    def __init__(self, config):
        super(BertPredictionHeadTransform, self).__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        if isinstance(config.hidden_act, str) or (sys.version_info[0] == 2 and isinstance(config.hidden_act, unicode)):
            self.transform_act_fn = ACT2FN[config.hidden_act]
        else:
            self.transform_act_fn = config.hidden_act
        self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.transform_act_fn(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        print('BertPredictionHeadTransform', hidden_states.shape)
        return hidden_states

```
And here i do not see any kind of “inheritance” of the pre-trained weights…
So is it internally handled by pytorch or am i missing something in the code itself?


`BertPredictionHeadTransform` is never used in itself AFAIK, and is only part of the full scale models (e.g. `BertModel`). As such, the weights for the prediction head are loaded when you run BertModel.from_pretrained(), since the `BertPredictionHeadTransform` is only a module in the whole model.<|||||>> `BertPredictionHeadTransform` is never used in itself AFAIK, and is only part of the full scale models (e.g. `BertModel`). As such, the weights for the prediction head are loaded when you run BertModel.from_pretrained(), since the `BertPredictionHeadTransform` is only a module in the whole model.

Thanks, Bram. As you said `BertModel` does take `BertPreTrainedModel` in `super`. I did notice it, But It's just that my mind doesn't get around how/where and when those weights are getting used exactly.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
