How do I train OpenAIGPTDoubleHeadsModel from scratch?
## ❓ Questions & Help

It looks like I cannot train `OpenAIGPTDoubleHeadsModel` from scratch because it necessarily needs to be initialized using the `from_pretrained()` method. How should I initialize this model to be able to train it from scratch?

As a hack, perhaps I could initialize it using `from_pretrained()` and then reset the pre-trained initialization so any fine-tuning would essentially be the same as pre-training?
You can initialize a model by calling its constructor with a configuration:

```py
from transformers import OpenAIGPTDoubleHeadsModel, OpenAIGPTConfig

config = OpenAIGPTConfig()
model = OpenAIGPTDoubleHeadsModel(config)
```<|||||>Cool, so if I instantiate the model with the default config, it will use the same vocab as the pre-trained model but it won't initialize the pre-trained weights, is that correct?

Also, could you please point me to where the model weights are initialized when we don't use pre-trained weights?<|||||>If you instantiate a model with a default configuration without specifying any argument, it will instantiate your model according to the default configuration values, [visible here](https://github.com/huggingface/transformers/blob/master/transformers/configuration_openai.py#L59).

The model weights are initialized in the `OpenAIGPTPreTrainedModel`'s [`_init_weights` method](https://github.com/huggingface/transformers/blob/master/transformers/modeling_openai.py#L267).