How i take an OpenAIGPTDoubleHeadsModel from run_language_modeling.py script? 
I train a gpt2 type model from scratch with run_language_modeling.py .
But i want to take an OpenAIGPTDoubleHeadsModel as my model.
My config.json is below. What i should change?

config = {
    "architectures": [
		"gpt2"
	],
    "model_name_or_path": None , 
    "model_type": "gpt2",
    "vocab_size":5000, 
    "n_positions":1024,
    "n_ctx":1024,
    "n_embd":768,
    "n_layer":6, 
    "n_head":12,
    "activation_function":'gelu_new',
    "resid_pdrop":0.1,
    "embd_pdrop":0.1,
    "attn_pdrop":0.1,
    "layer_norm_epsilon":1e-05,
    "initializer_range": 0.02,
    "summary_type":"cls_index",
    "summary_use_proj":True,
    "summary_activation":None,
    "summary_proj_to_labels":True,
    "summary_first_dropout":0.1,
    "bos_token_id":4999,
    "eos_token_id":4999,
}
What are you trying to do exactly? If using our scripts, you would usually pre-train the transformer model (without the heads, so it doesn't make sense to use a double heads model), and you would then fine-tune the model with the heads on a specific dataset. 

If you can explain a bit more what you're trying to do then I could guide you better towards the appropriate scripts.  <|||||>@LysandreJik I am new to all these , just i try to do things to learn. I saw this script:  https://github.com/huggingface/transfer-learning-conv-ai/blob/master/train.py 
and i try to do something similar for another language(non english).  
So my idea was, to start with pre-training a gpt2 with run_language_modeling, from scratch in a new language and after fine-tune it in dialogue.
From your answer, i think if i understood, first i must pretrain the gpt2 and after to fine-tune it with the heads in a spesific dataset as dialogue in my case.
But how after the pre-training i use the DoubleHeadsModel?
<|||||>That's right, that's how I would do it. I would use the `run_language_modeling.py` script with GPT-2 on your data (beware that pre-training requires a lot of data, and a lot of compute).

Once your model is trained on your *big* dataset, then you can use the double heads model and fine-tune it to dialog. We don't have a script for this now, but the link you shared alongside the [blog post](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) detail how to do it!

In order to do what I just mentioned, you would therefore need a large corpus for pre-training, and a dialog dataset in the specific language for fine-tuning.<|||||>thanks @LysandreJik. You cleared a litlle my mind at least for the pre-train. 
A small blind spot i have only in how i will pass to the double heads after the pre-training(in blog i don't think they say something for that).
Hope i will find an answer and to that  when i ll be in that spot.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
