Hi @abhilashpal, you can find distillation code here 29. The same script that produces distillbert can be used for GPT-2, it’s not documented though.
you should be able to use this command after processing your dataset
python train.py \
    --student_type gpt2 \
    --student_config training_configs/distilgpt2.json \
    --teacher_type gpt2 \
    --teacher_name gpt2 \ # or your own teacher model
    --alpha_ce 5.0  --alpha_cos 1.0 --alpha_clm 0.5 \
    --freeze_pos_embs \
    --dump_path serialization_dir/my_first_training \
    --data_file data/binarized_text.bert-base-uncased.pickle \ # your data path
    --token_counts data/token_counts.bert-base-uncased.pickle \ # your own pickle file path
    --force # overwrites the `dump_path` if it already exists.

pinging @julien-c for more info