gpt-2 implement issue
Thanks for your good implementation some model in pytorch!
gpt-2 paper mentioned that they did few  modifications with original gpt, included "A modified initialization which accounts
for the accumulation on the residual path with model depth
is used. We scale the weights of residual layers at initialization by a factor of 1/
√
N where N is the number of
residual layers."
I assume that will help in training, it is crucial to reimplement the gpt2, so consider add this to repo? 

Hi, this is for the initialization. We don't have any scripts that show how to pretrain GPT-2 (therefore no need for initialization), only scripts to fine-tune it from a checkpoint.<|||||>thanks for your reply
