transformers GPT-2 has wrong implementation in scale in Attention class
https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/model.py#L94
https://github.com/huggingface/transformers/blob/a14d62b0b11e6cb0c64059606e4f8dbf78e40e41/src/transformers/models/gpt2/modeling_gpt2.py#L196

w = w / (float(v.size(-1)) ** 0.5) or w = w * (float(v.size(-1)) ** 0.5) ??
@zixiliuUSC  `tf.rsqrt` stands for reciprocal of square root<|||||>> @zixiliuUSC `tf.rsqrt` stands for reciprocal of square root
Oh, I see. Thank you so much!
