GPT Neo - no attention weights scaling in pytorch implementation of GPT Neo
It seems that there is no scaling of attention weights in GPT-Neo implementation of SelfAttention. Probably https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L188  should be modified as follows:
```
attn_weights = torch.matmul(query, key.transpose(-1, -2))
scale = self.head_dim ** -0.5
attn_weights = attn_weights * scale
```

Do you have a source available that says they do use scaling here? Though it is common practice, I don't think it is necessarily required.<|||||>Yes, it seems you are right. Originally they use mesh_tensorflow realization of multi-head self-attention in which i don't see such scaling.

But at the same time in flax realization (https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neo/modeling_flax_gpt_neo.py#L230) flax.linen.attention.dot_product_attention_weights is used. And in its source code there is scaling. But i don't think that this incosistency is crucial.

