question for one parameter matrix in transformers/GPT2
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

In transformers/gpt-2 model, there's a weight matrix called "transformer.h.0.attn.bias" whose size is torch.Size([1, 1, 1024, 1024]). I checked the original paper but still get confused by what it is for? 
This parameter matrix is between the Layernomalization layer and the attention layer. Can anyone explain that? Thank you in advance.

I dont think something like that is there, please have a look. Or paste everything from h.0 as it is here. Then it will be easy.<|||||>> I dont think something like that is there, please have a look. Or paste everything from h.0 as it is here. Then it will be easy.

Hi, thank you for your reply. Here's the parameter list from embedding layer to the first decoder layer.

['transformer.wte.weight', 'transformer.wpe.weight', 'transformer.h.0.ln_1.weight', 'transformer.h.0.ln_1.bias', 'transformer.h.0.attn.bias', 'transformer.h.0.attn.c_attn.weight', 'transformer.h.0.attn.c_attn.bias', 'transformer.h.0.attn.c_proj.weight', 'transformer.h.0.attn.c_proj.bias', 'transformer.h.0.ln_2.weight', 'transformer.h.0.ln_2.bias', 'transformer.h.0.mlp.c_fc.weight', 'transformer.h.0.mlp.c_fc.bias', 'transformer.h.0.mlp.c_proj.weight', 'transformer.h.0.mlp.c_proj.bias']

The fifth item is 'transformer.h.0.attn.bias'. I guess it may be some random noise but I can't find any reference for that.<|||||>Hi! Indeed there is a `bias` item in the attention layer. The name is probably not as accurate as it could be, as it does not represent a bias but a triangular matrix that is used when computing the attention score.

As a causal language model, GPT-2 should only look at its left context. This triangular matrix makes sure that the values on the right of the focused token are set to zero so that they do not affect the resulting attention score.

You don't need to worry about this matrix as the model initializes it on its own :).<|||||>> Hi! Indeed there is a `bias` item in the attention layer. The name is probably not as accurate as it could be, as it does not represent a bias but a triangular matrix that is used when computing the attention score.
> 
> As a causal language model, GPT-2 should only look at its left context. This triangular matrix makes sure that the values on the right of the focused token are set to zero so that they do not affect the resulting attention score.
> 
> You don't need to worry about this matrix as the model initializes it on its own :).

Hi, thank you for your reply. So the 'attn.bias' is actually used as the masking matrix for attention here. That's why its size is 1024, which is the length of the context in gpt2. Hope this can help others!