facebook\opt layer norm
### System Info

transformers version 4.28.1.

I notice that in the facebook\optX models the LayerNorm weight is equal to 1 in all layers, means no parameter changed.

I checked the sizes 125m, 1.3b, 2.7b, 6.7b, 13b




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import  OPTModel
import torch

model = OPTModel.from_pretrained("facebook/opt-13b")
for m in model.modules():
      if isinstance(m,torch.nn.LayerNorm):
            (m.weight == 1).all()



### Expected behavior

I get: (Expected to get different values)

tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
tensor(True)
Hi @CompressTeam, thanks for raising this issue!

I believe this behaviour is likely coming from the fact that the layer norm layers are instantiated with `elementwise_affine=True` e.g. [here](https://github.com/huggingface/transformers/blob/fdd78d91532dffc4b2493d3b9bd9e19aaf78fe6b/src/transformers/models/opt/modeling_opt.py#L292) (as default [config value is `True`](https://github.com/huggingface/transformers/blob/fdd78d91532dffc4b2493d3b9bd9e19aaf78fe6b/src/transformers/models/opt/configuration_opt.py#L120)). This instantiates the layer with [all weight values as 1, and biases as 0](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html). 

Playing quickly with the snippet provided, I can see that the biases are all different values, so it would seem that either only the biases were updated when training the model, there's been a error in weight conversion or an issue with weight saving. 

I'll hand over to @younesbelkada who added the model as is most familiar with layer norm related logic like `config._remove_final_layer_norm`


<|||||>Hi @CompressTeam 

I think that this is expected, see this interesting thread from the authors: https://github.com/huggingface/transformers/issues/17653 and in particular these 2 messages: https://github.com/huggingface/transformers/issues/17653#issuecomment-1163065167 / https://github.com/huggingface/transformers/issues/17653#issuecomment-1163293340 from what I have understood the models somehow learned to get a layer norm of 1 <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.