IBert: What would be the possible reason `IntLayerNorm` does not decrease the loss?
### Who can help
@kssteven418 

## Information
IBert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Problem
Hello. I'm trying to use `IntLayerNorm` in my model. The model without the layer trains properly, but if I add the layer as follow the model suddenly stops being learned (loss does not change). What would be the possible reason?

```python
def forward(self, x, k: int = None):
        x, scaling_factor = self.pre_linear_act(x)                                # QuantAct
        x, scaling_factor = self.linear(x, scaling_factor)                     # QuantLinear
        x, scaling_factor = self.post_linear_act(x, scaling_factor)      # QuantAct

        # normalize
        if self.normalize:
            x, scaling_factor = self.layer_norm(x, scaling_factor)               # IntLayerNorm
            x, scaling_factor = self.post_layernorm_act(x, scaling_factor) # QuantAct
```

## Update
I observed that the output tensor of `self.layer_norm` is all zero.
It is likely that the IntLayerNorm layers are not warmed up. IntLayerNorm layer has to adjust its internal parameter (`self.shift`) during the quantization-aware training process. It is the one that prevents overflow (i.e. keeps the internal activation values to be less than 2**32) and is initialized with zero. Here is the relevant part in the code: https://github.com/huggingface/transformers/blob/master/src/transformers/models/ibert/quant_modules.py#L508

Therefore, if you skip the quantization-aware training process and immediately use the model for inference, those layers may produce some unexpected outcomes. Could this be your case?<|||||>Yeah I'm not using quantization-aware training so that'll be the reason. Thanks for the answer!