Quantize t5 v1_1 generates nonsense
## Environment info
- `transformers` version: 4.2.0
- Platform: Linux
- Python version: 2.10.0

Models:
- T5 v1_1 @patrickvonplaten, @patil-suraj


Hi, when quantizing DenseReluDense in T5 v1_1, it generates nonsense I guess it's related to #10830

Just wanted to check if there is an easy solution.

If anyone is interested, I wrote this code to quantize all layers except for the DenseReluDense, it decreases the size to about 30 GB instead of 44 GB for the xxl version of t5-v1_1

```
import functools

def rsetattr(obj, attr, val):
    pre, _, post = attr.rpartition('.')
    return setattr(rgetattr(obj, pre) if pre else obj, post, val)

def rgetattr(obj, attr, *args):
    def _getattr(obj, attr):
        return getattr(obj, attr, *args)
    return functools.reduce(_getattr, [obj] + attr.split('.'))

toFix=[]
for name, param in model.named_parameters():
    if not "DenseReluDense" in name and not "layer_norm" in name:
        
        name=name.replace(".weight","")
        if len(name.split(".")) >2:
            name=name[:-2]
        toFix.append(name)
        
toFix=set(toFix)
for item in toFix:
    try:
        cat=torch.quantization.quantize_dynamic(
            rgetattr(model,item),  
            {torch.nn.Linear},
            dtype=torch.qint8) 
        rsetattr(model,item,cat)
    except:
        pass
```
Hey @ViktorThink,

Could you try to use a newer version of `transformers` and upgrade to Python 3? <|||||>I'm tried using transformers=4.12.3, but I get this model when using the model even without quantization:

  File "train-vt5-data-5-pretrained.py", line 143, in trainOnText
    loss = model(b_input_ids, attention_mask=b_input_mask, labels=lm_labels).loss
  File "/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py", line 1578, in forward
    return_dict=return_dict,
  File "/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py", line 977, in forward
    if self.gradient_checkpointing and self.training:
  File "/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py", line 1178, in __getattr__
    type(self).__name__, name))
AttributeError: 'T5Stack' object has no attribute 'gradient_checkpointing'

In my tests 'google/t5-xxl-lm-adapt' doesn't work with transformers 4.12.3, maybe none of the v1_1 does? 

I tried 4.9.2, and there the quantized version prints nonsense, and I had python 3.6 or something like that.<|||||>Quantization is still very experimental in general and is not 100% supported by `transformers`. We will have better support for it in the `optimum` library: https://github.com/huggingface/optimum . Until then could you please try to use `transformers` on `master` so that we can debug together? :-)<|||||>The optimum library seems really interesting! I saw it already supports quantization with Intel Neural Compressor, so I could try it out.

I tested google/t5-v1_1-small in a colab, using transformers on master, and the model runs, but still it prints nonsense if the DenseReluDense layers are quantized.
https://colab.research.google.com/drive/1lJofhzTJd4Suym4OMgvKPlaKmk1VkgVR?usp=sharing
<|||||>Gently pinging @michaelbenayoun @echarlaix here<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.