AttributeError: 'QuantLinear' object has no attribute 'weight'
### System Info

Python = 3.9.10
Transformers = 4.30.0.dev0
PyTorch = 2.0.1
Model = Google/flan-ul2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Model quantized using `qwopqwop200/GPTQ-for-Llama` on the `t5` branch, using the following command:
```
python t5.py ../full-models/flan-ul2 wikitext2 --nsamples 256 --wbits 4 --act-order --groupsize 128 --save ../gptq-models/flan-ul2-gptq/flan-ul2-4bit-128g-gptq.pt
```
When performing benchmark using the following command (also applies to inference):
```
python t5.py ../full-models/flan-ul2 wikitext2 --load ../gptq-models/flan-ul2-gptq/flan-ul2-4bit-128g-gptq.pt --wbits 4 --groupsize 128 --benchmark --benchmark_mode mmlu
```
The following error occurs:
```
Traceback (most recent call last):
  File "/mnt/Storage/ai-dev/t5-gptq/t5.py", line 752, in <module>
    mmlu_benchmark(model, tokenizer, args)
  File "/mnt/Storage/ai-dev/t5-gptq/t5.py", line 542, in mmlu_benchmark
    cors, acc, probs = mmlu_eval(args, subject, model, tokenizer, dev_df, test_df, (idx,len(subjects)))
  File "~/anaconda3/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/Storage/ai-dev/t5-gptq/t5.py", line 473, in mmlu_eval
    logits = model(
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1683, in forward
    encoder_outputs = self.encoder(
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1090, in forward
    layer_outputs = layer_module(
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 753, in forward
    hidden_states = self.layer[-1](hidden_states)
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 319, in forward
    isinstance(self.wo.weight, torch.Tensor)
  File "~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'QuantLinear' object has no attribute 'weight'
```
According to my limited understanding, `QuantLinear` is a PyTorch class, and the error is occurring in `transformers`.

### Expected behavior

Successfully performing benchmark, inference, etc. of the 4-bit GPTQ flan-ul2 model.
Hi there. The script you are using is not one we have in Transformers, and we also do not have an object `QuantLinear`, so I'm really unsure why you are reporting this here?<|||||>Forgive me, I do not understand Python object mechanisms well, but I thought the following line was the error:
`File "~/anaconda3/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 319, in forward
    isinstance(self.wo.weight, torch.Tensor)`
I come from a C++ background, so my logic was that transformers is at fault because it's trying to access a variable on a class that does no exist. But of course, a Python object is nothing like a C++ class.

I am told that QuantLinear should have a `weight` attribute. So I am thinking maybe the model object is malformed? In that case I should report on the original project. Forgive my misunderstanding.<|||||>As I said before, we do not use that class (`QuantLinear`) anywhere in Transformers. So this comes from your script making modifications to the model that do not work. You should raise the issue in the repo where you found that script.