Movement pruning for DistilGPT2 - pre_trained model, issue while using dynamic_quantization
## Environment info

- `transformers` version: 4.3.3
- Platform: Ubuntu 20.04
- Python version: 3.8.8
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

[@VictorSanh](https://github.com/VictorSanh)

## Information

I am following through the Saving PruneBERT [notebook](https://github.com/huggingface/transformers/blob/b11386e158e86e62d4041eabd86d044cd1695737/examples/movement-pruning/Saving_PruneBERT.ipynb) from the *examples/movement-pruning/* directory, to have a pruned and quantized model for DistilGPT2. 

In cell 4:

```
# Elementary representation: we decompose the quantized tensors into (scale, zero_point, int_repr).
# See https://pytorch.org/docs/stable/quantization.html

# We further leverage the fact that int_repr is sparse matrix to optimize the storage: we decompose int_repr into
# its CSR representation (data, indptr, indices).

elementary_qtz_st = {}
for name, param in qtz_st.items():
    if "dtype" not in name and param.is_quantized:
        print("Decompose quantization for", name)
        # We need to extract the scale, the zero_point and the int_repr for the quantized tensor and modules
        scale = param.q_scale()                                # torch.tensor(1,) - float32
        zero_point = param.q_zero_point()                      # torch.tensor(1,) - int32
        elementary_qtz_st[f"{name}.scale"] = scale
        elementary_qtz_st[f"{name}.zero_point"] = zero_point

        # We assume the int_repr is sparse and compute its CSR representation
        # Only the FCs in the encoder are actually sparse
        int_repr = param.int_repr()                         # torch.tensor(nb_rows, nb_columns) - int8
        int_repr_cs = sparse.csr_matrix(int_repr)           # scipy.sparse.csr.csr_matrix

        elementary_qtz_st[f"{name}.int_repr.data"] = int_repr_cs.data                  # np.array int8
        elementary_qtz_st[f"{name}.int_repr.indptr"] = int_repr_cs.indptr              # np.array int32
        assert max(int_repr_cs.indices) < 65535 # If not, we shall fall back to int32
        elementary_qtz_st[f"{name}.int_repr.indices"] = np.uint16(int_repr_cs.indices) # np.array uint16
        elementary_qtz_st[f"{name}.int_repr.shape"] = int_repr_cs.shape                # tuple(int, int)
    else:
        elementary_qtz_st[name] = param
```

my model throws the below error:

```AttributeError: 'NoneType' object has no attribute 'is_quantized'```

which on digging into quantizing step in cell 2, shows a significant difference between the used BERT and DistilGPT2 (the one am using) quantized version:

1. The BERT quantized first few layers look like:

   ```
   bert.embeddings.position_ids
   bert.embeddings.word_embeddings.weight
   bert.embeddings.position_embeddings.weight
   bert.embeddings.token_type_embeddings.weight
   bert.embeddings.LayerNorm.weight
   bert.embeddings.LayerNorm.bias
   bert.encoder.layer.0.attention.self.query.scale
   bert.encoder.layer.0.attention.self.query.zero_point
   bert.encoder.layer.0.attention.self.query._packed_params.weight
   bert.encoder.layer.0.attention.self.query._packed_params.bias
   bert.encoder.layer.0.attention.self.key.scale
   bert.encoder.layer.0.attention.self.key.zero_point
   bert.encoder.layer.0.attention.self.key._packed_params.weight
   bert.encoder.layer.0.attention.self.key._packed_params.bias
   bert.encoder.layer.0.attention.self.value.scale
   bert.encoder.layer.0.attention.self.value.zero_point
   bert.encoder.layer.0.attention.self.value._packed_params.weight
   bert.encoder.layer.0.attention.self.value._packed_params.bias
   ```

2. The quantized DistilGPT2 first few layers look like:

   ```
   transformer.wte.weight
   transformer.wpe.weight
   transformer.h.0.ln_1.weight
   transformer.h.0.ln_1.bias
   transformer.h.0.attn.bias
   transformer.h.0.attn.masked_bias
   transformer.h.0.attn.c_attn.weight
   transformer.h.0.attn.c_attn.bias
   transformer.h.0.attn.c_proj.weight
   transformer.h.0.attn.c_proj.bias
   transformer.h.0.ln_2.weight
   transformer.h.0.ln_2.bias
   ```

3. As you would notice, there is a clear difference in the way layers are formed after quantization

   a) BERT has ```.scale``` and ```.zero_point``` added to every layer after embeddings, whereas the DistilGPT2 layers do not get these 2 extras.

   b) any ```.weight``` and ```.bias``` are converted to ```._packed_params.weight``` and ```._packed_params.bias``` respectively.

4. I beleive this is why when processing the cell 4:

   a) It does not even go to all layers that are missing ```._packed_params``` and just tries to process the last layer which is

   ```
   lm_head.scale
   lm_head.zero_point
   lm_head._packed_params.weight
   lm_head._packed_params.bias
   ```

   b) Where it fails with the error mentioned just before point 1.

## To reproduce

Steps to reproduce the behavior:

1. Clone `transformers` and follow the steps to install the `movement-pruning` example
2. Upgrade torch to v1.4
3. Try to run the `Saving_PruneBERT.ipynb` notebook with 1 change, in the cell 2 instantiate the model class with below line
   ```model = AutoModelForCausalLM.from_pretrained('distilgpt2')```
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.