Does `prune_heads` really speed up during inference?
### System Info

- `transformers` version: 4.22.1
- Platform: Linux-5.13.0-48-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi I have a code here, based on your examples. I try to use `bert` to test inference speed
The problem is no matter whether I pruned the model, the time seems to remain the same 

The code is belows

```

from transformers import AutoTokenizer, AutoModel
import torch
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
device='cuda'
prune_heads = {}
prune_heads[0] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[1] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[2] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[3] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[4] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[5] = [0,1,2,3,4,5,6,7,8,9]

prune_heads[6] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[7] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[8] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[9] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[10] = [0,1,2,3,4,5,6,7,8,9]
prune_heads[11] = [0,1,2,3,4,5,6,7,8,9]

'''Whether to prune'''
# model.prune_heads(prune_heads). 

inputs = tokenizer("Hello world!", return_tensors="pt").to(device)
model=model.to(device)
model.eval()

import time
cnt=0
for i in range(3):
    outputs = model(**inputs)


for i in range(10):
    torch.cuda.synchronize()
    start = time.perf_counter()
    outputs = model(**inputs)
    torch.cuda.synchronize()
    end = time.perf_counter()
    print(i,":",end-start)
    cnt += (end-start)
    # print(outputs)
print(cnt)

```

### Expected behavior

The problem is no matter whether I pruned the model, the time seems to remain the same 
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only. I am not aware of any place in the doc where we advertise head-pruning as a mean to speed up inference. I think you will need to look at converting your model to ONNX or quantize it for that.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.