About the `head_mask` of the Bert model `forward` really speed up?
### System Info

- `transformers` version: 4.20.1
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.10
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.13.0.dev20220709 (False)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@LysandreJik Many thanks!

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, in `bert` model or other model, when we try to get the output, we usually use `output = model(**batch)`. And in the source code of bert model, we actually have a parameter called `head_mask`. So when we give them different head_mask of the model, like `outputs = model(head_mask=head_mask, **batch)`, the time will be different?

So I try to run the two codes, and the time of the code that have more-zero `head_mask` do not speed up

```
head_mask = torch.ones(12, 12)
print(head_mask)
for batch in dataloader:
    for k, v in batch.items():
        batch[k] = v
    import time
    start=time.perf_counter()
    outputs = model(head_mask=head_mask, **batch)
    end = time.perf_counter()
    print(end-start)
```

```
head_mask = torch.zeros(12, 12)
for i  in range (12):
    head_mask[i][0] = 1
print(head_mask)
for batch in dataloader:
    for k, v in batch.items():
        batch[k] = v
    import time
    start=time.perf_counter()
    outputs = model(head_mask=head_mask, **batch)
    end = time.perf_counter()
    print(end-start)
```

### Expected behavior

So I try to run the two codes, and the time of the code that have more-zero `head_mask` do not speed up

Many thanks!
Hey @CaffreyR, the head mask isn't there to speed things up.

You can read a bit more about it in the [bertology](https://huggingface.co/docs/transformers/v4.23.1/en/bertology) documentation. It's mostly to see which heads impact your prediction, it's not made to speed things up.<|||||>So you means that only if we prune the head according to the `head mask`, we can speed up our model?<|||||>I mean that looking at `head_mask` as a way to speed up the model doesn't work :)
If you'd like to speed up your model, you can look at changing the precision, quantizing it, distillating it; but removing heads isn't going to speed it up or very very marginally.<|||||>Great! Many thanks!