BigBird attention type switching
### System Info

transformers 4.21.2 BigBirdModel

### Who can help?

@ydshieh

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Currently BigBird switches attention type from default 'block_sparse' to 'original_full' in the forward call if it encounters a batch that contains only sequences shorter than the minimum sequence length:
https://github.com/huggingface/transformers/blob/a2a3afbc8d26d6170909365ffba6bd75e186255f/src/transformers/models/big_bird/modeling_big_bird.py#L2064

However, it never switches back. This means that the exact same (long) sequence can be encoded differently depending on whether it was preceded by a batch containing only short sequences or not.



### Expected behavior

It should probably switch back to block_sparse if it can as well
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.