why training bigbird-512 model is much slower than bert-512?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.15.0
- Platform: 
- Python version: 3.6.8
- PyTorch version (GPU?): 1.10.1+cu102
- Tensorflow version (GPU?):
- Using GPU in script?: one GPU of V100-16GB
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten

Model I am using (Bert, XLNet ...): BigBird

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [*] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I have compared the training time of bigbird model and bert model, both in the sequence length of 512, with the official recommend pipeline (config -> MLM model -> dataset -> data_collator -> training_args -> trainer ).
But I find the training time of bigbird model is much bigger than bert model, even in the same seq_length, here are the detailed parameters:
num_train_epochs=1
learning_rate = 5e-5
warmup_steps = 0
save_strategy='no'
gradient_checkpointing=False,
seed=222
gradient_accumulation_steps=1  
max_length = 512
the total number of samples is 200_000
1. for bert-512, I can set the batchsize up to 13 (bigger would raise OOM error), the total training time is 2:04:18
2. for bigbird-512, I revise the block_size to 32 to meet the requirements of (5 + 2*num_random_blocks)*block_size < 512, and only can set the batchsize up to 11 (bigger would raise OOM error, this value is smaller than Bert's, I don't know the reaseon, the total parameters of bigger would be bigger than Bert's ?), the total training time is about 4:45:23, that is much slower than Bert-512, 

Anything wrong? looking forward to your response, thanks advance

If your maximum sequence length is just 512, it does indeed not make much sense to use BigBird and BERT, AlBERT, or DeBERTa make much more sense. BigBird should be used for models that require sequence lengths > 2048<|||||>The main promise of BigBird is not that it is faster than BERT for seq_len < 512, but rather that it can handle sequence lengths of up to 16K tokens<|||||>Got it. Thanks very much.