torch.jit.trace quantized bigbird leads to 0INTERNAL ASSERT FAILED runtime error 
Attempt to torch jit trace and save a quantized bigbird model leads to 0INTERNAL ASSERT FAILED runtime error. 
I also ran the same code for BERT and RoBERTa (see `example.ipynb`) but did not encounter the same issue and was able to trace the quantized models for both respectively.

## To Reproduce

Steps to reproduce the behavior:

1. Git clone this [repo](https://github.com/matthiaslmz/quantized_bigbird_issue) 
2. Run `example.ipynb`

### Stacktrace:
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-1dfdd2340788> in <module>
      4 )
      5 
----> 6 traced_model = torch.jit.trace(model, (input_ids, attention_mask))
      7 torch.jit.save(traced_model, "traced_bigbird.pt")

/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    742             strict,
    743             _force_outplace,
--> 744             _module_class,
    745         )
    746 

/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    957                 strict,
    958                 _force_outplace,
--> 959                 argument_names,
    960             )
    961             check_trace_method = module._c._get_method(method_name)

RuntimeError: 0INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/ir/alias_analysis.cpp":532, please report a bug to PyTorch. We don't have an op for aten::constant_pad_nd but it isn't a special case.  Argument types: Tensor, int[], bool, 

```
## Expected behavior
Quantized bigbird should be able to be saved. 

## Environment

 - PyTorch Version: 1.9.0+cu111
 - Transformers Version: 4.9.1
 - OS: Debian GNU/Linux 10 (buster)
 - Python version: 3.7.9
 - CUDA/cuDNN version: 11.0.194
 - GPU models and configuration: NVIDIA Tesla V100 16GB


### Who can help
@patrickvonplaten 
Uff trying to `torch.jit(...)` our most complex model BigBird won't be easy I think :-/ Sadly I won't find time to dig deeper into this as it will require a lot of work :-/ Could you maybe try to go wiht `Longformer` for now?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@patrickvonplaten Sadly this error is also raised for me when actually using Longformer.<|||||>In Longformer, looks like this bug comes from this single line: https://github.com/huggingface/transformers/blob/ee6674d45030161d8d60533b7d469a727d492113/src/transformers/models/longformer/modeling_longformer.py#L1573
```
            attention_mask = nn.functional.pad(
                attention_mask, (0, padding_len), value=False  # <-- should be 0
            )  # no attention on the padding tokens
```
`nn.functional.pad` expects a number, not a boolean, value

In BigBird, the same bug is here:
https://github.com/huggingface/transformers/blob/ee6674d45030161d8d60533b7d469a727d492113/src/transformers/models/big_bird/modeling_big_bird.py#L2252

<|||||>Great catch @dadamson - if you want feel free to open a PR for it :-)