Regression: ONNX export fails on Pytorch ToT (NVIDIA 22.03 pytorch container)
## Environment info

- `transformers` version: 4.17.0
- Platform: Linux-5.11.0-37-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyTorch version (GPU?): 1.12.0a0+2c916ef (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed  
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO


### Who can help
@LysandreJik

Models:
- ALBERT, BERT

## Information

Model I am using (Bert, XLNet ...):
ALBERT, BERT

The problem arises when using:

* [ ] my own modified scripts: (give details below)



The tasks I am working on is:

* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.   Install ToT NeMo (http://github.com/NVIDIA/NeMo) 
2.  Build NeMo container with Dockerfile
3.  In the container, uncomment 2 HG tests : 
```
--- a/tests/collections/nlp/test_huggingface.py
+++ b/tests/collections/nlp/test_huggingface.py
@@ -49,8 +49,7 @@ class TestHuggingFace:
         self.omega_conf.language_model.pretrained_model_name = 'bert-base-uncased'
         model = nemo_nlp.modules.get_lm_model(cfg=self.omega_conf)
         assert isinstance(model, nemo_nlp.modules.BertEncoder)
-        # TODO: Fix
-        # do_export(model, "bert-base-uncased")
+        do_export(model, "bert-base-uncased")

     @pytest.mark.with_downloads()
     @pytest.mark.unit
@@ -74,8 +73,7 @@ class TestHuggingFace:
         self.omega_conf.language_model.pretrained_model_name = 'albert-base-v1'
         model = nemo_nlp.modules.get_lm_model(cfg=self.omega_conf)
         assert isinstance(model, nemo_nlp.modules.AlbertEncoder)
-        # TODO: fix
-        # do_export(model, "albert-base-v1")
+        do_export(model, "albert-base-v1")
```
Run pytest ./tests/collections/nlp/test_huggingface.py --with_downloads
```
Observe the result: 
>               params_dict = torch._C._jit_pass_onnx_deduplicate_initializers(graph, params_dict,
                                                                               training == TrainingMode.TRAINING)
E                                                                              RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument other in method wrapper__equal)
```
.....
================================== short test summary info ===================================
FAILED tests/collections/nlp/test_huggingface.py::TestHuggingFace::test_get_pretrained_bert_model
FAILED tests/collections/nlp/test_huggingface.py::TestHuggingFace::test_get_pretrained_albert_model

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
No error. Same code works fine in previous (22.02) NVIDIA pytorch container.

<!-- A clear and concise description of what you would expect to happen. -->

I was able to track one of similar new ONNX errors in NeMo code down and fixed it with the diff below.
It appears exporter gets confused about device placement of inline constants (?) as well as class attributes that are scalars, not proper Variables. 
In the sample from NeMo code below both literal 0 and self.max_token_duration had to be moved to buffers to be exported properly. I am quite sure something like that has to be fixed in HF code to work around new Pytorch export issues:

```
-        self.max_token_duration = max_token_duration
+        self.register_buffer('max_token_duration', torch.tensor(max_token_duration))
+        self.register_buffer('min_token_duration', torch.tensor(0.0))
...
-        durs_predicted = torch.clamp(torch.exp(log_durs_predicted) - 1, 0, self.max_token_duration)
+        durs_predicted = torch.clamp(
+            torch.exp(log_durs_predicted) - 1.0, self.min_token_duration, self.max_token_duration
+        )

```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Please never mind - it was pytorch exporter issue and is now fixed.