Added OPT to models exportable with ONNX
# What does this PR do?

```python
# !python setup.py install 
```


```python
# !pip install -e ".[dev]"
```


```python
# pip install onnxruntime
```


```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", torch_dtype=torch.float32)

# the fast tokenizer currently does not work correctly
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m", use_fast=False)

prompt = "Hello, I'm am conscious and"


input_ids = tokenizer(prompt, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids)

tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

```




    ["Hello, I'm am conscious and I'm a bit of a noob. I'm looking"]




```python

```


```python
!python -m transformers.onnx --model=facebook/opt-350m onnx/opt-350m/
```

    Using framework PyTorch: 1.11.0+cu102
    Overriding 1 configuration item(s)
    	- use_cache -> False
    /root/transformers/src/transformers/models/opt/modeling_opt.py:513: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if input_shape[-1] > 1:
    /root/transformers/src/transformers/models/opt/modeling_opt.py:64: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
      mask = torch.full((tgt_len, tgt_len), torch.tensor(float("-inf")))
    /root/transformers/src/transformers/models/opt/modeling_opt.py:203: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
    /root/transformers/src/transformers/models/opt/modeling_opt.py:210: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if attention_mask.size() != (bsz, 1, tgt_len, src_len):
    /root/transformers/src/transformers/models/opt/modeling_opt.py:242: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
    Validating ONNX model...
    	-[✓] ONNX model output names match reference model ({'last_hidden_state'})
    	- Validating ONNX Model output "last_hidden_state":
    		-[✓] (2, 8, 512) matches (2, 8, 512)
    		-[x] values not close enough (atol: 1e-05)
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/root/transformers/src/transformers/onnx/__main__.py", line 107, in <module>
        main()
      File "/root/transformers/src/transformers/onnx/__main__.py", line 100, in main
        validate_model_outputs(onnx_config, preprocessor, model, args.output, onnx_outputs, args.atol)
      File "/root/transformers/src/transformers/onnx/convert.py", line 441, in validate_model_outputs
        "Outputs values doesn't match between reference model and ONNX exported model: "
    ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: 4.57763671875e-05



```python
from transformers.models.opt import OPTConfig, OPTOnnxConfig

config = OPTConfig()
onnx_config = OPTOnnxConfig(config)
output_keys = list(onnx_config.outputs.keys())

print(output_keys)
```

    ['last_hidden_state']



```python
from onnxruntime import InferenceSession
```


```python
import onnxruntime as ort

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")

ort_session = ort.InferenceSession("onnx/opt-350m/model.onnx")

inputs = tokenizer("Using OPT in ONNX!", return_tensors="np")
outputs = ort_session.run(["last_hidden_state"], dict(inputs))
print(outputs)
```

    [array([[[ -1.3539658 ,  -0.25787818,  -0.3093884 , ...,  -1.311745  ,
               0.26136506,  -1.4270447 ],
            [ -0.51148593,  -5.1948047 ,   3.1015701 , ...,   1.9010596 ,
              -2.0694203 ,   0.96382034],
            [ -1.4861462 ,  -4.3613157 ,  -2.8032331 , ...,  -0.65176994,
              -6.0503354 ,  -0.08128738],
            ...,
            [ -2.290329  ,  -9.395232  ,   3.9363523 , ...,  -0.5923378 ,
              -3.7993686 ,   0.13608676],
            [ -4.7750826 , -12.562761  ,   1.9932727 , ...,  -4.361832  ,
              -2.3446696 ,   1.2666583 ],
            [ -3.7153127 ,  -6.4608436 ,  -3.683312  , ...,  -2.824885  ,
              -0.75467056,  -1.9532645 ]]], dtype=float32)]




![image](https://user-images.githubusercontent.com/6279035/174467137-4c0f151e-16b3-436b-b412-2a39b7f361af.png)


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17771). All of your documentation changes will be reflected on that endpoint.<|||||>Great work!

It works for most use cases, but I did discover that `causal-lm-with-past` isn't working.

`python -m transformers.onnx --model=facebook/opt-350m --feature=causal-lm-with-past onnx/opt-350m/`

yields

```
2022-07-04 07:44:09.810891: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Using framework PyTorch: 1.11.0+cu113
Overriding 1 configuration item(s)
	- use_cache -> True
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 94, in main
    args.output,
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 335, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 142, in export_pytorch
    model_inputs = config.generate_dummy_inputs(preprocessor, framework=TensorType.PYTORCH)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/opt/configuration_opt.py", line 213, in generate_dummy_inputs
    self.num_attention_heads,
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/opt/configuration_opt.py", line 184, in num_attention_heads
    return self._config.n_head
  File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 253, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'OPTConfig' object has no attribute 'n_head'
```

That's because the naming is a bit different for the OPT config, but it can be fixed simply by replacing:

```
    @property
    def num_layers(self) -> int:
        return self._config.n_layer

    @property
    def num_attention_heads(self) -> int:
        return self._config.n_head
```

with

```
    @property
    def num_layers(self) -> int:
        return self._config.num_hidden_layers

    @property
    def num_attention_heads(self) -> int:
        return self._config.num_attention_heads
```

in the configuration_opt.py file.

But when I tried that, I ran into another problem:

```
python -m transformers.onnx --model=facebook/opt-350m --feature=causal-lm-with-past onnx/opt-350m/


2022-06-17 20:59:09.712912: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Error using standard tokenizer settings, testing specifying use_fast=False
Using framework PyTorch: 1.11.0+cu113
Overriding 1 configuration item(s)
	- use_cache -> True
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:513: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:64: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  mask = torch.full((tgt_len, tgt_len), torch.tensor(float("-inf")))
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:69: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if past_key_values_length > 0:
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:203: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:210: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py:242: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 112, in <module>
    main()
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 99, in main
    args.output,
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 335, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 198, in export_pytorch
    opset_version=opset,
  File "/usr/local/lib/python3.7/dist-packages/torch/onnx/__init__.py", line 309, in export
    export_modules_as_functions)
  File "/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py", line 122, in export
    custom_opsets=custom_opsets, export_modules_as_functions=export_modules_as_functions)
  File "/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py", line 724, in _export
    dynamic_axes=dynamic_axes)
  File "/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py", line 507, in _model_to_graph
    module=module)
  File "/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py", line 230, in _optimize_graph
    torch._C._jit_pass_onnx_set_dynamic_input_shape(graph, dynamic_axes, input_names)
RuntimeError: Dynamic shape axis should be no more than the shape dimension for past_sequence + sequence
```

And that's where I got stuck, and I don't really know how to solve it...


Another problem, that hopefully will be fixed soon, is that the fast tokenizer doesn't works, and local models tries to use the fast tokenizer when exporting to onnx:

```
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name="facebook/opt-125m"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
pt_model = AutoModelForCausalLM.from_pretrained(model_name)

tokenizer.save_pretrained("local-pt-checkpoint")
pt_model.save_pretrained("local-pt-checkpoint")
```
then
`
python -m transformers.onnx --model=local-pt-checkpoint --preprocessor=tokenizer  onnx/opt-350m/
`
Results in:

```
2022-07-04 08:02:21.613420: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 107, in <module>
    main()
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 64, in main
    preprocessor = AutoTokenizer.from_pretrained(args.model)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/tokenization_auto.py", line 580, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 1810, in from_pretrained
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 1948, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 152, in __init__
    "Currenty GPT2's fast tokenizer does NOT support adding a BOS token."
ValueError: Currenty GPT2's fast tokenizer does NOT support adding a BOS token.Instead you should use GPT2's slow tokenizer class `GPT2Tokenizer` as follows: 
`GPT2Tokenizer.from_pretrained('local-pt-checkpoint')`
or
`AutoTokenizer.from_pretrained('local-pt-checkpoint', use_fast=False)`
This issue will be fixed soon, see: https://github.com/huggingface/tokenizers/pull/1005. so that the fast tokenizer works correctly.
```

Just to try it out, I added this to the src/transformers/onnx__main__.py file (row 63):

```
    elif args.preprocessor == "tokenizer":
        try:
            preprocessor = AutoTokenizer.from_pretrained(args.model)
        except:
            logger.info(f"Error using standard tokenizer settings, testing specifying use_fast=False")
            preprocessor = AutoTokenizer.from_pretrained(args.model, use_fast=False)
```

Which fixed the issue, but it's not something we want there and, I just wanted to see if there was any other issues as well.

<|||||>@lewtun @rushic24 

Seems like the problem has to do with OPT handling past_key_values differently overall, I can't even make past key values work with the normal pytorch model.

I made a notebook to demonstrate, that passing past_key_values works for gpt2, but when doing the exact same thing with opt, it raises an error.

Here is the notebook:

https://colab.research.google.com/drive/14A-hm-aFzW64ZIxghJDVJaU6a4ZLGCBi?usp=sharing

Here is the error:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-17-d5f8e80cb463>](https://localhost:8080/#) in <module>()
      9 print()
     10 print(gpt_inputs["input_ids"])
---> 11 gpt_outputs = model(gpt_inputs.input_ids,return_dict=True,past_key_values=gpt_outputs.past_key_values)
     12 
     13 print(gpt_outputs.logits.shape)

4 frames
[/usr/local/lib/python3.7/dist-packages/transformers/models/opt/modeling_opt.py](https://localhost:8080/#) in _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length)
    527             expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])
    528             combined_attention_mask = (
--> 529                 expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
    530             )
    531 

RuntimeError: The size of tensor a (5) must match the size of tensor b (10) at non-singleton dimension 3

```
If anyone has any ideas on how to fix it, that would be appreciated. I think I will be able to look into this more deeply next week.<|||||>I had a deeper look into the past_key_values error, but am really struggling to understand code in the modeling_opt.py file.

The function that raises the error looks like this:

```
    # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
        # create causal mask
        # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
        combined_attention_mask = None
        if input_shape[-1] > 1:
            combined_attention_mask = _make_causal_mask(
                input_shape, inputs_embeds.dtype, past_key_values_length=past_key_values_length
            ).to(inputs_embeds.device)

        if attention_mask is not None:
            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
            expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])
            combined_attention_mask = (
                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
            )

        return combined_attention_mask
```

An what causes the error is this line:
```
            combined_attention_mask = (
                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
            )
```

Since expanded_attn_mask  and combined_attention_mask are different shapes, and can't be added together.

I tried printing both out before the error is rasied, and expanded_attn_mask   looks like this:

```
tensor([[[[0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0.]]]])
```

and combined_attention_mask looks like this:

```
tensor([[[[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
            0.0000e+00, -3.4028e+38, -3.4028e+38, -3.4028e+38, -3.4028e+38],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
            0.0000e+00,  0.0000e+00, -3.4028e+38, -3.4028e+38, -3.4028e+38],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
            0.0000e+00,  0.0000e+00,  0.0000e+00, -3.4028e+38, -3.4028e+38],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
            0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00, -3.4028e+38],
          [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
            0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00]]]])
```

I got stuck here, not really sure what's even going on...<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Gently pinging @younesbelkada who worked on the OPT port and may be able to shed some insight on why the generations don't work with past key-value pairs<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @ViktorThink @lewtun sorry for the delay, taking a look right now! 

Pinging also @ArthurZucker here<|||||>Hey all! Thanks for your awesome work! 

Regarding the `past_key_value`, given that the integration tests worked properly, I am not sure why it doesn't  work but will have a look to see what is wrong. 
<|||||>Hey @rushic24 @ViktorThink I believe several fixes have landed for the OPT models, so would you like to revisit this PR now? In particular, can you test if the `causal-lm-with-past` feature is now working as expected? 

If you want a fast way to test this, you can use `optimum` as follows (pointing to a local `model.onnx` file instead of a Hub checkpoint): https://huggingface.co/docs/optimum/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModelForCausalLM.forward.example<|||||>@lewtun I did a very simple test with past key values in this colab: https://colab.research.google.com/drive/1UO7uioZZs2Gu6nroQRgPilSpYQhNLiSZ?usp=sharing

I didn't try to export the model, but since past doesn't seem work in pytorch format, exporting it to Onnx shouldn't be possible.

It works for GPT models, but not for OPT. The link you sent leads to a 404 error.<|||||>> @lewtun I did a very simple test with past key values in this colab: https://colab.research.google.com/drive/1UO7uioZZs2Gu6nroQRgPilSpYQhNLiSZ?usp=sharing
> 
> I didn't try to export the model, but since past doesn't seem work in pytorch format, exporting it to Onnx shouldn't be possible.
> 
> It works for GPT models, but not for OPT. The link you sent leads to a 404 error.

Thanks for sharing a reproducible example @ViktorThink - this indeed looks like a bug! Pinging @younesbelkada to have a look since he implemented this model :)<|||||>I'll have a look when I can! <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey, there doesn't seem to be a bug with OPT's `past_key_value` scheme as the following works : 
```python 
import torch
from transformers import AutoTokenizer, OPTModel, set_seed
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
model = OPTModel.from_pretrained("facebook/opt-1.3b")

inputs = tokenizer("No I'm not missing the ", return_tensors="pt")

input_ids = inputs.input_ids
attention_mask = inputs.attention_mask

with torch.no_grad():
    model.config.use_cache = False
    set_seed(0)
    output = model(input_ids, attention_mask = attention_mask, use_cache =False)
    print(output.last_hidden_state[:,-1,:])

    model.config.use_cache = True
    output_1 = model(input_ids[:,:-1], use_cache = True, attention_mask = attention_mask[:,:-1])
    pkv = output_1.past_key_values
    output_2 = model(input_ids[:,-1:], past_key_values = pkv , attention_mask = attention_mask, use_cache = True)
    print(output_2.last_hidden_state[:,-1,:])
    torch.testing.assert_allclose(output.logits[:,-1,], output_2.logits[:,-1,:],  rtol = 1e-4, atol = 1e-4)
```
This is the expected format as inside the `generate` function, we are passing only the last inputs, while the full attention mask was already created.
The only issue I see here is consistency : the behaviours are different for `gpt2` and `opt`. 😅 

Moreover, the default attention mask created when the `past_key_values` are given also seem wrong : 

```python 
output_2 = model(input_ids[:,-1:], past_key_values = pkv , use_cache = True)
...
ValueError: Attention mask should be of size (1, 1, 0, 7), but is torch.Size([1, 1, 1, 1])
``` 
Should probably work, as the length of the previous sequence is given. 
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.