Cached sin and cos matrices for rotary at GPT-J model initialization for faster generation
# What does this PR do?

Makes generations faster by caching sincos matrices for rotary for the max sequence length at model initialisation so it won't be done every other forward.

around ~%15 percent speedup overall
Tested on A100-SXM4-80GB
Benchmarks:
1 token forward average runtime out of 100 iterations with cached sin cos:
0.023421470640283642s

1 token forward average runtime out of 100 iterations without cached sin cos:
0.02738137301433869s

10 generations with 1 token context and 40 tokens generated without sincos caching:
1.410405054409057s

10 generations with 1 token context and 40 tokens generated with sincos caching:
1.2199230638332665s

Test script:
```
torch.manual_seed(123)
input_ids = torch.randint(0, 100, (1, 1)).cuda().long()

iterations = 11
with torch.no_grad():
    for i in range(iterations):
        if i == 1:
            #start counting time after tier 1 due to pytorch warming up
            t = time.perf_counter()
        outputs = model.generate(input_ids, max_length=input_ids.shape[-1] + 50, min_length=input_ids.shape[-1] + 50, do_sample=True, use_cache=True)
        #outputs = model.forward(input_ids)
print((time.perf_counter() - t) / (iterations - 1))
```

Models:
- GPT-J


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19991). All of your documentation changes will be reflected on that endpoint.<|||||>Mmm, now it looks like the tests are not running for some reason. Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)? And/or push an empty commit?<|||||>Hi @kurumuz Thank you for the PR. I left a few comments, especially for the shape and the case where `self.rotary_dim = None`.
It is not very clear to me why the (changed) shapes of `sin` and `cos` doesn't cause any issue.

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.