Roberta model seemingly unable to take embeddings as input
### System Info

Python 3.9 on linux, transformers 4.24.0



### Who can help?

@lysandrejik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Trying to implement cross encoder with RoBERTa. I embed token ids separate from the model forward call:
```python
model = AutoModel.from_pretrained("roberta-base")
text_features : TensorType["batch", "sequence_length", "d_model"] = model.embeddings(text.input_ids)
embs = .... # same shape as text features
```
AFAIK this incorporates word embeddings, positional embeddings and special token embeddings, since those are all in the models embeddings. Sequence length here is max_posititon_embeddings from the models config. I then add on ViT embeddings to start of sequence and truncate accordingly (shape of tensor stays the same)
```python
out = model(
            inputs_embeds=embs,
            attention_mask=attn_mask,
            output_hidden_states=True,
            return_dict=True
        )
```
This gives error: 
```python
IndexError: index out of range in self
```
The traceback (image linked [here](https://media.discordapp.net/attachments/738882678711123980/1039276109423984711/unknown.png?width=719&height=146)) suggests it is trying to perform word and position embeddings again, and an error is occurring when position embeddings are called. If I'm understanding correctly, since I already did embeddings, and am providing them rather than tokens, it should not be calling on position embeddings again, yes? Running the same code with "bert-base-uncased" functions as expected with no errors.

### Expected behavior

Expect a successful forward call through model using the embeddings provided
Even if you provide input embeddings, the RoBERTa model will still add the token type embeddings and position embeddings.

As for the exact reason you get an error, we would need a full reproducer to be able to investigate.<|||||>> Even if you provide input embeddings, the RoBERTa model will still add the token type embeddings and position embeddings.

Is there any way to circumvent this? In my use-case position embeddings would probably be harmful to training.




<|||||>You'll need to modify the model code for that.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.