RuntimeError: [enforce fail at CPUAllocator.cpp:68] Out of memory during batched inference
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: p3.24xlarge
- Python version: 3.6
- PyTorch version (GPU?): 1.10.0+cu102
- Tensorflow version (GPU?): 2.6
- Using GPU in script?: No 
- Using distributed or parallel set-up in script?: No


 @LysandreJik 



## Information

I am using codebert model(RoBerta) for inference by wrapping it using huggingface. Below, I am trying to retrieve and store the  embeddings that is output from the model.
Code:

```
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained("microsoft/codebert-base")
# this is the pretrained model which we plan to fine-tune/transfer-learn further
model = RobertaModel.from_pretrained("codebert")
batch_size = 8
# Data is already loaded
tokens = tokenizer(data, return_tensors="pt", truncation=True, max_length=768, padding=True)["input_ids"]
conc = torch.empty(size=(len(tokens), 768))
model.eval()
for i in range(0, len(tokens), batch_size): 
    conc[i:i+batch_size] = model(tokens[i:i+batch_size])[1]
```
The shape of tokens is 5000 * 768 . 
System is a p3.24xlarge machine with about 700GB of RAM. 
Why is the model giving "DefaultCPUAllocator: can't allocate memory" even after batching the input and the system having large RAM?

Have you tried with a smaller amount of tokens to see where it fails?<|||||>Yes, I ran the "top" command in shell and observed that memory almost maxes out to 99% for just 2000. But, I was able to get over the memory issue by detaching the tensor and converting it to cpu() and then numpy() . 
```model(tokens[i:i+batch_size])[1].detach().cpu().numpy()```. My problem is fixed but just wondering what was eating up the memory.