when adding tokens for BlipModel,A bug has appeared

When I execute the following code to add a vocab, an error is reported - 

`
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
import sys
import logging
import pandas as pd
from dataclasses import dataclass, field
from typing import Optional

import torch
from datasets import Dataset
from datasets import load_dataset
from PIL import Image
from torchvision.io import ImageReadMode, read_image
from torchvision.transforms import CenterCrop, ConvertImageDtype, Normalize, Resize
from torchvision.transforms.functional import InterpolationMode
from transformers import BlipModel, BlipForImageTextRetrieval, BlipForConditionalGeneration, BlipProcessor, AutoConfig, AutoTokenizer
import transformers
from transformers import (
    HfArgumentParser,
    Trainer,
    TrainingArguments,
    set_seed,
)

model_path= r'D:\all_models_archives\models--Salesforce--blip-itm-large-coco'

tokenizer= AutoTokenizer.from_pretrained(model_path)
processor = BlipProcessor.from_pretrained(model_path)
model_config= AutoConfig.from_pretrained(model_path)

model= BlipModel.from_pretrained(pretrained_model_name_or_path= model_path, config= model_config)


`
![image](https://user-images.githubusercontent.com/36957508/213849376-3191b6b0-7715-4f30-92fc-5888258497da.png)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
import sys
import logging
import pandas as pd
from dataclasses import dataclass, field
from typing import Optional

import torch
from datasets import Dataset
from datasets import load_dataset
from PIL import Image
from torchvision.io import ImageReadMode, read_image
from torchvision.transforms import CenterCrop, ConvertImageDtype, Normalize, Resize
from torchvision.transforms.functional import InterpolationMode
from transformers import BlipModel, BlipForImageTextRetrieval, BlipForConditionalGeneration, BlipProcessor, AutoConfig, AutoTokenizer
import transformers
from transformers import (
    HfArgumentParser,
    Trainer,
    TrainingArguments,
    set_seed,
)

model_path= r'D:\all_models_archives\models--Salesforce--blip-itm-large-coco'

tokenizer= AutoTokenizer.from_pretrained(model_path)
processor = BlipProcessor.from_pretrained(model_path)
model_config= AutoConfig.from_pretrained(model_path)

model= BlipModel.from_pretrained(pretrained_model_name_or_path= model_path, config= model_config)



`
![image](https://user-images.githubusercontent.com/36957508/213849376-3191b6b0-7715-4f30-92fc-5888258497da.png)


### Expected behavior

Adding vocabulary can be done normally
Hi @ScottishFold007 
Thanks for the issue, I don't really see how the script you provided can add a new vocab to the model, can you either provide the full script or the full traceback of the error? Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.