blip2 always decode eos_token first
### System Info


- `transformers` version: 4.31.0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I just run the code provided by the example given in the doc:

https://huggingface.co/docs/transformers/main/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example





### Expected behavior

I got exactly the same caption of the image. 
generated_text
'two cats laying on a couch'

But its weird that generated ids look like:
generated_ids:
tensor([[    2,  7109, 10017, 11963,    15,    10, 16433, 50118]])

where he leading `2` means a `</s>` is generated. Is this expected or something wrong with the decoding?

@amyeroberts @ArthurZucker  do you guys have any ideas on it?
cc @younesbelkada <|||||>Hi @TobiasLee 
Thanks for the issue. I think this is expected, the way Blip2 has been trained always prepends the BOS (Beginning of Sentence) token to the input text due to the underlying tokenizer they use. If conditional text is added to the input, the tokenizer should normally take care of adding that token to the beginning of the sentence. 

See this line for reference: https://github.com/huggingface/transformers/blob/e75cb0cb3c5fef887abea6f099252e59a659af9d/src/transformers/models/blip_2/modeling_blip_2.py#L1837 / that manually adds the BOS token if no text is passed

And this line from the original code: https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_opt.py#L219 that uses the opt token which adds the bos token to any input by default according to the tokenizer_config file: https://huggingface.co/facebook/opt-350m/raw/main/tokenizer_config.json 