Add BLIP-2
# What does this PR do?

This PR adds BLIP-2 to the library.

To do:

- [x] make sure generation works exactly as the original implementation, (maybe @gante can have a look here - based on original code [here](https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/blip2_opt.py#L207-L211)). Edit: seems to be solved by properly setting the `eos_token_id`!
- [x] add more tests for BLIP-2 with `AutoModelForSeq2SeqLM` once designed gets approved
- [x] transfer checkpoints, update integration tests
- [ ] make it possible to instantiate Blip2Config with config objects, rather than dicts (also check default text config) - will be done in a separate PR

cc @younesbelkada 
_The documentation is not available anymore as the PR was closed or merged._<|||||>@sgugger all comments are addressed, feel free to approve :)<|||||>@NielsRogge Curious, what is the timeline for this to make it into a stable release version?<|||||>Usually there's a Transformers release once every 1 to 2 months, so at the very least in March.<|||||>Hi, thanks for the great work! I'm running into problems trying to use this in the multigpu setting and saw this was mentioned by @younesbelkada earlier -- is there an issue to follow for that? (Specifically, in line 2765 of transformers->generation->utils.py, the devices don't match -- `Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0!` because beam_scores is on cuda:0 while next_token_scores and next_token_scores_processed are on cuda:3 after using "auto" for the device_map when loading.)

I'm also getting a weirder error where it causes a CUDA illegal memory access error for any model used downstream of it on GPU 0, even when it's given no GPU memory on GPU 0 in max_memory. (This doesn't occur for the original BLIP2, which I'm trying to migrate from.)<|||||>Same problem here @sachit-menon 
"Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!"
https://github.com/TimDettmers/bitsandbytes/issues/153<|||||>Hi @sachit-menon @xszheng2020 
This is a known issue on my end, I can confirm this should be at least fixed for `blip2-opt` at https://github.com/huggingface/transformers/pull/21707
Can you try to checkout from this branch and let us know on the PR if the fix works? thanks!<|||||>Hi, @younesbelkada 
thanks! will test it on blip2-opt to see whether it works!
and hope the blip2-flant5 could be fixed soon