[BUG REPORT]  inconsistent inference results between batch of samples and a single sample in BLIP / BLIP2 
### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-1041-azure-x86_64-with-glibc2.31
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES (a single A100, 80GB)
- Using distributed or parallel set-up in script?: NO

### Who can help?

@ArthurZucker and @younesbelkada@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Inconsistent inference results between batch of samples and a single sample in BLIP / BLIP2.


Here is the script. We can change `DEVICE`, `CAPTION_PRETRAIN_MODEL`, and `pixel_values_shape` to test different models on different accelrators.
```python
import transformers
from transformers import AutoModel, AutoProcessor, AutoConfig
import torch
import numpy as np
from typing import Mapping, Sequence

SEED = 42
transformers.enable_full_determinism(SEED)

CAPTION_PRETRAIN_MODELS_NAMES = [
    "Salesforce/blip-image-captioning-base",
    "Salesforce/blip-image-captioning-large",
    "Salesforce/blip2-opt-2.7b",
]
CAPTION_PRETRAIN_MODEL = CAPTION_PRETRAIN_MODELS_NAMES[1]
# NOTE: If you use BLIP2 model, you need to change the `pixel_values_shape` below accordingly.
CACHE_DIR = ".model.cache/"
DEVICE = "cpu"
# DEVICE = "cuda"

# MODEL
config = AutoConfig.from_pretrained(CAPTION_PRETRAIN_MODEL, cache_dir=CACHE_DIR)
caption_architectures = config.architectures
if len(caption_architectures) != 1:
    print(f"captioner_architectures: {caption_architectures} has to be of length 1")
caption_architecture = caption_architectures[0]

module = getattr(transformers, caption_architecture)
model = module.from_pretrained(CAPTION_PRETRAIN_MODEL, cache_dir=CACHE_DIR)
processor = AutoProcessor.from_pretrained(CAPTION_PRETRAIN_MODEL, cache_dir=CACHE_DIR)
model.to(DEVICE)

# Data
pixel_values_shape = [1, 3, 384, 384]  # shape for BLIP
# pixel_values_shape = [1, 3, 224, 224]  # shape for BLIP2
input_ids_shape = [1, 17]
attention_mask_shape = [1, 17]
labels_shape = [1, 17]


single_sample_inputs = {
    "pixel_values": torch.ones(pixel_values_shape),
    "input_ids": torch.ones(input_ids_shape, dtype=torch.long),
    "attention_mask": torch.ones(attention_mask_shape, dtype=torch.long),
    "labels": torch.ones(labels_shape, dtype=torch.long),
}

batch_size = 2
batch_sample_inputs = {
    "pixel_values": single_sample_inputs["pixel_values"].repeat(batch_size, 1, 1, 1),
    "input_ids": single_sample_inputs["input_ids"].repeat(batch_size, 1),
    "attention_mask": single_sample_inputs["attention_mask"].repeat(batch_size, 1),
    "labels": single_sample_inputs["labels"].repeat(batch_size, 1),
}
for k in single_sample_inputs:
    single_sample_inputs[k] = single_sample_inputs[k].to(DEVICE)
for k in batch_sample_inputs:
    batch_sample_inputs[k] = batch_sample_inputs[k].to(DEVICE)

with torch.no_grad():
    single_sample_outputs = model(**single_sample_inputs)
    batch_sample_outputs = model(**batch_sample_inputs)

print(f"Model: {CAPTION_PRETRAIN_MODEL} with {caption_architecture}, using {DEVICE} device")


def recursive_compare_print(outputs_1, outputs_2, tensor_slice=None, key=None, depth=0):
    if type(outputs_1) != type(outputs_2):
        raise ValueError(f"outputs_1: {type(outputs_1)} vs outputs_2: {type(outputs_2)}")
    elif isinstance(outputs_1, torch.Tensor):
        if tensor_slice is None:
            tensor_slice = slice(None)
        if len(outputs_1.shape) == 0:
            print(
                "\t" * depth
                + f"diff of {key} (shape={outputs_1.shape}): {torch.max(torch.abs(outputs_1 - outputs_2))}"
            )
        else:
            print(
                "\t" * depth
                + f"diff of {key} (shape={outputs_1.shape}): {torch.max(torch.abs(outputs_1[tensor_slice] - outputs_2[tensor_slice]))}"
            )
    elif isinstance(outputs_1, Mapping):
        print("\t" * depth + f"Mapping {key} (type {type(outputs_1)}):")
        for k in outputs_1:
            recursive_compare_print(outputs_1[k], outputs_2[k], tensor_slice=tensor_slice, key=k, depth=depth + 1)
    elif isinstance(outputs_1, Sequence):
        print("\t" * depth + f"Sequence {key} (type {type(outputs_1)}):")
        for output_1, output_2 in zip(outputs_1, outputs_2):
            recursive_compare_print(output_1, output_2, tensor_slice=tensor_slice, depth=depth + 1)
    else:
        print("\t" * depth + f"Unexpected type with {k}: {type(outputs_1)}")


recursive_compare_print(single_sample_outputs, batch_sample_outputs, slice(0, 1))
```

- When `DEVICE=CPU`, the results are ok except for logits having a small difference of 1e-5
```
Model: Salesforce/blip-image-captioning-base with BlipForConditionalGeneration, using cpu device
Mapping: (type <class 'transformers.models.blip.modeling_blip.BlipForConditionalGenerationModelOutput'>)
        diff of loss (shape=torch.Size([])): 0.0
        diff of decoder_logits (shape=torch.Size([1, 17, 30524])): 1.049041748046875e-05
        diff of image_embeds (shape=torch.Size([1, 577, 768])): 0.0
        diff of last_hidden_state (shape=torch.Size([1, 577, 768])): 0.0
```
- When `DEVICE="cuda"`, the results are having a large difference.

```
Model: Salesforce/blip-image-captioning-base with BlipForConditionalGeneration, using cuda device
Mapping: (type <class 'transformers.models.blip.modeling_blip.BlipForConditionalGenerationModelOutput'>)
        diff of loss (shape=torch.Size([])): 7.62939453125e-06
        diff of decoder_logits (shape=torch.Size([1, 17, 30524])): 0.0015845298767089844
        diff of image_embeds (shape=torch.Size([1, 577, 768])): 0.19360780715942383
        diff of last_hidden_state (shape=torch.Size([1, 577, 768])): 0.19360780715942383
```

### Expected behavior

The result of GPU inference should be at least the same as those of CPU.
cc @younesbelkada , but @xk-huang  Could you first try all the suggestions in [Reproducibility](https://pytorch.org/docs/stable/notes/randomness.html) 🙏 Thanks a lot.

Also

```
# `False` is already the default
torch.backends.cuda.matmul.allow_tf32 = False

# The flag below controls whether to allow TF32 on cuDNN. This flag defaults to True.
torch.backends.cudnn.allow_tf32 = False
```<|||||>Thanks for your kind advice! @ydshieh 

I have already adopted the reproducibility suggestions in Torch documents by setting `transformers.enable_full_determinism(SEED)`. After I turn off `torch.backends.cudnn.allow_tf32`, the differences are largely reduced. Here is the comparison:

```
Model: Salesforce/blip-image-captioning-base with BlipForConditionalGeneration, using cuda device
Mapping None (type <class 'transformers.models.blip.modeling_blip.BlipForConditionalGenerationModelOutput'>):
        diff of loss (shape=torch.Size([])): 1.9073486328125e-06
        diff of decoder_logits (shape=torch.Size([1, 17, 30524])): 8.58306884765625e-06
        diff of image_embeds (shape=torch.Size([1, 577, 768])): 0.0
        diff of last_hidden_state (shape=torch.Size([1, 577, 768])): 0.0
```

I am wondering whether this level of error is acceptable. <|||||>Glad it works 🚀 !

I would say with strong confidence it's very acceptable :-).
(Welcome to the whole numeric world 😅 )

<|||||>Thank you so much for your reply! I'm ready to explore the numeric rabbit hole!