OwlVit gives different results compared to original colab version
### System Info

Using huggingface space and google colab

### Who can help?

@adirik

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

cat picture from http://images.cocodataset.org/val2017/000000039769.jpg
remote control image from https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSRUGcH7a3DO5Iz1sknxU5oauEq9T_q4hyU3nuTFHiO0NMSg37x

### Expected behavior

Being excited with the results of OwlVit, I tried to input some random image to see the results.
Having no experience on jax, my first option is to search out on huggingface space.

Given a query of remote control, and a cat picture, I wanted to get picture of remote controls.
https://huggingface.co/spaces/adirik/image-guided-owlvit
![Screenshot 2023-01-20 at 14 13 13](https://user-images.githubusercontent.com/44696192/213622384-23764ffc-8056-47e3-95e9-33dcf5ce53fa.png)
The results is not really what I expected (no box on remotes).

Then I checked for results on colab version, if they behave the same way.
 https://colab.research.google.com/github/google-research/scenic/blob/main/scenic/projects/owl_vit/notebooks/OWL_ViT_inference_playground.ipynb#scrollTo=AQGAM16fReow
![Screenshot 2023-01-20 at 14 14 02](https://user-images.githubusercontent.com/44696192/213622556-0546f12d-182c-4234-8763-7dc82a7498d0.png)
It correctly draw boxes on the remotes.

I am not sure what is happening, which part should I look at to determine what causes this difference?

Yes we had a hard time making the Space output the same bounding boxes as in Colab (eventually it worked on the cats image). It had to do with the Pillow version.

So I'm guessing there might be a difference in Pillow versions here as well

Cc @alaradirik <|||||>Do you mean Pillow changes the input value?
I tried another image
![Screenshot 2023-01-23 at 9 41 05](https://user-images.githubusercontent.com/44696192/213949366-d7647da2-bc34-4e9c-9e73-20f1970b4c43.png)
space model cant detect cat inside this image, but colab version can detect it
![Screenshot 2023-01-23 at 9 42 07](https://user-images.githubusercontent.com/44696192/213949408-d18d9246-64c6-4a1b-aedd-db53092e0030.png)


<|||||>@darwinharianto thanks for bringing the issue up, I'm looking into it!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Kindly bumping<|||||>Kindly reminder<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>cc @alaradirik and @amyeroberts <|||||>I got the same issues. 
This is original repo results.

![image](https://user-images.githubusercontent.com/27891090/233775042-afa92c14-d76c-4d2a-9a2f-5dd70e6dd797.png)

And this is [huggingface demo](https://huggingface.co/spaces/Jiayi-Pan/OWL-ViT).
```
text_queries = text_queries.split(",")
target_sizes = torch.Tensor([img.shape[:2]])
inputs = processor(text=text_queries, images=img, return_tensors="pt").to(device)  
with torch.no_grad():
    outputs = model(**inputs)

outputs.logits = outputs.logits.cpu()
outputs.pred_boxes = outputs.pred_boxes.cpu()
results = processor.post_process(outputs=outputs, target_sizes=target_sizes)
```

<img width="1036" alt="image" src="https://user-images.githubusercontent.com/27891090/233775093-ce8aee88-b0a0-4d81-b917-ab3136c5388d.png">

The `rocket` bounding box score is different. (0.15 vs more than 0.21)

With lvis-api, the performance is not reproduced. (mAP = 0.095)<|||||>It seems problem still exist. I mentioned about problem here. 

https://github.com/huggingface/transformers/pull/23157#issuecomment-1540056705

Maybe the best way is to cover with model predictions end-to-end tests on batch of images. This approach help us to be sure about changes
<|||||>@MaslikovEgor  I agree with you. I have end-to-end test with lvis-api (both huggingface owlvit and google/scenic owl-vit). But owl vit in huggingface is not reproduced. (mAP = 0.095)

- [baseline](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit): mAp 0.193 
<|||||>I want to fix this problem, but it would be efficient if I knew where to start. Can you give me a suggestion? @alaradirik <|||||>Hi @MaslikovEgor,

The demo didn't work before this fix as well (see https://github.com/huggingface/transformers/pull/20136). Try running coco evaluation with image conditioning before/after this fix, mAP@0.5 increases from 6 to 37. This is still below the expected 44, but closer to the reported/expected performance. I am still trying to figure out why.
Best,
Orr<|||||>@RRoundTable, the issues you are reporting seem to do with the text-conditioned evaluation. This means that the issues probably stem from the forward pass/post-processing. 

In your LVIS eval, did you make sure to implement a new post-processor that incorporates all the changes needed for eval? If helpful, I can add my function to 'processor' or something, please notice there are a few changes compared with normal inference.<|||||>@orrzohar, Yes. I tested with text-conditioned evaluation.

In my LVIS eval, I just used huggingface's postprocessor and preprocessor.  It would be helpful if you contribute some functions.

```
transformers[torch] == 4.28.1
```

```
# example script
import requests
from PIL import Image
import torch
import glob
import os
import argparse
import json
from tqdm import tqdm

from transformers import OwlViTProcessor, OwlViTForObjectDetection

parser = argparse.ArgumentParser()
parser.add_argument("--dataset-path", type=str, required=True)
parser.add_argument("--text-query-path", type=str required=True)
parser.add_argument("--save-path", default="owl-vit-result.json", type=str)
parser.add_argument("--batch-size", default=64, type=int)
args = parser.parse_args()

model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32")
processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model.to(device)


with open(args.text_query_path, "r") as f:
    text_query = f.read()

images = glob.glob(os.path.join(args.dataset_path, "*"))
image_ids = [img_path.split("/")[-1].split(".")[0] for img_path in images]

instances = []
N = len(images)

with torch.no_grad():
    for i in tqdm(range(N // args.batch_size + 1)):
        image_ids = []
        batch_images = []
        target_sizes = []
        for img_path in images[i * args.batch_size: (i+1) * args.batch_size]:
            image_ids.append(int(img_path.split("/")[-1].split(".")[0]))
            image = Image.open(img_path).convert("RGB")
            batch_images.append(image)
            target_sizes.append((image.size[1], image.size[0]))
        target_sizes = torch.Tensor(target_sizes)
        target_sizes = target_sizes.to(device)
        texts = [text_query.split(",")] * len(batch_images)
        inputs = processor(text=texts, images=batch_images, return_tensors="pt")
        inputs = inputs.to(device)
        outputs = model(**inputs)
        # Target image sizes (height, width) to rescale box predictions [batch_size, 2]

        # Convert outputs (bounding boxes and class logits) to COCO API
        results = processor.post_process(outputs=outputs, target_sizes=target_sizes)
        for image_id, res in zip(image_ids, results):
            for bbox, score, label in zip(res["boxes"], res["scores"], res["labels"]):
                # tensor to numpy
                bbox = bbox.cpu().detach().numpy()
                score = score.cpu().detach().numpy()
                label = label.cpu().detach().numpy()
                # bbox format: xyxy -> xywh
                x1, y1, x2, y2 = bbox
                bbox = [int(x1), int(y1), int(x2-x1), int(y2-y1)]
                instance = {}
                instance["image_id"] = image_id
                instance["bbox"] = bbox # TODO
                instance["score"] = float(score)
                instance["category_id"] = int(label) + 1 # TODO
                instances.append(instance)
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi @RRoundTable , 

I added a PR with the appropriate evaluation protocol

https://github.com/huggingface/transformers/pull/23982

Best,
Orr<|||||>Hi! @alaradirik,
I'm using transformers==4.30.2 but still encountered the same issue. Any thought on this?

**Query image:**
![image](https://github.com/huggingface/transformers/assets/26133523/11102139-e06a-4640-88ab-c52987578af0)

**Result from colab:**
![image](https://github.com/huggingface/transformers/assets/26133523/75ba1562-5130-424b-af52-1614ffd767c5)

**Result from huggingface:**
![image](https://github.com/huggingface/transformers/assets/26133523/ab0d8d87-a49e-4323-8389-371f787b1a56)
