Empty prediction masks after switching from transformers 4.26.1 to transformers 4.29.0
### System Info

Hey guys, I have been working on the 4.26.1 version perfectly fine, but I wanted to switch to 4.29.0 to make use of the latest models (such as SAM). The issue I encounter is that when running my prediction code for models such as MaskFormer and Mask2Former, my outputs between versions 4.26.1 and 4.29.0 do not match at all (4.26.1 works fine for all models, while 4.29.0 gives me empty or wrong predictions). Anything I am missing here?

- `transformers` version: 4.29.0
- Platform: Linux-4.15.0-194-generic-x86_64-with-glibc2.27
- Python version: 3.10.9
- Huggingface_hub version: 0.12.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running a very simple prediction pipeline that looks like this:

model instance:
```python
mask2former = Mask2Former(<args_of_my_pytorch_lightning_model>)
checkpoint = torch.load(<path_to_weights>)
mask2former.load_state_dict(checkpoint['state_dict'])
mask2former.eval()
```
data module instance:

```python
dm.setup(stage="test")
dl = dm.test_dataloader()
```
simple prediction loop
```python
for sample in dl:
    pred_mask2former = mask2former.forward(sample["pixel_values"])
```

### Expected behavior

The outputs produced by the model do not match between versions `4.26.1` and `4.29.0` of the `transformers` package. I get the expected behavior with `4.26.1`, but empty or (very) wrong predictions with 4.29.0` on the exact same data (using the same code/models/...).
Hi @alzaia, thanks for reporting this issue. 

To help us dig into the problem, we need to be able to reproduce the issue. Could you share a minimal snippet that we could use to compare the versions? Specifically, a model checkpoint, any processing logic to produce `sample` and any additional information which might affect the model e.g. `<args_of_my_pytorch_lightning_model>`. <|||||>Hello @amyeroberts , thanks for the reply. Unfortunately I cannot provide data or checkpoints since this is not a public project, but I can give you more information on the processing logic.

On the model side, I have a pretty standard pytorch lightning module that looks like this (example with MaskFormer):
```python
# Load pretrained model weights (I am using the `facebook/maskformer-swin-tiny-coco` weights here)
self.model = MaskFormerForInstanceSegmentation.from_pretrained(args)
# Load image processor
self.img_processor = MaskFormerImageProcessor.from_pretrained(<name_of_pretrained_model>)
# In the forward method
outputs = self.model(pixel_values=pixel_values)
# Then calling the right post-processing method from self.img_processor to reformat the output for my needs
```
On the data side, my `sample` is a standard pre-processed object with keys `['image', 'mask', 'pixel_values', 'pixel_mask', 'mask_labels', 'class_labels']`. I feed the `pixel_values` tensor in my forward, which is a tensor of shape `torch.Size([1, 3, 256, 256])`. 

Do you guys feel like this is enough information to try to reproduce it? It may be that some default arguments of the `img_processor` or the `from_pretrained` method of the model changed in the newest version? I will try to look more into that.





<|||||>@alzaia OK, I understand if you are unable to share e.g. weights. However, to be able to diagnose the model behaviour, it is necessary to know any changes to the default model architecture / behaviour. 

* Is `facebook/maskformer-swin-tiny-coco` the checkpoint being used for both the image processor and the model? 
* For `self.model = MaskFormerForInstanceSegmentation.from_pretrained(args)` - could you share `args`? Specifically, are there any settings which might affect the model's weights e.g. `load_in_8bit`, `device_map`? Are there any kwargs overidding the model config defaults e.g. `mask_feature_size`, `backbone_config` etc? 
* What format are the images being passed to the image processor e.g. PIL images? 
* Are any settings of the image processor being overriden in the processing call e.g. `do_resize=False`? 
* When you say `"calling the right post-processing method"` - which one is being used? Are the empty predictions being observed in the raw model outputs or after post processing? <|||||>Thanks for the fast reply @amyeroberts, I can give more specific details on that.

- Right, I am using the same checkpoint (`facebook/maskformer-swin-tiny-coco`) for both the model and the image processor here.
- For the loading of the pretrained model, I am using the following:
```python
# Doing multiclass (8 classes) with my own label to id mapping dictionary (all args not specified here are using the default values of course):
self.model = MaskFormerForInstanceSegmentation.from_pretrained(
            `facebook/maskformer-swin-tiny-coco`,
            num_labels=8,
            id2label=self.id2label,
            label2id=self.label2id,
            ignore_mismatched_sizes=True,
        )
```
- For the image processor, I am using, for example:

```python
self.img_processor = MaskFormerImageProcessor.from_pretrained(`facebook/maskformer-swin-tiny-coco`)
        self.img_processor.do_resize = True
        self.img_processor.size = 256
```
- For the post-processing method, it looks like this:
```python
post_processed_output = self.img_processor.post_process_semantic_segmentation(
            outputs,
            target_sizes=[
                [pixel_values.shape[2], pixel_values.shape[3]]
                for _ in range(pixel_values.shape[0])
            ],
        )
```
After running a prediction on the exact same sample to compare the `output` object returned by the model, I realize that it does not return the same logits. For instance, with `v4.26.1` I get an `outputs["masks_queries_logits"]` that looks like this:
```
tensor([[[[-1.0812e+01, -1.1791e+01, -1.1831e+01,  ..., -1.1767e+01,
           -1.2532e+01, -1.0802e+01],
          [-1.1643e+01, -1.1272e+01, -1.1423e+01,  ..., -1.1962e+01,
           -1.2646e+01, -1.1662e+01],
          [-1.1235e+01, -1.0905e+01, -1.0670e+01,  ..., -1.1819e+01,
           -1.2373e+01, -1.1361e+01],
          ...,
```
While with `v4.29` I get the following `outputs["masks_queries_logits"]`:
```
tensor([[[[ 5.7369,  6.7416,  6.6841,  ...,  4.9752,  5.1076,  4.8850],
          [ 7.2394,  8.4208,  8.4004,  ...,  6.0630,  6.2035,  5.7476],
          [ 7.2572,  8.4786,  8.4989,  ...,  5.8866,  6.0229,  5.6737],
          ...,
```
Which seems to indicate that the post-processing is fine, the problem arises during the prediction using the model.

In terms of data, I do not do anything fancy, I do start with PIL images, but convert them to numpy arrays, and then process them with the right model processor:
```python
self.img_processor:
            processed_inputs = self.img_processor(
                images=image, segmentation_maps=mask, return_tensors="pt"
            )
```
Thanks for any insights to what may be causing the issue.





<|||||>@alzaia Thanks for the additional info, it really helps :) 

There's a few things to note from the examples: 

**1. Model instantiation**

In the snippet: 

```python
model = MaskFormerForInstanceSegmentation.from_pretrained(
    `facebook/maskformer-swin-tiny-coco`,
    num_labels=8,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)
```

when you change the number of prediction classes, the pretrained weights for the classification head will be thrown away, and new randomly initialized weights with the correct dimensions created. Creating the model you should see: 

```
Some weights of MaskFormerForInstanceSegmentation were not initialized from the model checkpoint at facebook/maskformer-swin-tiny-coco and are newly initialized because the shapes did not match:
- class_predictor.weight: found shape torch.Size([134, 256]) in the checkpoint and torch.Size([9, 256]) in the model instantiated
- class_predictor.bias: found shape torch.Size([134]) in the checkpoint and torch.Size([9]) in the model instantiated
- criterion.empty_weight: found shape torch.Size([134]) in the checkpoint and torch.Size([9]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

This means that the output `output.class_queries_logits` will essentially be nonsense until the model has been finetuned on the downstream task. This is the case for both transformers 4.26.1 and the most recent version. This then has an effect on the segmentation masks post processing, [specifically here](https://github.com/huggingface/transformers/blob/deff5979fee1f989d26e4946c92a5c35ce695af8/src/transformers/models/maskformer/image_processing_maskformer.py#LL988C9-L988C9).

When I ran the following script multiple times with transformers==4.26.1, I saw many different predicted (sometimes empty) masks:

```python
import requests

import matplotlib.pyplot as plt
import numpy as np
import torch
from PIL import Image

import transformers
from transformers import MaskFormerForInstanceSegmentation, MaskFormerImageProcessor

CHECKPOINT = "facebook/maskformer-swin-tiny-coco"

id2label = {i: str(i) for i in range(8)}
label2id = {str(i): i for i in range(8)}

model = MaskFormerForInstanceSegmentation.from_pretrained(
    CHECKPOINT, num_labels=8, id2label=id2label, label2id=label2id, ignore_mismatched_sizes=True,
)

image_processor = MaskFormerImageProcessor.from_pretrained(
    CHECKPOINT,
    size={"shortest_edge": 256, "longest_edge": 1333}
)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = image_processor(images=image, return_tensors="pt")
target_sizes = [[pv.shape[-2], pv.shape[-1]] for pv in inputs["pixel_values"]]

with torch.no_grad():
    outputs = model(**inputs)

segmentation_masks = image_processor.post_process_semantic_segmentation(
    outputs, target_sizes
)

plt.imshow(segmentation_masks)
plt.show()
```

Some examples from v.4.26.1:
![maskformer_4_26_1__1](https://github.com/huggingface/transformers/assets/22614925/205d9a3e-7bc8-4df8-af53-54010b3f75ab)
![maskformer_4_26_1__2](https://github.com/huggingface/transformers/assets/22614925/af572f31-f8bb-4125-860b-9171180a5e35)
![maskformer_4_26_1__3](https://github.com/huggingface/transformers/assets/22614925/522abca2-767b-4a78-8930-6e0b749f2db3)

I observed the same for the most recent release, 4.30.1. and 4.29.2.

Could you try running this in your environment to see if you observe the same behaviour? This way we can try and pin down the differences between our respective environments.  

**2. Image Processor**

The `size` parameter for the image processors is now a dictionary. Although it should still work because of efforts to create backwards compatibility, the equivalent dictionary should be used to change the behaviour. Note: this can also be set in the `from_pretrained` call:

```python
image_processor = MaskFormerImageProcessor.from_pretrained(
    CHECKPOINT,
    size={"shortest_edge": 256, "longest_edge": 1333}
)
```

**3. Mask Queries Logits**

This is interesting - if I save out `outputs.mask_queries_logits` from a run in v4.30.1/v4.29.2 and v.4.26.1 there's 0 difference. However, if I find the largest absolute difference for `outputs_class_queries_logits` between the two versions, it's typically ~3. This will be due to the randomly initialized head.

**4. Image processing**

You don't need to convert to numpy images before passing to the image processor, you can pass in PIL images directly :) 
 



<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.