Compared OneFormer by transformers with  original GitHub code, It's worked bad . Why ?
### System Info

transformers == 4.26.0

Python == 3.8.8

### Who can help?

@praeclarumjj3 @NielsRogge

### Reproduction

Thanks for  your great Work !
I compared OneFormer in transformers with  original [ original  GitHub code](https://github.com/SHI-Labs/OneFormer), It sometimes worked bad .

OneFormer by transformers:
```
from transformers import OneFormerProcessor, OneFormerForUniversalSegmentation
from collections import defaultdict
import matplotlib.pyplot as plt
from matplotlib import cm
import matplotlib.patches as mpatches

def draw_panoptic_segmentation(segmentation, segments_info):
    segmentation= segmentation.to(cpu_device)
    # get the used color map
    viridis = cm.get_cmap('viridis', torch.max(segmentation))
    fig, ax = plt.subplots()
    ax.imshow(segmentation)
    instances_counter = defaultdict(int)
    handles = []
    # for each segment, draw its legend
    for segment in segments_info:
        segment_id = segment['id']
        segment_label_id = segment['label_id']
        segment_label = model_oneformer.config.id2label[segment_label_id]
        label = f"{segment_label}-{instances_counter[segment_label_id]}"
        instances_counter[segment_label_id] += 1
        color = viridis(segment_id)
        handles.append(mpatches.Patch(color=color, label=label))
        
    ax.legend(handles=handles)
    plt.savefig('./panoptic.png')

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_coco_dinat_large")
model = OneFormerForUniversalSegmentation.from_pretrained("shi-labs/oneformer_coco_dinat_large").to(device)

image = original_image.resize((512, 512))
inputs = processor(image, ["panoptic"], return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)

# you can pass them to processor for panoptic postprocessing
panoptic_segmentation = processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]
    print(panoptic_segmentation.keys())

draw_panoptic_segmentation(**panoptic_segmentation)
```

original  GitHub code:
```
from detectron2.config import get_cfg
from detectron2.projects.deeplab import add_deeplab_config
# from detectron2.data import MetadataCatalog

from oneformer import (
    add_oneformer_config,
    add_common_config,
    # add_swin_config,
    add_dinat_config,
)

from demo.defaults import DefaultPredictor

def setup_cfg():
    # load config from file and command-line arguments
    cfg = get_cfg()
    add_deeplab_config(cfg)
    add_common_config(cfg)
    # add_swin_config(cfg)
    add_oneformer_config(cfg)
    add_dinat_config(cfg)
    cfg_path = "OneFormer/configs/coco/oneformer_dinat_large_bs16_100ep.yaml"
    cfg.merge_from_file(cfg_path)
    if torch.cuda.is_available():
        cfg.MODEL.DEVICE = 'cuda'
    else:
        cfg.MODEL.DEVICE = 'cpu'
    # cfg.MODEL.WEIGHTS = hf_hub_download(repo_id="shi-labs/oneformer_coco_dinat_large", 
    #                                         filename="150_16_dinat_l_oneformer_coco_100ep.pth", local_dir=local_dir)
    cfg.MODEL.WEIGHTS = 'OneFormer/oneformer_coco_dinat_large/150_16_dinat_l_oneformer_coco_100ep.pth'
    cfg.freeze()
    return cfg

predictor = setup_modules()
img = cv2.resize(img.astype(np.uint8), (512, 512),interpolation=cv2.INTER_AREA)
predictions = predictor(img, "panoptic")
panoptic_seg, segments_info = predictions["panoptic_seg"]

```


### Expected behavior

input image:
![13201678780389_ pic](https://user-images.githubusercontent.com/21029719/229674653-c03cb9c3-240c-458c-904a-f9c3a5d6e71c.jpg)

OneFormer by transformers output:
![panoptic](https://user-images.githubusercontent.com/21029719/229674824-18bf9b5b-412d-4fd6-82cb-cac8c058a178.png)

OneFormer by original GitHub output:
![out](https://user-images.githubusercontent.com/21029719/229674928-f9f59080-d47d-4b4e-b97a-58d74faf1269.png)

cc @alaradirik and @amyeroberts <|||||>Hi @onefish51, thanks for reporting this issue! It's definitely not the desired behaviour, and I'm digging into it now. 

As a first pass, it seems this is related to specific checkpoints and inputs. For example, for the checkpoint you provided, `shi-labs/oneformer_coco_dinat_large` I also see poor segmentation for the example image: 
![image](https://user-images.githubusercontent.com/22614925/230117033-ecbb3585-be95-4146-a13e-bc6720d83fa1.png)

But good segmentation for a different input image:
![image](https://user-images.githubusercontent.com/22614925/230115145-c1269b8d-d97a-40db-8801-b44a8456904e.png)

And a different oneformer checkpoint `shi-labs/oneformer_coco_swin_large` outputs a reasonable segmentation map:
![image](https://user-images.githubusercontent.com/22614925/230115821-3980230e-471a-4335-b983-784fe1084055.png)

This indicates to me that the differences are more likely to be coming from the model outputs than the pre/post processing steps, TBC.

To help narrow down the effects, could you share some more information about the environment you're using (run `transformers-cli env` to get the info) and the device the model is being run on e.g. `"cpu"`?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>cc @praeclarumjj3<|||||>> Hi @onefish51, thanks for reporting this issue! It's definitely not the desired behaviour, and I'm digging into it now.
> 
> As a first pass, it seems this is related to specific checkpoints and inputs. For example, for the checkpoint you provided, `shi-labs/oneformer_coco_dinat_large` I also see poor segmentation for the example image: ![image](https://user-images.githubusercontent.com/22614925/230117033-ecbb3585-be95-4146-a13e-bc6720d83fa1.png)
> 
> But good segmentation for a different input image: ![image](https://user-images.githubusercontent.com/22614925/230115145-c1269b8d-d97a-40db-8801-b44a8456904e.png)
> 
> And a different oneformer checkpoint `shi-labs/oneformer_coco_swin_large` outputs a reasonable segmentation map: ![image](https://user-images.githubusercontent.com/22614925/230115821-3980230e-471a-4335-b983-784fe1084055.png)
> 
> This indicates to me that the differences are more likely to be coming from the model outputs than the pre/post processing steps, TBC.
> 
> To help narrow down the effects, could you share some more information about the environment you're using (run `transformers-cli env` to get the info) and the device the model is being run on e.g. `"cpu"`?

hi @amyeroberts  thanks for reporting !  I guess it has nothing to do with the env, and I run it by GPU V100
```sh
$ transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-146-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
and you said 
>the differences are more likely to be coming from the model outputs than the pre/post processing steps, TBC.

may right<|||||>and I will  create a colab to compare it , if needed<|||||>@onefish51 it's also worth mentioning that OneFormer was added by the authors themselves. I'd recommend comparing the output logits of models, which should be the same.<|||||>Hi, @onefish51, thanks for your interest in OneFormer and bringing up this issue. I am sorry for not replying earlier. I am currently occupied with a few other things.

This strange case could be image specific for the DiNAT-L OneFormer checkpoint. As @alaradirik suggested, comparing the HF model outputs **for this image** to the model outputs using the official GitHub repo could provide some hints. The tests suggest they should be the same, but it's good to look.

I'll do this myself once I get some time. Please let us know if you compare these on your end.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.